Backup and Restore
This topic describes the backup and restore capabilities available in Paragon Automation. Although Paragon Automation is a GUI-based application, the backup and restore operations are managed from the Paragon Insights cMGD CLI. Postgres is the primary persistent storage database for microservices. Backup files are saved in a local persistent volume on the cluster nodes. The backup procedure can be performed while microservices are running and does not affect the operation of the cluster. However, for restore procedures, microservices are stopped and the cluster is not functional until the databases are restored.
Currently, you cannot custom select applications to be backed up and restored. You can back up and restore only a preconfigured and fixed set of applications and administrations settings for each component, as listed in Table 1.
Devices |
Alerts/Alarm Settings |
Admin Groups |
Topics |
Plot Settings |
User Defined Actions and Functions |
Playbooks |
Summarization Profiles |
Auditlogs |
Device Groups |
Ingest Settings |
Topology Filter Configuration |
Network Groups |
SNMP Proxy Configuration |
Pathfinder Settings |
Notification Settings |
IAM Settings |
LSP Policies and Profiles |
Retention Policies |
Workflows |
Report Generation Settings (Destination, Report and Scheduler Settings) |
The backup procedure has the following limitations:
-
Telemetry data-Data captured from the devices will not be backed up, by default. Telemetry data must be backed up manually.
For more information, see Backup and Restore the TSDB.
-
Transient and logging data-Data which is being processed and expired events will not be backed up. For example:
-
Alerts and alarms generated
-
Configuration changes which are not committed
-
Most application logs
-
-
Non-Paragon-Automation Configuration-Configuration done on third-party services supported by Paragon Automation will not be backed up. For example:
-
Device configuration on Anuta ATOM
-
LDAP user details
-
-
Topology Ingest Configuration-The cRPD configuration to peer with BGP-LS routers for topology information will not be backed up. This must be manually reconfigured again as required. For more information, see Modify cRPD Configuration.
You use containerized scripts invoked through Kubernetes jobs to implement the backup and restore procedures.
You can manually back up your cluster using the instructions described in Back Up the Configuration. You can also, use a backup script to backup your cluster using the instructions described in Backup and Restore Scripts.
Similarly, you can manually restore the backed up configuration using the instructions described in Restore the Configuration. While a restore script is available and described in Backup and Restore Scripts, restore is a manual process and you will still need to execute the restore commands manually.
For Paragon Automation Release 23.1, you cannot restore a backed up configuration from releases earlier than Release 22.1.0-SP2. To use the restore operation on a Release 23.1 cluster, we recommend that you:
-
Upgrade your current Paragon Automation cluster to Release 22.1.0-SP2.
-
Back up the Release 22.1.0-SP2 configuration.
-
Upgrade the Release 22.1.0-SP2 cluster to Release 23.1.
-
Restore the backed up configuration.
Back Up the Configuration
Data across most Paragon Automation applications is primarily stored in Postgres. When you back up a configuration, system-determined and predefined data is backed up. When you perform a backup, the operational system and microservices are not affected. You can continue to use Paragon Automation while a backup is running. You'll use the management daemon (MGD) CLI, managed by Paragon Insights (formerly Healthbot), to perform the backup.
To back up the current Paragon Automation configuration:
Frequently Used kubectl Commands to View Backup Details
To view the status of your backup or the location of your backup files, or to view more information on the backup files, use the following commands.
-
Backup jobs exist in the common namespace and use the
common=db-backup
label. To view all backup jobs:root@primary-node:~# kubectl get -n common jobs -l common=db-backup NAME COMPLETIONS DURATION AGE db-backup-hello-world 1/1 3m11s 2d20h
-
To view more details of a specific Kubernetes job:
root@primary-node:~# kubectl describe -n common jobs/db-backup-hello-world
-
To view the logs of a specific Kubernetes job:
root@primary-node:~# kubectl logs -n common --tail 50 jobs/db-backup-hello-world
-
To determine the location of the backup files:
root@primary-node:~# kubectl get -n common pvc db-backup-pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE db-backup-pvc Bound local-pv-cb20f386 145Gi RWO local-storage 3d3h
The output points you to the local persistent volume. Use that persistent volume to determine the node on which the backup files are stored.
root@primary-node:~# kubectl describe -n common pv local-pv-cb20f386 Node Affinity: Required Terms: Term 0: kubernetes.io/hostname in [10.49.xxx.x2] Message: Source: Type: LocalVolume (a persistent volume backed by local storage on a node) Path: /export/local-volumes/pv*
To view all the backup files, log in to the node and navigate to the location of the backup folder.
root@primary-node:~# ssh root@10.49.xxx.x2 root@10.49.xxx.x2:~# ls -l /export/local-volumes/pv*
To view commonly seen backup and restore failure scenarios, see Common Backup and Restore Issues.
Restore the Configuration
You can restore a Paragon Automation configuration from a previously backed-up configuration folder. A restore operation rewrites the databases with all the backed-up configuration information. You cannot selectively restore databases. When you perform a restore operation, a Kubernetes job is spawned, which stops the affected microservices. The job restores the backed-up configuration and restarts the microservices. Paragon Automation remains nonfunctional until the restoration procedure is complete.
You cannot run multiple restore jobs at the same time because the Kubernetes job stops the microservices during the restoration process. Also, you cannot run both backup and restore processes concurrently.
We strongly recommend that you restore a configuration during a maintenance window, otherwise the system can go into an inconsistent state.
To restore the Paragon Automation configuration to a previously backed-up configuration:
Frequently Used kubectl Commands to View Restore Details
To view more information and the status of your restore process, use the following commands:
-
Restore jobs exist in the common namespace and use the
common=db-restore
label. To view all restore jobs:root@primary-node:~# kubectl get -n common jobs -l common=db-restore NAME COMPLETIONS DURATION AGE db-restore-hello-world 0/1 20s 21s
-
To view more details of a specific Kubernetes job:
root@primary-node:~# kubectl describe -n common jobs/db-restore-hello-world
-
To view the logs of a particular Kubernetes job:
root@primary-node:~# kubectl logs -n common --tail 50 jobs/db-restore-hello-world
To view commonly seen backup and restore failure scenarios, see Common Backup and Restore Issues.
Backup and Restore Scripts
You can also use the Paragon Automation backup and restore scripts to simplify the backup and restore operations. This topic describes the backup and restore script operations and the caveats around the usage of the scripts.
Backup Script Operation
The backup script automatically backs up your current configuration. The primary benefit of the backup script is that you can run it as a cron job with the required frequency so as to schedule regular backups. Additionally, the backup script creates distinguishable date stamped backup folders and the folders do not get overwritten if the script is run on different days.
To back up your configuration using the backup script:
-
Log in to any one of the primary nodes.
-
Execute the backup script.
root@primary-node:~# data.sh --backup
The script runs a backup job to back up your current configuration. A backup folder is created and saved in a local persistent volume on one of the cluster nodes. The folder name is in the <name>-year_month_day format. The folder in your cluster node contains all your backed up configuration metadata.
The script also creates a folder of the same name in the current path in your primary node. The backup folder in your primary node contains the JSON files required for Anuta ATOM and base platform used while restoring the backed up configuration.
As the script is running, a backup summary is generated and displayed onscreen. The summary contains the node and location of the backup files. For example:
===============================Get Backup file location====================== Name: local-pv-3a1d8444 Labels: <none> Annotations: pv.kubernetes.io/bound-by-controller: yes pv.kubernetes.io/provisioned-by: local-volume-provisioner-10.16.18.22-505b0d01-d1d8-4042-b001-e97091b34355 Finalizers: [kubernetes.io/pv-protection] StorageClass: local-storage Status: Bound Claim: common/db-backup-pvc Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 149Gi Node Affinity: Required Terms: Term 0: kubernetes.io/hostname in [10.16.18.22] Message: Source: Type: LocalVolume (a persistent volume backed by local storage on a node) Path: /export/local-volumes/pv4 Events: <none>
In this example, the backup folder containing all the backup metadata is stored in your cluster node with IP address 10.16.18.22 in the /export/local-volumes/pv4 folder.
Restore Script Operation
The restore operation continues to be a manual process. However, the restore script provides you a quick reference of the commands you must execute in order to restore your configuration. You can execute the displayed commands in sequence to restore the configuration.
To restore your configuration using the restore script:
-
Log in to any one of the primary nodes.
-
Execute the restore script.
root@primary-node:~# data.sh --restore
The restore script echoes the following commands onscreen:
echo "Restoring Paragon Automation is a manual process." echo "Please see the following website for more information." echo "https://www.juniper.net/documentation/us/en/software/paragon-automation23.1/paragon-automation-installation-guide/topics/task/paragon-backup-restore.html" echo "=============================================================================" echo " " echo "The following commands are examples of using the utility to restore from backup." echo " " echo "To obtain the current available backup jobs run the cmd : kubectl -n common get jobs " echo " " echo "To restore run the following cmd: kubectl exec -ti -n healthbot mgd-779b6dff95-vstbn -- cli -c "request system restore path %backup-path% "" echo " " echo "To check the progress of the restore run the following cmd: kubectl describe -n common jobs db-backup-paa-2023-04-20" echo " " echo "=============================================================================" echo "The following cmds are for running an EMS Restore." echo "=============================================================================" echo "kubectl delete secrets -n ems jobmanager-identitysrvcreds devicemodel-connector-default-scope-id" echo "kubectl apply -f ems-scope-bkup.yaml" echo " " echo "=============================================================================" echo "The following cmds are for running Pathfinder Kubernetes Config Restore." echo "=============================================================================" echo "kubectl delete -n northstar secret --selector proxy=anuta-restconf" echo "kubectl delete -n northstar configmap --selector proxy=anuta-restconf" echo "kubectl apply -f ns-anuta-secret-bkup.json" echo "kubectl apply -f ns-anuta-cmap-bkup.json" echo "kubectl delete -n northstar pod --selector app=ns-anuta-proxy" echo " "
-
Execute these commands in sequence to restore the configuration from the required backup folder.
Caveats of Backup and Restore Scripts
The caveats of the backup and restore scripts are as following:
-
You can run the scripts either on a weekly basis or only once daily. Running them multiple times in a 24-hour period returns an error since there is already a backup folder for that day named <name>-year_month_day. If you need to take a manual backup in the same 24-hour period, you must remove the job using the
kubectl delete -n common jobs
command. For example:# kubectl delete -n common jobs db-backup-paa-2023_20_04
-
The scripts fill disk space with backup files depending on the frequency and size of backup files. Consider removing outdated backup metadata and files to free up disk space. You can remove the Kubernetes metadata using the
kubectl delete -n common jobs
command. For example:# kubectl delete -n common jobs db-backup-paa-2023_20_04
You can remove the backup files by deleting the <name>-year-month-day folders created in the /root/ folder in the local volume path displayed in the summary when you run the backup script.