General Troubleshooting Techniques
-
Before you begin any troubleshooting investigation, confirm that all pods associated with the system processes are running.
To confirm whether a pod is running, do the following:
-
Use your server credentials to log in to the primary node of Paragon Automation.
-
From the primary node, use the following command to confirm whether the pods associated with the system processes are running:
kubectl -n northstar get pod
The output displays the list of pods and their details. For each pod that is running, the Status column corresponding to that pod displays the word RUNNING. Pods that have stopped running then restart automatically.
-
-
If a pod is not in the Running state, view the events associated with the pod by using the following command. The events help you identify the problem:
kubectl -n northstar describe pod pod-name
In the command, pod-name is the name of the pod for which you want to view the events.
-
Confirm whether a system process is running by logging in to the pod associated with the process and view the logs. See the View Logs for details.
-
Run CLI commands—You can run CLI commands (ping, traceroute, and vendor-specific command sets) on the following network elements without manually logging in to the devices. You can perform this task from the Diagnostics page in the GUI.
Note:You must be able to connect to the devices through SSH; otherwise, you will not be able to run CLI commands.
-
Nodes—For details about running CLI commands on nodes, see About the Node Tab in the Paragon Automation User Guide.
-
Links—For details about running CLI commands on links, see About the Link Tab in the Paragon Automation User Guide.
-
Tunnels—For details about running CLI commands on tunnels, see About the Tunnel Tab in the Paragon Automation User Guide.
-
Interfaces—For details about running CLI commands on interfaces, see About the Interface Tab in the Paragon Automation User Guide.
The CLI commands for nodes, links, tunnels, and interfaces are present in the corresponding configuration files located in their respective pods. You can modify these configuration files by using a text editor of your choice.
For example, the CLI commands configuration file (clicommands.json) contains vendor-specific command sets. By default, the commands in the file are organized into categories such as Accounting Info, Chassis, or File Operation. The same categories appear in the Diagnostics list when you run CLI commands from the GUI. These categories are vendor-specific. When you select a node on which to run CLI commands, Paragon Pathfinder collects the vendor type of the node. Paragon Pathfinder then displays the categories and command sets that are applicable. You can modify the category names and the organization of commands within the categories. You can also add commands. The same is true for the ping and traceroute categories.
The following example shows the structure of entries in the clicommands.json file:
{ "cmd": "show config | display inheritance|no-more", "displayName": "Show configuration", "cmdId": 380 }, { "cmd": "show config logical (LR) | display inheritance|no-more", "displayName": "Show configuration (LR)", "cmdId": 1571 },
To modify the configuration files, do the following:
-
Log in to the primary node of Paragon Automation by using your server credentials.
-
From the primary node, obtain the name of the pod associated with the ns-web process by using the following command:
.
kubectl -n northstar get pod | grep ns-web
The log file for the ns-web process contains log entries for the REST API requests from the GUI
-
Log in to the pod associated with the ns-web process by using the following command:
kubectl -n northstar exec -it ns-web-pod-name -c ns-web -- bash
In the command, ns-web-pod-name is the name of the pod that you obtained in Step 2.
-
View the list of configuration files present in the pod by using the following command:
ls /opt/pcs/db/diagnostic/
The output displays the ping (
pingcommands.json
), traceroute (traceroutecommands.json
), and CLI (clicommands.json
) configuration files.
-
-
View logs—You can view logs for all Paragon Automation components on the Logs page (Monitoring > Logs) in the GUI.
Alternatively, you can view the logs by using the CLI.
The following example explains the procedure to view the logs associated with the Config Server process by using the CLI. You can use the same procedure to view the logs for other processes.
-
Log in to the primary node of Paragon Automation by using your server credentials.
-
From the primary node, obtain the name of the pod associated with the Config Server process by using the following command:
kubectl -n northstar get pod | grep configserver
-
View the logs associated with the Config Server process by using the following command:
kubectl -n northstar logs -f ns-configserver-pod-name -c syslog
In the command, ns-configserver-pod-name is the name of the pod that you obtained in Step 2.
The output displays the logs associated with the Config Server process.
Table 1 provides information about the most commonly used log files for troubleshooting Paragon Pathfinder.
Table 1: Commonly Used Paragon Pathfinder Log Files Log File
Description
ns-pceserver
The log file contains log entries related to the Path Computation Element (PCE) server. The PCE server maintains the Path Computation Element Protocol (PCEP) session. The logs contain information about communication between the Path Computation Client (PCC) and the PCE in both directions.
ns-pcserver
The log file contains log entries related to the Path Computation Server (PCS). The PCS is responsible for path computation. The logs include events that the PCS receives from the Toposerver, including provisioning orders. The logs also contain notifications of communication errors and issues that prevent the PCS from starting properly.
ns-toposerver
The log file contains log entries that are related to the topology server (also called a Toposerver in Paragon Pathfinder). The Toposerver is responsible for maintaining the topology. The logs contain a record of the events between the PCS and the Toposerver, the Toposerver and the BGP Monitoring Protocol (BMP), and the Toposerver and the PCE server.
ns-web
The log file contains log entries related to the REST API requests from the Paragon Pathfinder GUI.
-
-
View historical events for links and tunnels—On the Events page in the GUI, you can view historical events associated with links and tunnels. A graph with a timeline view displays the events for a selected time range. The Events table at the bottom of the page displays the details of these events.
Note:The events displayed on the Events page pertain only to external communication to and from the PCE. Most of the communications internal to the PCE are captured only in the log files.
-
To access the Events page for links, navigate to the Link tab in the network information table (Network > Topology). Select the link for which you want to view the events, and select View > Link Events. The Events page appears. For more information, see About the Link Tab in the Paragon Automation User Guide.
-
To access the Events page for tunnels, navigate to the Tunnel tab in the network information table (Network > Topology). Select the tunnel for which you want to view the events, and select View > Events. The Events page appears. For more information, see About the Tunnel Tab in the Paragon Automation User Guide.
-
If you encounter a problem with the network that Paragon Pathfinder controls, you can run CLI commands for various elements in the network to identify the cause of the problem. See Run CLI Commands for details.