ON THIS PAGE
Upgrade APM to a New Version Using the APM installation Utility
Display the Running Database Using the Kubernetes Command Line Tool
Perform a Database Switchover Using the Kubernetes Command Line Tool
Check the Status of APM Services Using the Kubernetes Command Line Tool
Display the APM IP Addresses Using the Kubernetes Command Line Tool
Archive the APM Configuration Using the Kubernetes Command Line Tool
How to Access APM Configuration and Operational Commands Using the APM Utility
How to Use the APM Command Line Tool Without Using the APM Utility
How to Use Command Line Tools to Administer APM
After you've installed the Address Pool Manager (APM) application, you can perform the following administration functions.
Address Pool Manager gives you two command line options for perform administrator tasks. You
can either use the APM utility script (apm
) or the Kubernetes Command Line
tool to administer APM.
Access APM Utility Commands
Use the APM utility commands to perform administration functions.
You can use the APM utility script (apm
) to administer the application
and to access the CLI that you use to configure the address management functions. The
Juniper APM installation places the utility script in /var/local/apm
and creates a symbolic link to the script in /usr/local/bin/apm.
You can use the apm
utility script (which uses the Kubernetes command
line tool and Helm commands ) to do the following:
- Create and delete objects.
- Provide log access.
- Conduct interactive sessions with pod containers.
- Display the status of the APM objects.
Using the apm
utility script simplifies many of your administrative
duties. The script performs the tasks you need to manage APM, while masking the complexity
of the kubectl
command.
Table 1 lists the commands that you can invoke with the apm
utility script and
describes the action that occurs. Many of the individual commands have options that you
can specify.
Name | Action |
---|---|
clean | Clean up unneeded releases and/or docker cache. To run this command, you need
sudo privileges. Note: Introduced in Release 3.2. |
cli | Access the CLI that you can use to configure APM features and to monitor the current status for managed BNGs. |
contexts | Display the available cluster contexts for control with APM Note: Introduced in Release 3.2. |
db-info | Displays current state of APM’s database microservice including the current
version, stateful set pods, and their roles. Note: Introduced in Release 3.2. |
db-switchover | Forces the persistent state database (DB) primary pod to switchover to an
eligible backup pod. To run this command, you need sudo privileges. Introduced in
Release 3.2. Note: DB switchover is a service disrupting event and you only use it with the upgrade procedure. |
ip | Displays the IP addresses of Address Pool Manager. Note: Introduced in Release 3.2. |
link | Links a cluster to a specific software version. Note: Introduced in Release 3.2. |
logs | Display APM logs. |
rename-context | Rename a context. Does not effect the running Address Pool Manager on the
cluster. To run this command, you need sudo privileges. Note: Introduced in Release 3.2. |
restart | Restart one or more specified services. To run this command, you need sudo privileges. Note: Introduced in Release 3.1. |
rollout | Upgrade or start an APM service. To run this command, you need sudo privileges. Note: Introduced in Release 3.2. |
save-config | Saves the current configuration of the Address Pool Manager to a file outside
the pod. To run this command, you need sudo privileges. Note: Introduced in Release 3.2. |
shell | Connect to a running APM microservice. To run this command, you need sudo privileges. |
setup | Set up the APM application as part of the installation process. To run this command, you need sudo privileges. |
start | Start all APM services. To run this command, you need sudo privileges. |
status | Display the current status of the APM services. To run this command, you need sudo root privileges. |
storage | Provides the status of the storage drivers for APM. Note: Introduced in Release 3.2. |
stop | Stop all APM services. To run this command, you need sudo privileges. |
unlink | Unlink components associated with the context. To run this command, you need
sudo privileges. Note: Introduced in Release 3.2. |
version | Displays the version of every running microservice in the APM instance as well as the APM utility. It also lists all available APM software releases on the system. |
Use the following general syntax to issue a command:
For a short option:
content_copy zoom_out_map$ apm command-name -option
For a long option:
content_copy zoom_out_map$ apm command-name ––option
To target a command at a particular cluster context, use the context option.
$ apm command-name --context context-name
To display a list of available commands with a brief description, use either the
h
or help
option:
$ apm -h
$ apm --help
To display the options for a specific command:
$ apm command-name -h
To specify the --no-color option to disable colored-text output (used to distinguish logs from different microservices):
$ apm command-name --nocolor
Upgrade APM to a New Version Using the APM installation Utility
Use this procedure to upgrade to a new version of APM which is installed on a cluster that was created by the BBE Cloudsetup utility or by Red Hat OpenShift Container Platform Console. This procedure assumes APM is running on your system.
Upgrade APM to a New Version Without Using the APM Utility
The instructions in this section describes the upgrade steps for installing APM on a preexisting Kubernetes cluster of your choice. This process is a manual process and does not use the APM utility that comes with the APM installation package.
Start or Stop APM Services Using the APM Utility
Use the apm
utility script to start or stop all APM services. The
services start in order of dependency. Essential services (db and mgmt) start first,
followed by the other services. The services stop in reverse order of dependency.
To start all APM services:
content_copy zoom_out_map$ sudo -E apm start --context context-name
Note:We recommend that you use the
sudo -E apm start ––services
option to start individual services or a set of services only for troubleshooting. Use under the guidance of a Juniper Networks support representative. Use with caution as this command is like rebooting to factory-default.Note:APM starts from it's initial settings when you execute the
apm setup
command. Any persistent state is lost when theapm stop
command is executed. The current configuration can be saved using theapm save-config
command. The saved configuration is the configuration that is used the next time APM is started.To stop all APM services:
content_copy zoom_out_map$ sudo -E apm stop --context context-name WARNING Shutting down your Address Pool Manager will reset it to factory defaults and you will lose all state on the system. Shutdown will begin in 2 minutes. Please use CTRL+C to cancel.
Restart APM Services Using the Kubernetes Command Line Tool
Use the kubectl delete pods
Kubernetes command to restart APM services.
For example:
$ kubectl delete pod -n jnpr-apm jnpr-apm-addrman-758cc8885 pod "jnpr-apm-addrman-7585cc8885" deleted
To determine the pod name, you can use the kubectl get pods -n
jnpr-apm
Kubernetes command (see Check the Status of APM Services Using the Kubernetes Command Line Tool).
Setup Secrets Using the APM Utility
You can setup secrets during setup or run the sudo -E apm setup --context
context-name --secrets
to setup secrets or update
them.
$ sudo -E apm setup --context context-name --secrets APMi Secret Name (deployed: ) > APMi certificate (default: ) > ./apm.crt Copied /home/user/apm.crt to /var/local/apm/e476597324/secrets/apmi with 600 permissions APMi private key (default: ) > ./apm.key Copied /home/user/apm.keyto /var/local/apm/e476597324/secrets/apmi with 600 permissions APMi root certificate (default: ) > ./rootCA.crt Copied /home/user/rootCA.crt to /var/local/apm/e476597324/secrets/apmi with 600 permissions
If you enter a value for the secret name, you will not be asked for the key or certification files.
Display Database Information Using the APM Utility
The apm db-info
command displays current state of APM’s database
microservice including the current version, stateful set pods, and their roles.
apm db-info [--context context-name] [-o|--output
json]
$ apm db-info [--context context-name] [-o|--output json]
Version: 6.2.13
Primary: jnpr-apm-redis-0
Backup(s): jnpr-apm-redis-1
Display the Running Database Using the Kubernetes Command Line Tool
Use the kubectl exec
Kubernetes command to display the running database
(DB) to see which pod is the primary pod and to determine whether to upgrade the
persistent state database. You should run the kubectl exec
on both the
primary and secondary redis server instances. For example:
$ kubectl exec -i -n jnpr-apm jnpr-apm-redis-0 -- redis-cli -p 7380 role master 174738135 10.42.1.152 7380 174738135$
$ kubectl exec -i -n jnpr-apm jnpr-apm-redis-1 -- redis-cli -p 7380 role slave 10.42.2.22 7380 connected 174740637
Perform a Database Switchover Using the Kubernetes Command Line Tool
To force the persistent state database (DB) primary pod to switchover to an eligible backup pod, perform the following:
Check the Status of APM Services Using the APM Utility
Use the apm status
utility script to check the status of each APM
service (functional component) listed in Table 2. The status shows whether a service is running, has exited, or has not started. It also
displays the service name on the Kubernetes pod. You can compare uptime for the services
to quickly see whether any service has been restarted.
Microservice | Pod Prefix |
---|---|
addrman—Address manager | jnpr-apm-addrman |
mgmt—CLI management | jnpr-apm-mgmt |
redis (Remote Dictionary Server)—consists of a set of pods which provide the persistent database. | jnpr-apm-redis |
entman—Entity manager | jnpr-apm-entman |
provman—Provisioning manager | jnpr-apm-provman |
To check the status:
For example:
$ apm status --context context-name --detail MICROSERVICE POD STATE RESTARTS UPTIME NODE addrman jnpr-apm-addrman-7b778979b6-5vk44 Running 0 11 days, 23:25:14.629150 test-node-1 mgmt jnpr-apm-mgmt-6b4cf98d4d-hmjd9 Running 0 11 days, 23:25:33.629206 test-node-1 entman jnpr-apm-entman-7d66d89d6b-5295d Running 0 11 days, 23:25:14.629224 test-node-1 provman jnpr-apm-provman-849fb9cc4-vswm6 Running 0 11 days, 22:50:05.629258 test-node-1 redis jnpr-apm-redis-0 Running 0 11 days, 23:25:36.629275 test-node-1 redis jnpr-apm-redis-1 Running 0 11 days, 23:25:23.629290 test-node-1 redis jnpr-apm-redis-sentinels-0 Running 0 11 days, 23:25:26.629306 test-node-1 redis jnpr-apm-redis-sentinels-1 Running 0 11 days, 23:25:26.629322 test-node-1 redis jnpr-apm-redis-sentinels-2 Running 0 11 days, 23:25:26.629337 test-node-1 Storage: Healthy
$ apm status --context context-name MICROSERVICE PODS RESTARTS addrman 1/1 0 mgmt 2/2 0/0 entman 1/1 0 provman 1/1 0 redis 5/5 0/0/0/0/0 Storage: Healthy
Check the Status of APM Services Using the Kubernetes Command Line Tool
Use the Kubernetes Command Line tool to check the status of each APM service (functional component) listed in Table 2. The status shows whether a service is running, has exited, or has not started. It also displays the service name on the Kubernetes pod. You can compare uptime for the services to quickly see whether any service has been restarted.
Microservice | Pod Prefix |
---|---|
addrman—Address manager | jnpr-apm-addrman |
mgmt—CLI management | jnpr-apm-mgmt |
redis (Remote Dictionary Server)—consists of a set of pods which provide the persistent database. | jnpr-apm-redis |
entman—Entity manager | jnpr-apm-entman |
provman—Provisioning manager | jnpr-apm-provman |
To check the status, run the following command:
$ kubectl get pods -n jnpr-apm -o wide
For example:
$ kubectl get pods -n jnpr-apm -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES jnpr-apm-addrman-7585cc8885-5xr24 1/1 Running 0 11m 10.42.0.31 binnacle.englab.juniper.net <none> <none> jnpr-apm-entman-5dcf659676-4mq6g 1/1 Running 0 5d22h 10.42.2.24 jib.englab.juniper.net <none> <none> jnpr-apm-mgmt-6d7c4f47dc-2v8ss 1/1 Running 0 5d22h 10.42.0.30 binnacle.englab.juniper.net <none> <none> jnpr-apm-provman-65c66bc689-bvhb6 1/1 Running 0 5d22h 10.42.1.153 keel.englab.juniper.net <none> <none> jnpr-apm-redis-0 1/1 Running 0 5d22h 10.42.2.22 jib.englab.juniper.net <none> <none> jnpr-apm-redis-1 1/1 Running 0 5d22h 10.42.1.152 keel.englab.juniper.net <none> <none> jnpr-apm-redis-sentinels-0 1/1 Running 0 5d22h 10.42.1.151 keel.englab.juniper.net <none> <none> jnpr-apm-redis-sentinels-1 1/1 Running 0 5d22h 10.42.0.28 binnacle.englab.juniper.net <none> <none> jnpr-apm-redis-sentinels-2 1/1 Running 0 5d22h 10.42.2.23 jib.englab.juniper.net <none> <none>
Display APM IP Addresses Using the APM Utility
Use the apm ip
utility script to display the Kubernetes objects that are
necessary for the orchestration of the APM pods. For example:
$ apm ip --context context-name --detail SERVICE MICROSERVICE EXTERNAL IP INTERNAL IP PORT(S) apm-ssh mgmt 198.19.224.215 10.43.35.110 22 jnpr-apm-mgmt-svc mgmt 10.43.131.131 8066 jnpr-apm-redis-sentinels-0-svc redis 10.43.142.53 7381 jnpr-apm-redis-sentinels-1-svc redis 10.43.109.206 7381 jnpr-apm-redis-sentinels-2-svc redis 10.43.104.100 7381 jnpr-apm-redis-svc redis 10.43.6.207 7380 provman-apmi provman 198.19.224.212 10.43.221.12 20557
Display the APM IP Addresses Using the Kubernetes Command Line Tool
Use the kubectl get services
Kubernetes command to display the
Kubernetes objects that are necessary for the orchestration of the APM pods. For
example:
$ kubectl get services -n jnpr-apm | egrep "TYPE|LoadBalancer" NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE apm-apmi LoadBalancer 10.43.99.79 198.19.224.212 20557:32136/TCP 5d22h apm-ssh LoadBalancer 10.43.35.110 198.19.224.215 22:30261/TCP 146d
Display Logging Using the APM Utility
Use the apm logs
utility script to display the logs of events that
occur while APM is running. You can also use the BBE Event Collection and Visualization
utility to display file-based logs collected and stored since the time APM is started. BBE
Event Collection and Visualization is a cloud-based centralized utility that provides a way
to capture APM logs that span the life-cycle of APM micro-services. You link to the BBE
Event Collection and Visualization logging utility when you set up APM. See the Broadband Edge Event Collection and Visualization
Installation Guide.
Display APM Logging
BBE Event Collection and Visualization is a cloud-based centralized utility that provides a way to capture APM logs that span the life-cycle of APM micro-services. If you have BBE Event Collection and Visualization setup, during the APM setup, you can point BBE Event Collection and Visualization to perform the logging. BBE Event Collection and Visualization has a web-based interface to OpenSearch’s capabilities for advanced searching, aggregation, viewing, and data analysis of collected syslog events.
Use the apm logs
utility script to display the logs of events
that occur while APM is running. The event logs include events such as those
shown in the following non-exhaustive list:
Pool-domain registration events
Address allocation failures
Network entity connection failures
Startup messages
Network entity resynchronization events
Pool and partition utilization threshold and depletion warnings
By default, APM sends logs to the standard output (stdout) of the service. The output displays the circular buffer of all services or of a specified service. You can also enable logging to follow the log output of the running services. Following the log output creates an open session that continuously streams the logs to stdout.
The APM logging functions mask the underlying complexities of the kubectl
log
command that is collecting the log information. You can still
use the kubectl log
command, but that is outside the scope of
this documentation.
You can use third-party applications to capture and redirect the stdout stream for the container. Refer to your third-party documentation for assistance. You can also configure Docker with different logging drivers to redirect stdout. Refer to your Docker documentation for assistance.
To display APM logs for all services:
$ apm logs [--context contextName] [-f] [--previous] [--services SERVICES [SERVICES ...]] [--logset LOGSET] [--nocolor]
Use the apm logs ––services
option only when you are
troubleshooting under the guidance of a Juniper Networks support
representative.
To follow the logs for all services, specify the follow option
(-f
):
$ apm logs -f
To view logs from previous instances of the containers in a Kubernetes pod,
specify the previous container option (-p
):
$ apm logs -p
You can use standard Ubuntu conventions to redirect the logs to a file or to the terminal and to a file. Refer to the Ubuntu documentation for more information, but you can use the following examples as a starting point:
Redirect all logs to only a file.
content_copy zoom_out_map$ apm logs > file-path
Redirect all logs to both the screen and to a file.
content_copy zoom_out_map$ apm logs | tee file-path
Use the ––services
option only for troubleshooting under the
guidance of a Juniper Networks support representative.
You can specify any of the following severity levels, in order of increasing severity:
Severity Level | Description |
---|---|
debug | Detailed information that is typically of interest only when you are trying to diagnose a problem. These logs are often very frequent. |
info | Events or non-error conditions of interest. Logs at this level provide confirmation that everything is working as expected. These logs are generally not very frequent. |
warning | Indicates that something unexpected happened or that some problem might occur in the near future. A simple example of the latter is the disk space low warning that indicates that you might run out of disk space soon. In either case, the software is still working as expected, but you might want to monitor it more closely. These logs are generally not very frequent. |
error | Indicates that a more serious problem has prevented the software from performing some function, but the software has handled the problem as gracefully as possible to continue functioning. |
critical | A serious error that indicates that the program itself might be unable to continue running. |
You can use the --logset
option to display logs either for only
APM services or for only prebuilt services. If you do not use this option, then
only the APM services logs are displayed.
To display logs for only APM services:
$ apm logs ––logset apm $ apm logs
To display logs for prebuilt services:
$ apm logs ––logset apm-infra
Prebuilt services are services borrowed from other sources to provide infrastructure functions for APM. These sources include MGMT, redis, and redis-sentinel. For example, redis provides database and messaging services, MGMT provides a configuration/CLI service, and so on.
Display Logs Using the Kubernetes Command Line Tool
To display all the logs of events that occur while APM is running, run the following command:
$ kubectl logs -n jnpr-apm -l jnpr/logset=jnpr-apm --tail=-1
To display logs for a specific microservice, replace the label selector
(-l jnpr/logset=jnpr-apm
) with the pod name. For
example:
$ kubectl logs -n jnpr-apm microservice-pod-name --tail=-1
To determine the pod name, you can use the kubectl get pods -n
jnpr-apm
Kubernetes command (see Check the Status of APM Services Using the Kubernetes Command Line Tool).
Determine the APM Version Using the APM Utility
Use the apm version [--context <context name>] [-o|--output json]
[--detail| --compare <software-version>]
utility script to determine the
version number of the installed APM release.
To display the release version:
$ apm version --context contextName --detail Address_Pool_Manager versions: Microservice Release (version) apm: 3.3.0 addrman: 3.3.0 entman: 3.3.0 mgmt: 3.3.0 (24.2R2) provman: 3.3.0 redis: 3.3.0 (6.2.14-debian-12-r21)
To compare the specified software release versions against the current deployed release for the specified context:
apm version --compare 3.2.1 –-context myCluster components: apm: 3.2.2-2 -> 3.3.0 addrman: 3.2.2-2 -> 3.3.0 entman: 3.2.2-2 -> 3.3.0 mgmt: 3.2.2-2 -> 3.3.0 provman: 3.2.2-2 -> 3.3.0 redis: 6.2.13 -> 6.2.14-debian-12-r21
Use the -j
option to render the version information in JavaScript Object
Notation (JSON) format.
Archive the APM Configuration Using the Kubernetes Command Line Tool
To archive a copy of the currently running APM configuration, enter the following command:
$ kubectl cp jnpr-apm/$( kubectl get pods -n jnpr-apm -l jnpr/cli=cli --no-headers=true | awk '{print $1}'):config/juniper.conf.gz ./juniper.conf.gz
Uninstall and Remove APM Using the APM Utility
Use the apm
utility script to uninstall the APM configuration. The
uninstall
command reverts the actions you performed when setting up
APM. Use this command to return APM to the state it was in immediately after you installed
the application but before you did any setup configuration.
To uninstall APM:
$ sudo -E apm stop –-context context-name sudo -E apm unlink –-context context-name
After you uninstall APM, we recommend that you use the Debian uninstall procedure to remove the entire package.
$ sudo sudo -E apm clean [-h] [--log {error | warn | info | debug}] [--no-color] [--docker] [--release release-number] [--dry-run] [--uninstall] [--cluster-repos] ]
For the apm clean
command options, see the following:
-h
or--help
—Show the help message and exit.--log
or-l
{error | warn | info | debug}—Adjust the log level of the utility scrip.--no-color
—Print messages without colors.--docker
—Clean the local docker cache.--release release-number
—The release to clean (defaults to unused releases).--dry-run
—List releases or containers that will be removed.--uninstall
—Uninstall all software releases and remove APM from the system.--cluster-repos
—Clean the cluster repos for the clusters that have been removed.
Uninstall and Remove APM Without Using the APM Utility
This is the uninstall procedure that you use if you installed APM without using the APM
utility. You use the helm uninstall
command to uninstall your APM
configuration. To completely remove APM, you must run the helm uninstall
command for each microservice.
To uninstall APM, run the following:
helm uninstall --kube-context <cluster-context> -n jnpr-apm provman
helm uninstall --kube-context <cluster-context> -n jnpr-apm entman
helm uninstall --kube-context <cluster-context> -n jnpr-apm addrman
helm uninstall --kube-context <cluster-context> -n jnpr-apm mgmt
helm uninstall --kube-context <cluster-context> -n jnpr-apm redis
How to Access APM Configuration and Operational Commands Using the APM Utility
- Access the APM CLI Using the APM Utility
- Access and Use CLI Configuration Statements Using the APM Utility
- Access and Use CLI Operational Commands Using the APM Utility
Access the APM CLI Using the APM Utility
To access the CLI prompt, enter the following apm
utility script
command:
$ apm cli --context contextName [-p|--pipe]
root@jnpr-apm-mgmt
Enter a question mark to see the available top-level CLI commands. This list of commands is a subset of the Junos OS top-level commands.
root@jnpr-apm-mgmt> ? Possible completions: clear Clear information in the system configure Manipulate software configuration information file Perform file operations help Provide help information monitor Show real-time debugging information op Invoke an operation script quit Exit the management session request Make system-level requests restart Restart software process set Set CLI properties, date/time, craft interface message show Show system information ssh Start secure shell on another host start Start shell telnet Telnet to another host test Perform diagnostic debugging traceroute Trace route to remote host
For an overview of Junos OS CLI basics, see Day One: Exploring the Junos CLI. For more detailed information, see the CLI User Guide.
Access and Use CLI Configuration Statements Using the APM Utility
Access and Use CLI Operational Commands Using the APM Utility
To monitor APM, view APM configuration and statistics, or run certain operations manually:
How to Use the APM Command Line Tool Without Using the APM Utility
This section describes how you use the Kubernetes Command Line tool commands to perform administration functions.
You can use the Kubernetes Command Line tool to administer the application and to access the CLI that you use to configure the address management functions.
You can use the Kubernetes Command Line tool to do the following:
- Create and delete objects.
- Provide log access.
- Conduct interactive sessions with pod containers.
- Display the status of the APM objects.
Access the APM Command Line Tool Without Using the APM Utility
To access the APM Command Line tool using the Kubernetes commands, enter the following:
$ kubectl exec -it -n jnpr-apm $(kubectl get pods -n jnpr-apm -l jnpr/cli=cli --no-headers=true | awk '{print $1}') -- cli
Enter a question mark to see the available top-level CLI commands.
? Possible completions: clear Clear information in the system configure Manipulate software configuration information file Perform file operations help Provide help information monitor Show real-time debugging information op Invoke an operation script quit Exit the management session request Make system-level requests restart Restart software process set Set CLI properties, date/time, craft interface message show Show system information ssh Start secure shell on another host start Start shell telnet Telnet to another host test Perform diagnostic debugging traceroute Trace route to remote host