Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

 
ON THIS PAGE
 

Configuring a NorthStar Cluster for High Availability

Before You Begin

Configuring a NorthStar application cluster for high availability (HA) is an optional process. This topic describes the steps for configuring, testing, deploying, and maintaining an HA cluster. If you are not planning to use the NorthStar application HA feature, you can skip this topic.

Note:

See High Availability Overview in the NorthStar Controller User Guide for overview information about HA. For information about analytics HA, see Installing Data Collectors for Analytics.

Note:

Throughout your use of NorthStar Controller HA, be aware that you must replicate any changes you make to northstar.cfg to all cluster nodes so the configuration is uniform across the cluster. NorthStar CLI configuration changes, on the other hand, are replicated across the cluster nodes automatically.

  • Download the NorthStar Controller and install it on each server that will be part of the cluster. Each server must be completely enabled as a single node implementation before it can become part of a cluster.

    This includes:

    • Creating passwords

    • License verification steps

    • Connecting to the network for various protocol establishments such as PCEP or BGP-LS

    Note:

    All of the servers must be configured with the same database and RabbitMQ passwords.

  • All server time must be synchronized by NTP using the following procedure:

    1. Install NTP.

    2. Specify the preferred NTP server in ntp.conf.

    3. Verify the configuration.

    Note:

    All cluster nodes must have the same time zone and system time settings. This is important to prevent inconsistencies in the database storage of SNMP and LDP task collection delta values.

  • Run the net_setup.py utility to complete the required elements of the host and JunosVM configurations. Keep that configuration information available.

    Note:

    If you are using an OpenStack environment, you will have one JunosVM that corresponds to each NorthStar Controller VM.

  • Know the virtual IPv4 address you want to use for Java Planner client and web UI access to NorthStar Controller (required). This VIP address is configured for the router-facing network for single interface configurations, and for the user-facing network for dual interface configurations. This address is always associated with the active node, even if failover causes the active node to change.

  • A virtual IP (VIP) is required when setting up a NorthStar cluster. Ensure that all servers that will be in the cluster are part of the same subnet as the VIP.

  • Decide on the priority that each node will have for active node candidacy upon failover. The default value for all nodes is 0, the highest priority. If you want all nodes to have equal priority for becoming the active node, you can just accept the default value for all nodes. If you want to rank the nodes in terms of their active node candidacy, you can change the priority values accordingly—the lower the number, the higher the priority.

Set Up SSH Keys

Set up SSH keys between the selected node and each of the other nodes in the cluster, and each JunosVM.

  1. Obtain the public SSH key from one of the nodes. You will need the ssh-rsa string from the output:
  2. Copy the public SSH key from each node to each of the other nodes, from each machine.

    From node 1:

    From node 2:

    From node 3:

  3. Copy the public SSH key from the selected node to each remote JunosVM (JunosVM hosted on each other node). To do this, log in to each of the other nodes and connect to its JunosVM.

Access the HA Setup Main Menu

The /opt/northstar/utils/net_setup.py utility (the same utility you use to configure NorthStar Controller) includes an option for configuring high availability (HA) for a node cluster. Run the /opt/northstar/utils/net_setup.py utility on one of the servers in the cluster to set up the entire cluster.

  1. Select one of the nodes in the cluster on which to run the setup utility to configure all the nodes in the cluster.
  2. On the selected node, launch the NorthStar setup utility to display the NorthStar Controller Setup Main Menu.
  3. Type E and press Enter to display the HA Setup main menu.

    Figure 1 shows the top portion of the HA Setup main menu in which the current configuration is listed. It includes the five supported interfaces for each node, the VIP addresses, and the ping interval and timeout values. In this figure, only the first of the nodes is included, but you would see the corresponding information for all three of the nodes in the cluster configuration template. HA functionality requires an odd number of nodes in a cluster, and a minimum of three.

    Note:

    If you have a cRPD installation, the JunosVM information is not displayed as it is not applicable.

    Figure 1: HA Setup Main Menu, Top PortionHA Setup Main Menu, Top Portion
    Note:

    If you are configuring a cluster for the first time, the IP addresses are blank and other fields contain default values. If you are modifying an existing configuration, the current cluster configuration is displayed, and you have the opportunity to change the values.

    Note:

    If the servers are located in geodiverse locations, you can use Site Name to indicate which servers are in the same or different geographical locations.

    Figure 2 shows the lower portion of the HA Setup main menu. To complete the configuration, you type the number or letter of an option and provide the requested information. After each option is complete, you are returned to the HA Setup main menu so you can select another option.

    Figure 2: HA Setup Main Menu, Lower PortionHA Setup Main Menu, Lower Portion
    Note:

    If you have a cRPD installation, options 3, 4, and 8 are not displayed as they are not applicable. The remaining options are not renumbered.

Configure the Three Default Nodes and Their Interfaces

The HA Setup main menu initially offers three nodes for configuration because a cluster must have a minimum of three nodes. You can add more nodes as needed.

For each node, the menu offers five interfaces. Configure as many of those as you need.

  1. Type 5 and press Enter to modify the first node.
  2. When prompted, enter the number of the node to be modified, the hostname, the site name, and the priority, pressing Enter between entries.
    Note:

    The NorthStar Controller uses root as a username to access other nodes.

    The default priority is 0. You can just press Enter to accept the default or you can type a new value.

    For each interface, enter the interface name, IPv4 address, and switchover (yes/no), pressing Enter between entries.

    Note:

    For each node, interface #1 is reserved for the cluster communication interface which is used to facilitate communication between nodes. For this interface, it is required that switchover be set to Yes, and you cannot change that parameter.

    When finished, you are returned to the HA Setup main menu.

    The following example configures Node #1 and two of its available five interfaces.

  3. Type 5 and press Enter again to repeat the data entry for each of the other two nodes.

Configure the JunosVM for Each Node

To complete the node-specific setup, configure the JunosVM for each node in the cluster.

  1. From the HA Setup main menu, type 8 and press Enter to modify the JunosVM for a node.
  2. When prompted, enter the node number, the JunosVM hostname, and the JunosVM IPv4 address, pressing Enter between entries.

    Figure 3 shows these JunosVM setup fields.

    Figure 3: Node 1 JunosVM Setup FieldsNode 1 JunosVM Setup Fields

    When finished, you are returned to the HA Setup main menu.

  3. Type 8 and press Enter again to repeat the JunosVM data entry for each of the other two nodes.

(Optional) Add More Nodes to the Cluster

If you want to add additional nodes, type 1 and press Enter. Then configure the node and the node’s JunosVM using the same procedures previously described. Repeat the procedures for each additional node.

Note:

HA functionality requires an odd number of nodes and a minimum of three nodes per cluster.

The following example shows adding an additional node, node #4, with two interfaces.

The following example shows configuring the JunosVM that corresponds to node #4.

Configure Cluster Settings

The remaining settings apply to the cluster as a whole.

  1. From the HA Setup main menu, type 9 and press Enter to configure the VIP address for the external (router-facing) network. This is the virtual IP address that is always associated with the active node, even if failover causes the active node to change. The VIP is required, even if you are configuring a separate user-facing network interface. If you have upgraded from an earlier NorthStar release in which you did not have VIP for external0, you must now configure it.
    Note:

    Make a note of this IP address. If failover occurs while you are working in the NorthStar Planner UI, the client is disconnected and you must re-launch it using this VIP address. For the NorthStar Controller web UI, you would be disconnected and would need to log back in.

    The following example shows configuring the VIP address for the external network.

  2. Type 9 and press Enter to configure the VIP address for the user-facing network for dual interface configurations. If you do not configure this IP address, the router-facing VIP address also functions as the user-facing VIP address.
  3. Type D and press Enter to configure the setup mode as cluster (local cluster).
  4. Type E and press Enter to configure the PCEP session. The default is physical_ip. If you are using the cluster VIP for your PCEP session, configure the PCEP session as vip.
    Note:

    All of your PCC sessions must use either physical IP or VIP (no mixing and matching), and that must also be reflected in the PCEP configuration on the router.

Test and Deploy the HA Configuration

You can test and deploy the HA configuration from within the HA Setup main menu.

  1. Type G to test the HA connectivity for all the interfaces. You must verify that all interfaces are up before you deploy the HA cluster.
  2. Type H and press Enter to launch a script that connects to and deploys all the servers and all the JunosVMs in the cluster. The process takes approximately 15 minutes, after which the display is returned to the HA Setup menu. You can view the log of the progress at /opt/northstar/logs/net_setup.log.
    Note:

    If the execution has not completed within 30 minutes, a process might be stuck. You can sometimes see this by examining the log at /opt/northstar/logs/net_setup.log. You can press Ctrl-C to cancel the script, and then restart it.

  3. To check if the election process has completed, examine the processes running on each node by logging into the node and executing the supervisorctl status script.

    For the active node, you should see all processes listed as RUNNING as shown here.

    Note:

    The actual list of processes depends on the version of NorthStar and your deployment setup.

    For a standby node, processes beginning with “northstar”and “northstar_pcs” should be listed as STOPPED. Also, if you have analytics installed, some of the processes beginning with “collector” are STOPPED. Other processes, including those needed to preserve connectivity, remain RUNNING. An example is shown here.

    Note:

    This is just an example; the actual list of processes depends on the version of NorthStar, your deployment setup, and the optional features you have installed.

  4. Set the web UI admin password using either the web UI or net_setup.

    • For the web UI method, use the external IP address that was provided to you when you installed the NorthStar application. Type that address into the address bar of your browser (for example, https://10.0.1.29:8443). A window is displayed requesting the confirmation code in your license file (the characters after S-NS-SDN=), and the password you wish to use. See Figure 4.

      Figure 4: Web UI Method for Setting the Web UI Password Web UI Method for Setting the Web UI Password

    • For the net_setup method, select D from the net_setup Main Menu (Maintenance & Troubleshooting), and then 3 from the Maintenance & Troubleshooting menu (Change UI Admin Password).

      Type Y to confirm you wish to change the UI Admin password, and enter the new password when prompted.

  5. Once the web UI admin password has been set, return to the HA Setup menu (select E from the Main Menu). View cluster information and check the cluster status by typing K, and pressing Enter. In addition to providing general cluster information, this option launches the ns_check_cluster.sh script. You can also run this script outside of the setup utility by executing the following commands:

Replace a Failed Node if Necessary

On the HA Setup menu, options I and J can be used when physically replacing a failed node. They allow you to replace a node without having to redeploy the entire cluster which would wipe out all the data in the database.

CAUTION:

While a node is being replaced in a three-node cluster, HA is not guaranteed.

  1. Replace the physical node in the network and install NorthStar Controller on the replacement node.
  2. Run the NorthStar setup utility to configure the replaced node with the necessary IP addresses. Be sure you duplicate the previous node setup, including:

    • IP address and hostname

    • Initialization of credentials

    • Licensing

    • Network connectivity

  3. Go to one of the existing cluster member nodes (preferably the same node that was used to configure the HA cluster initially). Going forward, we will refer to this node as the anchor node.
  4. Set up the SSH key from the anchor node to the replacement node and JunosVM.

    Copy the public SSH key from the anchor node to the replacement node, from the replacement node to the other cluster nodes, and from the other cluster nodes to the replacement node.

    Note:

    Remember that in your initial HA setup, you had to copy the public SSH key from each node to each of the other nodes, from each machine.

    Copy the public SSH key from the anchor node to the replacement node’s JunosVM (the JunosVM hosted on each of the other nodes). To do this, log in to each of the replacement nodes and connect to its JunosVM.

  5. From the anchor node, remove the failed node from the Cassandra database. Run the command nodetool removenode host-id. To check the status, run the command nodetool status.

    The following example shows removing the failed node with IP address 10.25.153.10.

  6. From the HA Setup menu on the anchor node, select option I to copy the HA configuration to the replacement node.
  7. From the HA Setup menu on the anchor node, select option J to deploy the HA configuration, only on the replacement node.

Configure Fast Failure Detection Between JunosVM and PCC

You can use Bidirectional Forward Detection (BFD) in deploying the NorthStar application to provide faster failure detection as compared to BGP or IGP keepalive and hold timers. The BFD feature is supported in PCC and JunosVM.

To utilize this feature, configure bfd-liveness-detection minimum-interval milliseconds on the PCC, and mirror this configuration on the JunosVM. We recommend a value of 1000 ms or higher for each cluster node. Ultimately, the appropriate BFD value depends on your requirements and environment.

Related Documentation