Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Announcement: Try the Ask AI chatbot for answers to your technical questions about Juniper products and solutions.

close
external-header-nav
Documentation
Menu
License Guide
Quick Starts
Validated Designs
Home Documentation Junos Space Network Management Platform Monitoring and Troubleshooting Guide Log Files and Debug Utilities Troubleshooting Network Devices by Using Junos Space Debug Utilties

Monitoring and Troubleshooting Guide

keyboard_arrow_up
close
keyboard_arrow_left
Monitoring and Troubleshooting Guide
Table of Contents Expand all
list Table of Contents
file_download PDF
{ "lLangCode": "en", "lName": "English", "lCountryCode": "us", "transcode": "en_US" }
English
ON THIS PAGE
keyboard_arrow_right

Executing Job Management Scripts and Java Applications

date_range 06-May-21
arrow_backward
arrow_forward

You execute job management scripts and Java applications to view information about jobs triggered from Junos Space nodes. The JobInfoCollector.jar Java application and the job management scripts are stored at the following location: /var/log/space-debug/debug-utilities/jobManagement. When you execute the JobInfoCollector.jar Java application and job management scripts, the output is stored as .txt files at the same location. You can also specify a custom path to store the output. The following sections list the steps to execute the scripts to collect information about jobs.

Executing the Java Application to Collect Job Information

You execute the JobInfoCollector.jar application to collect job information. Ensure that you have updated the MySQL query in the JobInfoCollector.properties file before executing the Java application. The file also contains an example configuration.

To execute the Java application to collect job information:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

    The following is a sample output from a virtual appliance:

    content_copy zoom_out_map
    admin@10.206.41.183's password:
Last login: Mon Aug 17 06:17:58 2015 from 10.206.41.42

Welcome to the Junos Space network settings utility.

Initializing, please wait


Junos Space Settings Menu

1> Change Password
2> Change Network Settings
3> Change Time Options
4> Retrieve Logs
5> Security
6> Expand VM Drive Size
7> (Debug) run shell

A> Apply changes
Q> Quit
R> Redraw Menu

Choice [1-7,AQR]: 7
  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, enter ls and press Enter.
  7. Type /usr/bin/java -jar JobInfoCollector.jar and press Enter.

    The following options will be displayed:

    content_copy zoom_out_map
    Please select an option from below:
1. Display job parent-children hierarchy
2. Filter from job data
3. Display jobs not yet picked up by Job Dispatcher
  8. Type 1, 2, or 3 to select an option.
  9. Enter the database username.
  10. Enter the database password.
  11. The inputs displayed depend on the option you selected.

    If you selected option 1, you need to enter the parent job ID.

    1. Enter a job ID.

      The output from this command is saved to the JobHierarchy-<job-ID>.txt file in the same directory. The following is a sample output if you selected 1:

      content_copy zoom_out_map
      =====Job details for 491529 =====

NODE: 10.206.41.57
STARTTIMESTAMP: 17 Jul 2015 17:51:00 GMT
JOBSTATE: Done
JOBSTATUS: Success
ENDTIMESTAMP: 17 Jul 2015 17:52:15 GMT
CHILDCOUNT: 2

 Child job details:

ID || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP || CHILDCOUNT ||
 491530 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |
 491531 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |

    If you selected option 2, you need to enter a MySQL query.

    1. Enter a MySQL query—for example, mostate=’Failure’.

      This query filters the jobs that failed. The following is a sample output if you selected 2:

      content_copy zoom_out_map
      ID || NAME || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP ||
 196608 | Discover Network Elements-196608 | 10.206.41.184 | 10 Aug 2015 10:44:48 GMT | Done | Failure | 10 Aug 2015 10:48:47 GMT |
 196624 | Discover Network Elements-196624 | 10.206.41.184 | 10 Aug 2015 10:49:52 GMT | Done | Failure | 10 Aug 2015 10:49:58 GMT |
 196643 | Mail User Password-196643 | 10.206.41.184 | 10 Aug 2015 14:11:21 GMT | Done | Failure | 10 Aug 2015 14:11:21 GMT |
 196649 | Mail User Password-196649 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
 196650 | Mail User Password-196650 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
 196660 | Sync Files-196660 | 10.206.41.187 | 11 Aug 2015 11:54:13 GMT | Done | Failure | 11 Aug 2015 11:54:46 GMT |
 360448 | Resync Network Elements-360448 | 10.206.41.184 | 11 Aug 2015 18:06:07 GMT | Done | Failure | 11 Aug 2015 18:06:08 GMT |
 360449 | Discover Network Elements-360449 | 10.206.41.187 | 11 Aug 2015 18:12:43 GMT | Done | Failure | 11 Aug 2015 18:13:16 GMT |

    If you selected option 3, the list of unscheduled jobs is saved to the UnscheduledJobs.txt file in the same directory.

    1. The following is a sample output if you selected 3:
      content_copy zoom_out_map
      ID || NAME || JOBSTATE || PARENTJOBID ||
 393501 | ND Discovery-393501 | Done | 0 |
 393858 | Backup Configuration Files-393858 | Done | 0 |
 720906 | ND Discovery-720906 | Done | 0 |
 721067 | Backup Configuration Files-721067 | Done | 0 |
 721884 | Backup Configuration Files-721884 | Done | 0 |
 1048659 | Backup Configuration Files-1048659 | Done | 0 |
 1182260 | Enable Script-1182260 | InProgress | 0 |
 1182413 | Execute Script-1182413 | InProgress | 0 |
 1183914 | Backup Configuration Files-1183914 | Done | 0 |
 1184310 | Delete Device-1184310 | Done | 0 |
 1189992 | Stage Script-1189992 | InProgress | 0 |
 1190105 | Update Network Element-1190105 | InProgress | 0 |
 1442231 | Backup Configuration Files-1442231 | Done | 0 |
 1442232 | -1442232 | InProgress | 1184310 |
 1442233 | -1442233 | InProgress | 1184310 |
 1442234 | -1442234 | InProgress | 1184310 |
 1442235 | -1442235 | InProgress | 1184310 |
 1442236 | -1442236 | InProgress | 1184310 |
 1442237 | -1442237 | InProgress | 1184310 |
 1442238 | -1442238 | InProgress | 1184310 |
 1442239 | -1442239 | InProgress | 1184310 |
 1442240 | -1442240 | InProgress | 1184310 |
 1445400 | Generate SD LR Report-1445400 | Scheduled | 0 |
 1445528 | Cloud Infrastructure Event Purge-1445528 | Scheduled | 0 |
 1445602 | Policy Hits Collection-1445602 | Scheduled | 0 |
 1903749 | useraccounts-1903749 | Done | 0 |

You can view the hierarchy of a subjob (input as the parent job ID), list of jobs that are currently unscheduled, or the output based on the query you entered in the JobInfoCollector.properties file.

Executing the Script to View the Stack Trace of a Job

You execute the getJobThreadDump.sh script to view the stack trace of a job.

To execute the script to view the stack trace of a job:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
  7. Type ./getJobThreadDump.sh <job-ID> and press Enter.

    The output from this command is saved to the JobThreadDump-<job-ID>.txt file in the same directory.

  8. (Optional) To save the stack trace of the job at a custom output location, type ./getJobThreadDump.sh <job-ID> <output-file-path> and press Enter.

Executing the Script to View Job Information on Nodes

You execute the SystemLoadViewer.sh script to view job information on a node.

To execute the script to view job information on a node:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
  7. Type ./SystemLoadViewer.sh and press Enter.

    The output from this command is saved to the ./SystemLoadInfo.txt file in the same directory. The following is a sample output from the command:

    content_copy zoom_out_map
    ------------------------------------------------------------
Time of execution: Mon Jul 20 06:54:14 UTC 2015
------------------------------------------------------------

=====================NODE SUMMARY======================
Node IP: 10.206.41.35
Maximum JVM memory: 2040528896
Estimated memory usage per node: 40465
Reserved memory size: 20000000
(Available Memory = Maximum JVM memor - Estimated memory usage per node - Reserved memory)
Available Memory:2020488431
Number of first root jobs on this node:1
Number of next root jobs on this node:0
Number of sub jobs on this node:0
-------------------------------------------
===========================================
Node load statistics:
===========================================
Resource Id:autoresync-group:132832:user1@host:0360471000000000008G
Type:autoresync-group
SubType:132832
Context type:FIRST_ROOT_JOB
State:RUNNING
Queue:
Node IP:10.206.41.35
Creation Time:2015-07-20 06:54:13.166
Last modification time:2015-07-20 06:54:13.166
Estimated Memory:40465
===========================================
============================SYSTEM LOAD SUMMARY FOR LOCAL NODE=============================
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/jmpvgnocf-lvvar
                      52772604   2584204  47373632   6% /var
======Command executed: LINES=20 COLUMNS=120 top -n 2 -b -c | tail -n +22==================
top - 06:54:18 up 3 days, 19:01,  2 users,  load average: 1.24, 0.88, 0.82
Tasks: 271 total,   2 running, 268 sleeping,   0 stopped,   1 zombie
Cpu(s): 42.2%us, 23.5%sy,  0.5%ni, 32.7%id,  0.3%wa,  0.1%hi,  0.7%si,  0.0%st
Mem:   7937672k total,  7827836k used,   109836k free,   539116k buffers
Swap:  8193140k total,   132684k used,  8060456k free,  1246572k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30383 user1      20   0 83616  24m 2508 S 36.8  0.3   0:02.16 /usr/bin/perl /var/www/cgi-bin/secure/resourceMonitoring
 4204 jboss     20   0 4081m 2.7g  14m S 27.6 36.0 395:59.73 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
31240 user1      20   0  124m  15m 2448 S  7.6  0.2   0:00.23 perl -e use lib("/usr/nma/lib"); use NmaUtil; print NmaUtil
31251 user1      20   0 72016  14m 2384 R  6.9  0.2   0:00.21 perl -e use lib("/usr/nma/lib"); use NmaDb; print NmaDb::ge
30719 mysql     16  -4 2936m 744m 6628 S  3.6  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
32329 opennms   25   5 3446m 877m  21m S  3.0 11.3 164:35.59 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
 3120 user1      20   0 84612  21m 2460 S  1.0  0.3  13:51.23 /usr/bin/perl /usr/bin/jmp-sync
30651 user1      20   0 12896 1436  952 R  1.0  0.0   0:00.07 top -n 2 -b -c
13332 user1      20   0  154m 9824 4480 S  0.7  0.1  26:56.92 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
18186 postgres  20   0  268m  12m 9252 S  0.7  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
 2589 jboss     20   0  924m 197m  10m S  0.3  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
15335 apache    23   3  141m 4936 3012 S  0.3  0.1   0:00.62 /usr/sbin/httpd
======Command executed: COLUMNS=120 top -n 1 -b -c | egrep '(mysqld|java|postgres)'========
 4204 jboss     20   0 4081m 2.7g  14m S  1.9 36.0 395:59.74 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
32329 opennms   25   5 3446m 877m  21m S  1.9 11.3 164:35.60 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
 2556 jboss     20   0  527m  47m 9800 S  0.0  0.6   4:44.43 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
 2589 jboss     20   0  924m 197m  10m S  0.0  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
 2886 postgres  20   0  268m  10m 7524 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55289) idle
 2888 postgres  20   0  268m  10m 7580 S  0.0  0.1   0:00.08 postgres: opennms opennms 127.0.0.1(55290) idle
 2889 postgres  20   0  268m  10m 8040 S  0.0  0.1   0:00.15 postgres: opennms opennms 127.0.0.1(55291) idle
 2891 postgres  20   0  268m  12m 9352 S  0.0  0.2   0:00.31 postgres: opennms opennms 127.0.0.1(55292) idle
 2893 postgres  20   0  267m  10m 7612 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55293) idle
 2895 postgres  20   0  267m 8968 6548 S  0.0  0.1   0:00.03 postgres: opennms opennms 127.0.0.1(55294) idle
 3507 jboss     20   0  528m  47m 9856 S  0.0  0.6   4:51.80 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
 3561 jboss     20   0  566m 143m  10m S  0.0  1.8   5:43.77 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
 5570 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36280) idle
 5572 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36281) idle
 5573 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36282) idle
 5575 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36283) idle
 5578 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36284) idle
 5579 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36285) idle
 5581 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36286) idle
 5583 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36287) idle
 5586 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36288) idle
 5587 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36289) idle
 5590 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36290) idle
 5591 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36291) idle
 5593 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36292) idle
 5595 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36293) idle
 5597 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36294) idle
 5599 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36295) idle
 5601 postgres  20   0  268m  10m 8172 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36296) idle
 5603 postgres  20   0  268m  10m 7376 S  0.0  0.1   0:00.07 postgres: opennms opennms 127.0.0.1(36297) idle
 5605 postgres  20   0  267m 9.8m 7228 S  0.0  0.1   0:00.05 postgres: opennms opennms 127.0.0.1(36298) idle
 5607 postgres  20   0  267m 9972 7132 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36299) idle
 5609 postgres  20   0  268m  10m 8252 S  0.0  0.1   0:00.20 postgres: opennms opennms 127.0.0.1(36300) idle
 5611 postgres  20   0  268m  11m 8848 S  0.0  0.2   0:00.21 postgres: opennms opennms 127.0.0.1(36301) idle
 8123 postgres  20   0  268m  10m 8160 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(56849) idle
17000 user1      20   0 2299m  68m 8072 S  0.0  0.9   6:32.60 ./jre/bin/java -Djava.compiler=NONE -cp /usr/StorMan/RaidMa
18186 postgres  20   0  268m  12m 9256 S  0.0  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
18187 postgres  20   0  268m  11m 8800 S  0.0  0.2   0:00.18 postgres: opennms opennms 127.0.0.1(55331) idle
18188 postgres  20   0  268m  10m 7484 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55332) idle
18190 postgres  20   0  268m  10m 7800 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55333) idle
20287 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36339) idle
20289 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36340) idle
20291 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36341) idle
20297 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36342) idle
20298 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36343) idle
20301 postgres  20   0  267m 7448 5228 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36344) idle
20306 postgres  20   0  267m 7804 5548 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36345) idle
20308 postgres  20   0  267m 9660 6872 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36346) idle
20311 postgres  20   0  268m  10m 7692 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36347) idle
22848 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56892) idle
22850 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56893) idle
22852 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56894) idle
22858 postgres  20   0  266m 6388 4312 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56895) idle
22860 postgres  20   0  268m  10m 7452 S  0.0  0.1   0:00.11 postgres: opennms opennms 127.0.0.1(56896) idle
22863 postgres  20   0  268m  10m 7968 S  0.0  0.1   0:00.22 postgres: opennms opennms 127.0.0.1(56897) idle
22864 postgres  20   0  267m 7608 5368 S  0.0  0.1   0:00.02 postgres: opennms opennms 127.0.0.1(56898) idle
22866 postgres  20   0  268m  10m 7528 S  0.0  0.1   0:00.13 postgres: opennms opennms 127.0.0.1(56899) idle
29715 user1      16  -4 13052 1068 1064 S  0.0  0.0   0:00.11 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid
30719 mysql     16  -4 2936m 744m 6628 S  0.0  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
31064 postgres  20   0  265m  17m  16m S  0.0  0.2   0:32.35 /usr/pgsql-9.4/bin/postmaster -p 5432 -D /var/lib/pgsql/9.4
31072 postgres  20   0  119m 1728  868 S  0.0  0.0   0:03.28 postgres: logger process
31074 postgres  20   0  265m  42m  41m S  0.0  0.5   0:39.87 postgres: checkpointer process
31075 postgres  20   0  265m 7304 6368 S  0.0  0.1   0:05.45 postgres: writer process
31076 postgres  20   0  265m 6076 5156 S  0.0  0.1   0:53.84 postgres: wal writer process
31077 postgres  20   0  265m 2688 1496 S  0.0  0.0   0:12.46 postgres: autovacuum launcher process
31078 postgres  20   0  119m 2056  924 S  0.0  0.0   0:58.40 postgres: stats collector process
31304 user1      20   0 61220  768  664 S  0.0  0.0   0:00.00 egrep (mysqld|java|postgres)
  8. (Optional) To save job information about a node at a custom output location, enter ./SystemLoadViewer.sh <output-file-path>.

You can view information such as the memory on the nodes, number of root jobs and subjobs on each of the nodes, and so on.

Related Documentation

arrow_backward PREVIOUS Executing Device Import Detail Script and Java Application
NEXT arrow_forward Executing HornetQ Scripts
external-footer-nav