Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Announcement: Try the Ask AI chatbot for answers to your technical questions about Juniper products and solutions.

close
external-header-nav
keyboard_arrow_up
close
keyboard_arrow_left
Monitoring and Troubleshooting Guide
Table of Contents Expand all
list Table of Contents
file_download PDF
{ "lLangCode": "en", "lName": "English", "lCountryCode": "us", "transcode": "en_US" }
English
keyboard_arrow_right

Executing Job Management Scripts and Java Applications

date_range 06-May-21

You execute job management scripts and Java applications to view information about jobs triggered from Junos Space nodes. The JobInfoCollector.jar Java application and the job management scripts are stored at the following location: /var/log/space-debug/debug-utilities/jobManagement. When you execute the JobInfoCollector.jar Java application and job management scripts, the output is stored as .txt files at the same location. You can also specify a custom path to store the output. The following sections list the steps to execute the scripts to collect information about jobs.

Executing the Java Application to Collect Job Information

You execute the JobInfoCollector.jar application to collect job information. Ensure that you have updated the MySQL query in the JobInfoCollector.properties file before executing the Java application. The file also contains an example configuration.

To execute the Java application to collect job information:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

    The following is a sample output from a virtual appliance:

    content_copy zoom_out_map
    admin@10.206.41.183's password:
    Last login: Mon Aug 17 06:17:58 2015 from 10.206.41.42
    
    Welcome to the Junos Space network settings utility.
    
    Initializing, please wait
    
    
    Junos Space Settings Menu
    
    1> Change Password
    2> Change Network Settings
    3> Change Time Options
    4> Retrieve Logs
    5> Security
    6> Expand VM Drive Size
    7> (Debug) run shell
    
    A> Apply changes
    Q> Quit
    R> Redraw Menu
    
    Choice [1-7,AQR]: 7
  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, enter ls and press Enter.
  7. Type /usr/bin/java -jar JobInfoCollector.jar and press Enter.

    The following options will be displayed:

    content_copy zoom_out_map
    Please select an option from below:
    1. Display job parent-children hierarchy
    2. Filter from job data
    3. Display jobs not yet picked up by Job Dispatcher
    
  8. Type 1, 2, or 3 to select an option.
  9. Enter the database username.
  10. Enter the database password.
  11. The inputs displayed depend on the option you selected.

    If you selected option 1, you need to enter the parent job ID.

    1. Enter a job ID.

      The output from this command is saved to the JobHierarchy-<job-ID>.txt file in the same directory. The following is a sample output if you selected 1:

      content_copy zoom_out_map
      =====Job details for 491529 =====
      
      NODE: 10.206.41.57
      STARTTIMESTAMP: 17 Jul 2015 17:51:00 GMT
      JOBSTATE: Done
      JOBSTATUS: Success
      ENDTIMESTAMP: 17 Jul 2015 17:52:15 GMT
      CHILDCOUNT: 2
      
       Child job details:
      
      ID || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP || CHILDCOUNT ||
       491530 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |
       491531 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |

    If you selected option 2, you need to enter a MySQL query.

    1. Enter a MySQL query—for example, mostate=’Failure’.

      This query filters the jobs that failed. The following is a sample output if you selected 2:

      content_copy zoom_out_map
      ID || NAME || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP ||
       196608 | Discover Network Elements-196608 | 10.206.41.184 | 10 Aug 2015 10:44:48 GMT | Done | Failure | 10 Aug 2015 10:48:47 GMT |
       196624 | Discover Network Elements-196624 | 10.206.41.184 | 10 Aug 2015 10:49:52 GMT | Done | Failure | 10 Aug 2015 10:49:58 GMT |
       196643 | Mail User Password-196643 | 10.206.41.184 | 10 Aug 2015 14:11:21 GMT | Done | Failure | 10 Aug 2015 14:11:21 GMT |
       196649 | Mail User Password-196649 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
       196650 | Mail User Password-196650 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT |
       196660 | Sync Files-196660 | 10.206.41.187 | 11 Aug 2015 11:54:13 GMT | Done | Failure | 11 Aug 2015 11:54:46 GMT |
       360448 | Resync Network Elements-360448 | 10.206.41.184 | 11 Aug 2015 18:06:07 GMT | Done | Failure | 11 Aug 2015 18:06:08 GMT |
       360449 | Discover Network Elements-360449 | 10.206.41.187 | 11 Aug 2015 18:12:43 GMT | Done | Failure | 11 Aug 2015 18:13:16 GMT |

    If you selected option 3, the list of unscheduled jobs is saved to the UnscheduledJobs.txt file in the same directory.

    1. The following is a sample output if you selected 3:
      content_copy zoom_out_map
      ID || NAME || JOBSTATE || PARENTJOBID ||
       393501 | ND Discovery-393501 | Done | 0 |
       393858 | Backup Configuration Files-393858 | Done | 0 |
       720906 | ND Discovery-720906 | Done | 0 |
       721067 | Backup Configuration Files-721067 | Done | 0 |
       721884 | Backup Configuration Files-721884 | Done | 0 |
       1048659 | Backup Configuration Files-1048659 | Done | 0 |
       1182260 | Enable Script-1182260 | InProgress | 0 |
       1182413 | Execute Script-1182413 | InProgress | 0 |
       1183914 | Backup Configuration Files-1183914 | Done | 0 |
       1184310 | Delete Device-1184310 | Done | 0 |
       1189992 | Stage Script-1189992 | InProgress | 0 |
       1190105 | Update Network Element-1190105 | InProgress | 0 |
       1442231 | Backup Configuration Files-1442231 | Done | 0 |
       1442232 | -1442232 | InProgress | 1184310 |
       1442233 | -1442233 | InProgress | 1184310 |
       1442234 | -1442234 | InProgress | 1184310 |
       1442235 | -1442235 | InProgress | 1184310 |
       1442236 | -1442236 | InProgress | 1184310 |
       1442237 | -1442237 | InProgress | 1184310 |
       1442238 | -1442238 | InProgress | 1184310 |
       1442239 | -1442239 | InProgress | 1184310 |
       1442240 | -1442240 | InProgress | 1184310 |
       1445400 | Generate SD LR Report-1445400 | Scheduled | 0 |
       1445528 | Cloud Infrastructure Event Purge-1445528 | Scheduled | 0 |
       1445602 | Policy Hits Collection-1445602 | Scheduled | 0 |
       1903749 | useraccounts-1903749 | Done | 0 |

You can view the hierarchy of a subjob (input as the parent job ID), list of jobs that are currently unscheduled, or the output based on the query you entered in the JobInfoCollector.properties file.

Executing the Script to View the Stack Trace of a Job

You execute the getJobThreadDump.sh script to view the stack trace of a job.

To execute the script to view the stack trace of a job:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
  7. Type ./getJobThreadDump.sh <job-ID> and press Enter.

    The output from this command is saved to the JobThreadDump-<job-ID>.txt file in the same directory.

  8. (Optional) To save the stack trace of the job at a custom output location, type ./getJobThreadDump.sh <job-ID> <output-file-path> and press Enter.

Executing the Script to View Job Information on Nodes

You execute the SystemLoadViewer.sh script to view job information on a node.

To execute the script to view job information on a node:

  1. Log in to the CLI of the Junos Space node.
  2. Enter the administrator username and password at the Junos Space login prompt and press Enter.

    The default username is admin and the default password is abc123.

    The Junos Space Settings Menu is displayed.

  3. Enter 6 (if you are using a hardware appliance) or 7 (if you are using a virtual appliance) at the Junos Space Settings Menu prompt to run shell commands.

    You are prompted to enter your password.

  4. Type the administrator password and press Enter.

    The shell prompt appears.

  5. Type cd /var/log/space-debug/debug-utilities/jobManagement at the shell prompt and press Enter.
  6. (Optional) To view the list of job management scripts and Java applications, type ls and press Enter.
  7. Type ./SystemLoadViewer.sh and press Enter.

    The output from this command is saved to the ./SystemLoadInfo.txt file in the same directory. The following is a sample output from the command:

    content_copy zoom_out_map
    ------------------------------------------------------------
    Time of execution: Mon Jul 20 06:54:14 UTC 2015
    ------------------------------------------------------------
    
    =====================NODE SUMMARY======================
    Node IP: 10.206.41.35
    Maximum JVM memory: 2040528896
    Estimated memory usage per node: 40465
    Reserved memory size: 20000000
    (Available Memory = Maximum JVM memor - Estimated memory usage per node - Reserved memory)
    Available Memory:2020488431
    Number of first root jobs on this node:1
    Number of next root jobs on this node:0
    Number of sub jobs on this node:0
    -------------------------------------------
    ===========================================
    Node load statistics:
    ===========================================
    Resource Id:autoresync-group:132832:user1@host:0360471000000000008G
    Type:autoresync-group
    SubType:132832
    Context type:FIRST_ROOT_JOB
    State:RUNNING
    Queue:
    Node IP:10.206.41.35
    Creation Time:2015-07-20 06:54:13.166
    Last modification time:2015-07-20 06:54:13.166
    Estimated Memory:40465
    ===========================================
    ============================SYSTEM LOAD SUMMARY FOR LOCAL NODE=============================
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/mapper/jmpvgnocf-lvvar
                          52772604   2584204  47373632   6% /var
    ======Command executed: LINES=20 COLUMNS=120 top -n 2 -b -c | tail -n +22==================
    top - 06:54:18 up 3 days, 19:01,  2 users,  load average: 1.24, 0.88, 0.82
    Tasks: 271 total,   2 running, 268 sleeping,   0 stopped,   1 zombie
    Cpu(s): 42.2%us, 23.5%sy,  0.5%ni, 32.7%id,  0.3%wa,  0.1%hi,  0.7%si,  0.0%st
    Mem:   7937672k total,  7827836k used,   109836k free,   539116k buffers
    Swap:  8193140k total,   132684k used,  8060456k free,  1246572k cached
    
      PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    30383 user1      20   0 83616  24m 2508 S 36.8  0.3   0:02.16 /usr/bin/perl /var/www/cgi-bin/secure/resourceMonitoring
     4204 jboss     20   0 4081m 2.7g  14m S 27.6 36.0 395:59.73 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
    31240 user1      20   0  124m  15m 2448 S  7.6  0.2   0:00.23 perl -e use lib("/usr/nma/lib"); use NmaUtil; print NmaUtil
    31251 user1      20   0 72016  14m 2384 R  6.9  0.2   0:00.21 perl -e use lib("/usr/nma/lib"); use NmaDb; print NmaDb::ge
    30719 mysql     16  -4 2936m 744m 6628 S  3.6  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
    32329 opennms   25   5 3446m 877m  21m S  3.0 11.3 164:35.59 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
     3120 user1      20   0 84612  21m 2460 S  1.0  0.3  13:51.23 /usr/bin/perl /usr/bin/jmp-sync
    30651 user1      20   0 12896 1436  952 R  1.0  0.0   0:00.07 top -n 2 -b -c
    13332 user1      20   0  154m 9824 4480 S  0.7  0.1  26:56.92 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a
    18186 postgres  20   0  268m  12m 9252 S  0.7  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
     2589 jboss     20   0  924m 197m  10m S  0.3  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
    15335 apache    23   3  141m 4936 3012 S  0.3  0.1   0:00.62 /usr/sbin/httpd
    ======Command executed: COLUMNS=120 top -n 1 -b -c | egrep '(mysqld|java|postgres)'========
     4204 jboss     20   0 4081m 2.7g  14m S  1.9 36.0 395:59.74 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav
    32329 opennms   25   5 3446m 877m  21m S  1.9 11.3 164:35.60 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor
     2556 jboss     20   0  527m  47m 9800 S  0.0  0.6   4:44.43 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
     2589 jboss     20   0  924m 197m  10m S  0.0  2.5   6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
     2886 postgres  20   0  268m  10m 7524 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55289) idle
     2888 postgres  20   0  268m  10m 7580 S  0.0  0.1   0:00.08 postgres: opennms opennms 127.0.0.1(55290) idle
     2889 postgres  20   0  268m  10m 8040 S  0.0  0.1   0:00.15 postgres: opennms opennms 127.0.0.1(55291) idle
     2891 postgres  20   0  268m  12m 9352 S  0.0  0.2   0:00.31 postgres: opennms opennms 127.0.0.1(55292) idle
     2893 postgres  20   0  267m  10m 7612 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55293) idle
     2895 postgres  20   0  267m 8968 6548 S  0.0  0.1   0:00.03 postgres: opennms opennms 127.0.0.1(55294) idle
     3507 jboss     20   0  528m  47m 9856 S  0.0  0.6   4:51.80 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma
     3561 jboss     20   0  566m 143m  10m S  0.0  1.8   5:43.77 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log
     5570 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36280) idle
     5572 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36281) idle
     5573 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36282) idle
     5575 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36283) idle
     5578 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36284) idle
     5579 postgres  20   0  266m 4688 2792 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36285) idle
     5581 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36286) idle
     5583 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36287) idle
     5586 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36288) idle
     5587 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36289) idle
     5590 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36290) idle
     5591 postgres  20   0  266m 4692 2796 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36291) idle
     5593 postgres  20   0  266m 4696 2800 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36292) idle
     5595 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36293) idle
     5597 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36294) idle
     5599 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36295) idle
     5601 postgres  20   0  268m  10m 8172 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36296) idle
     5603 postgres  20   0  268m  10m 7376 S  0.0  0.1   0:00.07 postgres: opennms opennms 127.0.0.1(36297) idle
     5605 postgres  20   0  267m 9.8m 7228 S  0.0  0.1   0:00.05 postgres: opennms opennms 127.0.0.1(36298) idle
     5607 postgres  20   0  267m 9972 7132 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36299) idle
     5609 postgres  20   0  268m  10m 8252 S  0.0  0.1   0:00.20 postgres: opennms opennms 127.0.0.1(36300) idle
     5611 postgres  20   0  268m  11m 8848 S  0.0  0.2   0:00.21 postgres: opennms opennms 127.0.0.1(36301) idle
     8123 postgres  20   0  268m  10m 8160 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(56849) idle
    17000 user1      20   0 2299m  68m 8072 S  0.0  0.9   6:32.60 ./jre/bin/java -Djava.compiler=NONE -cp /usr/StorMan/RaidMa
    18186 postgres  20   0  268m  12m 9256 S  0.0  0.2   0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle
    18187 postgres  20   0  268m  11m 8800 S  0.0  0.2   0:00.18 postgres: opennms opennms 127.0.0.1(55331) idle
    18188 postgres  20   0  268m  10m 7484 S  0.0  0.1   0:00.09 postgres: opennms opennms 127.0.0.1(55332) idle
    18190 postgres  20   0  268m  10m 7800 S  0.0  0.1   0:00.10 postgres: opennms opennms 127.0.0.1(55333) idle
    20287 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36339) idle
    20289 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36340) idle
    20291 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36341) idle
    20297 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36342) idle
    20298 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(36343) idle
    20301 postgres  20   0  267m 7448 5228 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36344) idle
    20306 postgres  20   0  267m 7804 5548 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(36345) idle
    20308 postgres  20   0  267m 9660 6872 S  0.0  0.1   0:00.06 postgres: opennms opennms 127.0.0.1(36346) idle
    20311 postgres  20   0  268m  10m 7692 S  0.0  0.1   0:00.14 postgres: opennms opennms 127.0.0.1(36347) idle
    22848 postgres  20   0  266m 4684 2788 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56892) idle
    22850 postgres  20   0  266m 4680 2784 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56893) idle
    22852 postgres  20   0  266m 4676 2780 S  0.0  0.1   0:00.00 postgres: opennms opennms 127.0.0.1(56894) idle
    22858 postgres  20   0  266m 6388 4312 S  0.0  0.1   0:00.01 postgres: opennms opennms 127.0.0.1(56895) idle
    22860 postgres  20   0  268m  10m 7452 S  0.0  0.1   0:00.11 postgres: opennms opennms 127.0.0.1(56896) idle
    22863 postgres  20   0  268m  10m 7968 S  0.0  0.1   0:00.22 postgres: opennms opennms 127.0.0.1(56897) idle
    22864 postgres  20   0  267m 7608 5368 S  0.0  0.1   0:00.02 postgres: opennms opennms 127.0.0.1(56898) idle
    22866 postgres  20   0  268m  10m 7528 S  0.0  0.1   0:00.13 postgres: opennms opennms 127.0.0.1(56899) idle
    29715 user1      16  -4 13052 1068 1064 S  0.0  0.0   0:00.11 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid
    30719 mysql     16  -4 2936m 744m 6628 S  0.0  9.6  62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --
    31064 postgres  20   0  265m  17m  16m S  0.0  0.2   0:32.35 /usr/pgsql-9.4/bin/postmaster -p 5432 -D /var/lib/pgsql/9.4
    31072 postgres  20   0  119m 1728  868 S  0.0  0.0   0:03.28 postgres: logger process
    31074 postgres  20   0  265m  42m  41m S  0.0  0.5   0:39.87 postgres: checkpointer process
    31075 postgres  20   0  265m 7304 6368 S  0.0  0.1   0:05.45 postgres: writer process
    31076 postgres  20   0  265m 6076 5156 S  0.0  0.1   0:53.84 postgres: wal writer process
    31077 postgres  20   0  265m 2688 1496 S  0.0  0.0   0:12.46 postgres: autovacuum launcher process
    31078 postgres  20   0  119m 2056  924 S  0.0  0.0   0:58.40 postgres: stats collector process
    31304 user1      20   0 61220  768  664 S  0.0  0.0   0:00.00 egrep (mysqld|java|postgres)
  8. (Optional) To save job information about a node at a custom output location, enter ./SystemLoadViewer.sh <output-file-path>.

You can view information such as the memory on the nodes, number of root jobs and subjobs on each of the nodes, and so on.

external-footer-nav