Executing Job Management Scripts and Java Applications
You execute job management scripts and Java applications to view information about jobs triggered from Junos Space nodes. The JobInfoCollector.jar Java application and the job management scripts are stored at the following location: /var/log/space-debug/debug-utilities/jobManagement. When you execute the JobInfoCollector.jar Java application and job management scripts, the output is stored as .txt files at the same location. You can also specify a custom path to store the output. The following sections list the steps to execute the scripts to collect information about jobs.
Executing the Java Application to Collect Job Information
You execute the JobInfoCollector.jar application to collect job information. Ensure that you have updated the MySQL query in the JobInfoCollector.properties file before executing the Java application. The file also contains an example configuration.
To execute the Java application to collect job information:
- Log in to the CLI of the Junos Space node.
- Enter the administrator username and password at the Junos
Space login prompt and press Enter.
The default username is admin and the default password is abc123.
The Junos Space Settings Menu is displayed.
- Enter 6 (if you are using a
hardware appliance) or 7 (if you are using
a virtual appliance) at the Junos Space Settings Menu prompt to run
shell commands.
You are prompted to enter your password.
The following is a sample output from a virtual appliance:
admin@10.206.41.183's password: Last login: Mon Aug 17 06:17:58 2015 from 10.206.41.42 Welcome to the Junos Space network settings utility. Initializing, please wait Junos Space Settings Menu 1> Change Password 2> Change Network Settings 3> Change Time Options 4> Retrieve Logs 5> Security 6> Expand VM Drive Size 7> (Debug) run shell A> Apply changes Q> Quit R> Redraw Menu Choice [1-7,AQR]: 7
- Type the administrator password and press Enter.
The shell prompt appears.
- Type
cd /var/log/space-debug/debug-utilities/jobManagement
at the shell prompt and press Enter. - (Optional) To view the list of job management scripts
and Java applications, enter
ls
and press Enter. - Type
/usr/bin/java -jar JobInfoCollector.jar
and press Enter.The following options will be displayed:
Please select an option from below: 1. Display job parent-children hierarchy 2. Filter from job data 3. Display jobs not yet picked up by Job Dispatcher
- Type 1, 2, or 3 to select an option.
- Enter the database username.
- Enter the database password.
- The inputs displayed depend on the option you selected.
If you selected option 1, you need to enter the parent job ID.
- Enter a job ID.
The output from this command is saved to the JobHierarchy-<job-ID>.txt file in the same directory. The following is a sample output if you selected 1:
=====Job details for 491529 ===== NODE: 10.206.41.57 STARTTIMESTAMP: 17 Jul 2015 17:51:00 GMT JOBSTATE: Done JOBSTATUS: Success ENDTIMESTAMP: 17 Jul 2015 17:52:15 GMT CHILDCOUNT: 2 Child job details: ID || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP || CHILDCOUNT || 491530 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 | 491531 | 10.206.41.57 | 17 Jul 2015 17:51:01 GMT | Done | Success | 17 Jul 2015 17:51:13 GMT | 0 |
If you selected option 2, you need to enter a MySQL query.
- Enter a MySQL query—for example,
mostate=’Failure’
.This query filters the jobs that failed. The following is a sample output if you selected 2:
ID || NAME || NODE || STARTTIMESTAMP || JOBSTATE || JOBSTATUS || ENDTIMESTAMP || 196608 | Discover Network Elements-196608 | 10.206.41.184 | 10 Aug 2015 10:44:48 GMT | Done | Failure | 10 Aug 2015 10:48:47 GMT | 196624 | Discover Network Elements-196624 | 10.206.41.184 | 10 Aug 2015 10:49:52 GMT | Done | Failure | 10 Aug 2015 10:49:58 GMT | 196643 | Mail User Password-196643 | 10.206.41.184 | 10 Aug 2015 14:11:21 GMT | Done | Failure | 10 Aug 2015 14:11:21 GMT | 196649 | Mail User Password-196649 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT | 196650 | Mail User Password-196650 | 10.206.41.184 | 11 Aug 2015 02:39:18 GMT | Done | Failure | 11 Aug 2015 02:39:18 GMT | 196660 | Sync Files-196660 | 10.206.41.187 | 11 Aug 2015 11:54:13 GMT | Done | Failure | 11 Aug 2015 11:54:46 GMT | 360448 | Resync Network Elements-360448 | 10.206.41.184 | 11 Aug 2015 18:06:07 GMT | Done | Failure | 11 Aug 2015 18:06:08 GMT | 360449 | Discover Network Elements-360449 | 10.206.41.187 | 11 Aug 2015 18:12:43 GMT | Done | Failure | 11 Aug 2015 18:13:16 GMT |
If you selected option 3, the list of unscheduled jobs is saved to the UnscheduledJobs.txt file in the same directory.
- The following is a sample output if you selected 3:
ID || NAME || JOBSTATE || PARENTJOBID || 393501 | ND Discovery-393501 | Done | 0 | 393858 | Backup Configuration Files-393858 | Done | 0 | 720906 | ND Discovery-720906 | Done | 0 | 721067 | Backup Configuration Files-721067 | Done | 0 | 721884 | Backup Configuration Files-721884 | Done | 0 | 1048659 | Backup Configuration Files-1048659 | Done | 0 | 1182260 | Enable Script-1182260 | InProgress | 0 | 1182413 | Execute Script-1182413 | InProgress | 0 | 1183914 | Backup Configuration Files-1183914 | Done | 0 | 1184310 | Delete Device-1184310 | Done | 0 | 1189992 | Stage Script-1189992 | InProgress | 0 | 1190105 | Update Network Element-1190105 | InProgress | 0 | 1442231 | Backup Configuration Files-1442231 | Done | 0 | 1442232 | -1442232 | InProgress | 1184310 | 1442233 | -1442233 | InProgress | 1184310 | 1442234 | -1442234 | InProgress | 1184310 | 1442235 | -1442235 | InProgress | 1184310 | 1442236 | -1442236 | InProgress | 1184310 | 1442237 | -1442237 | InProgress | 1184310 | 1442238 | -1442238 | InProgress | 1184310 | 1442239 | -1442239 | InProgress | 1184310 | 1442240 | -1442240 | InProgress | 1184310 | 1445400 | Generate SD LR Report-1445400 | Scheduled | 0 | 1445528 | Cloud Infrastructure Event Purge-1445528 | Scheduled | 0 | 1445602 | Policy Hits Collection-1445602 | Scheduled | 0 | 1903749 | useraccounts-1903749 | Done | 0 |
- Enter a job ID.
You can view the hierarchy of a subjob (input as the parent job ID), list of jobs that are currently unscheduled, or the output based on the query you entered in the JobInfoCollector.properties file.
Executing the Script to View the Stack Trace of a Job
You execute the getJobThreadDump.sh
script
to view the stack trace of a job.
To execute the script to view the stack trace of a job:
- Log in to the CLI of the Junos Space node.
- Enter the administrator username and password at the Junos
Space login prompt and press Enter.
The default username is admin and the default password is abc123.
The Junos Space Settings Menu is displayed.
- Enter 6 (if you are using a
hardware appliance) or 7 (if you are using
a virtual appliance) at the Junos Space Settings Menu prompt to run
shell commands.
You are prompted to enter your password.
- Type the administrator password and press Enter.
The shell prompt appears.
- Type
cd /var/log/space-debug/debug-utilities/jobManagement
at the shell prompt and press Enter. - (Optional) To view the list of job management scripts
and Java applications, type
ls
and press Enter. - Type
./getJobThreadDump.sh <job-ID>
and press Enter.The output from this command is saved to the JobThreadDump-<job-ID>.txt file in the same directory.
- (Optional) To save the stack trace of the job at a custom
output location, type
./getJobThreadDump.sh <job-ID> <output-file-path>
and press Enter.
Executing the Script to View Job Information on Nodes
You execute the SystemLoadViewer.sh
script
to view job information on a node.
To execute the script to view job information on a node:
- Log in to the CLI of the Junos Space node.
- Enter the administrator username and password at the Junos
Space login prompt and press Enter.
The default username is admin and the default password is abc123.
The Junos Space Settings Menu is displayed.
- Enter 6 (if you are using a
hardware appliance) or 7 (if you are using
a virtual appliance) at the Junos Space Settings Menu prompt to run
shell commands.
You are prompted to enter your password.
- Type the administrator password and press Enter.
The shell prompt appears.
- Type
cd /var/log/space-debug/debug-utilities/jobManagement
at the shell prompt and press Enter. - (Optional) To view the list of job management scripts
and Java applications, type
ls
and press Enter. - Type
./SystemLoadViewer.sh
and press Enter.The output from this command is saved to the ./SystemLoadInfo.txt file in the same directory. The following is a sample output from the command:
------------------------------------------------------------ Time of execution: Mon Jul 20 06:54:14 UTC 2015 ------------------------------------------------------------ =====================NODE SUMMARY====================== Node IP: 10.206.41.35 Maximum JVM memory: 2040528896 Estimated memory usage per node: 40465 Reserved memory size: 20000000 (Available Memory = Maximum JVM memor - Estimated memory usage per node - Reserved memory) Available Memory:2020488431 Number of first root jobs on this node:1 Number of next root jobs on this node:0 Number of sub jobs on this node:0 ------------------------------------------- =========================================== Node load statistics: =========================================== Resource Id:autoresync-group:132832:user1@host:0360471000000000008G Type:autoresync-group SubType:132832 Context type:FIRST_ROOT_JOB State:RUNNING Queue: Node IP:10.206.41.35 Creation Time:2015-07-20 06:54:13.166 Last modification time:2015-07-20 06:54:13.166 Estimated Memory:40465 =========================================== ============================SYSTEM LOAD SUMMARY FOR LOCAL NODE============================= Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/jmpvgnocf-lvvar 52772604 2584204 47373632 6% /var ======Command executed: LINES=20 COLUMNS=120 top -n 2 -b -c | tail -n +22================== top - 06:54:18 up 3 days, 19:01, 2 users, load average: 1.24, 0.88, 0.82 Tasks: 271 total, 2 running, 268 sleeping, 0 stopped, 1 zombie Cpu(s): 42.2%us, 23.5%sy, 0.5%ni, 32.7%id, 0.3%wa, 0.1%hi, 0.7%si, 0.0%st Mem: 7937672k total, 7827836k used, 109836k free, 539116k buffers Swap: 8193140k total, 132684k used, 8060456k free, 1246572k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 30383 user1 20 0 83616 24m 2508 S 36.8 0.3 0:02.16 /usr/bin/perl /var/www/cgi-bin/secure/resourceMonitoring 4204 jboss 20 0 4081m 2.7g 14m S 27.6 36.0 395:59.73 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav 31240 user1 20 0 124m 15m 2448 S 7.6 0.2 0:00.23 perl -e use lib("/usr/nma/lib"); use NmaUtil; print NmaUtil 31251 user1 20 0 72016 14m 2384 R 6.9 0.2 0:00.21 perl -e use lib("/usr/nma/lib"); use NmaDb; print NmaDb::ge 30719 mysql 16 -4 2936m 744m 6628 S 3.6 9.6 62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql -- 32329 opennms 25 5 3446m 877m 21m S 3.0 11.3 164:35.59 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor 3120 user1 20 0 84612 21m 2460 S 1.0 0.3 13:51.23 /usr/bin/perl /usr/bin/jmp-sync 30651 user1 20 0 12896 1436 952 R 1.0 0.0 0:00.07 top -n 2 -b -c 13332 user1 20 0 154m 9824 4480 S 0.7 0.1 26:56.92 /usr/sbin/snmpd -Lsd -Lf /dev/null -p /var/run/snmpd.pid -a 18186 postgres 20 0 268m 12m 9252 S 0.7 0.2 0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle 2589 jboss 20 0 924m 197m 10m S 0.3 2.5 6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log 15335 apache 23 3 141m 4936 3012 S 0.3 0.1 0:00.62 /usr/sbin/httpd ======Command executed: COLUMNS=120 top -n 1 -b -c | egrep '(mysqld|java|postgres)'======== 4204 jboss 20 0 4081m 2.7g 14m S 1.9 36.0 395:59.74 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79.x86_64/jre/bin/jav 32329 opennms 25 5 3446m 877m 21m S 1.9 11.3 164:35.60 /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java -Djava.endor 2556 jboss 20 0 527m 47m 9800 S 0.0 0.6 4:44.43 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma 2589 jboss 20 0 924m 197m 10m S 0.0 2.5 6:11.24 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log 2886 postgres 20 0 268m 10m 7524 S 0.0 0.1 0:00.10 postgres: opennms opennms 127.0.0.1(55289) idle 2888 postgres 20 0 268m 10m 7580 S 0.0 0.1 0:00.08 postgres: opennms opennms 127.0.0.1(55290) idle 2889 postgres 20 0 268m 10m 8040 S 0.0 0.1 0:00.15 postgres: opennms opennms 127.0.0.1(55291) idle 2891 postgres 20 0 268m 12m 9352 S 0.0 0.2 0:00.31 postgres: opennms opennms 127.0.0.1(55292) idle 2893 postgres 20 0 267m 10m 7612 S 0.0 0.1 0:00.09 postgres: opennms opennms 127.0.0.1(55293) idle 2895 postgres 20 0 267m 8968 6548 S 0.0 0.1 0:00.03 postgres: opennms opennms 127.0.0.1(55294) idle 3507 jboss 20 0 528m 47m 9856 S 0.0 0.6 4:51.80 java -D[Process Controller] -server -Xms32m -Xmx128m -XX:Ma 3561 jboss 20 0 566m 143m 10m S 0.0 1.8 5:43.77 java -D[Host Controller] -Dorg.jboss.boot.log.file=/var/log 5570 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36280) idle 5572 postgres 20 0 266m 4688 2792 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36281) idle 5573 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36282) idle 5575 postgres 20 0 266m 4684 2788 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36283) idle 5578 postgres 20 0 266m 4696 2800 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36284) idle 5579 postgres 20 0 266m 4688 2792 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36285) idle 5581 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36286) idle 5583 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36287) idle 5586 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36288) idle 5587 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36289) idle 5590 postgres 20 0 266m 4692 2796 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36290) idle 5591 postgres 20 0 266m 4692 2796 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36291) idle 5593 postgres 20 0 266m 4696 2800 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36292) idle 5595 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36293) idle 5597 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36294) idle 5599 postgres 20 0 266m 4684 2788 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36295) idle 5601 postgres 20 0 268m 10m 8172 S 0.0 0.1 0:00.14 postgres: opennms opennms 127.0.0.1(36296) idle 5603 postgres 20 0 268m 10m 7376 S 0.0 0.1 0:00.07 postgres: opennms opennms 127.0.0.1(36297) idle 5605 postgres 20 0 267m 9.8m 7228 S 0.0 0.1 0:00.05 postgres: opennms opennms 127.0.0.1(36298) idle 5607 postgres 20 0 267m 9972 7132 S 0.0 0.1 0:00.06 postgres: opennms opennms 127.0.0.1(36299) idle 5609 postgres 20 0 268m 10m 8252 S 0.0 0.1 0:00.20 postgres: opennms opennms 127.0.0.1(36300) idle 5611 postgres 20 0 268m 11m 8848 S 0.0 0.2 0:00.21 postgres: opennms opennms 127.0.0.1(36301) idle 8123 postgres 20 0 268m 10m 8160 S 0.0 0.1 0:00.14 postgres: opennms opennms 127.0.0.1(56849) idle 17000 user1 20 0 2299m 68m 8072 S 0.0 0.9 6:32.60 ./jre/bin/java -Djava.compiler=NONE -cp /usr/StorMan/RaidMa 18186 postgres 20 0 268m 12m 9256 S 0.0 0.2 0:00.38 postgres: opennms opennms 127.0.0.1(55330) idle 18187 postgres 20 0 268m 11m 8800 S 0.0 0.2 0:00.18 postgres: opennms opennms 127.0.0.1(55331) idle 18188 postgres 20 0 268m 10m 7484 S 0.0 0.1 0:00.09 postgres: opennms opennms 127.0.0.1(55332) idle 18190 postgres 20 0 268m 10m 7800 S 0.0 0.1 0:00.10 postgres: opennms opennms 127.0.0.1(55333) idle 20287 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36339) idle 20289 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36340) idle 20291 postgres 20 0 266m 4684 2788 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36341) idle 20297 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36342) idle 20298 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(36343) idle 20301 postgres 20 0 267m 7448 5228 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36344) idle 20306 postgres 20 0 267m 7804 5548 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(36345) idle 20308 postgres 20 0 267m 9660 6872 S 0.0 0.1 0:00.06 postgres: opennms opennms 127.0.0.1(36346) idle 20311 postgres 20 0 268m 10m 7692 S 0.0 0.1 0:00.14 postgres: opennms opennms 127.0.0.1(36347) idle 22848 postgres 20 0 266m 4684 2788 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(56892) idle 22850 postgres 20 0 266m 4680 2784 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(56893) idle 22852 postgres 20 0 266m 4676 2780 S 0.0 0.1 0:00.00 postgres: opennms opennms 127.0.0.1(56894) idle 22858 postgres 20 0 266m 6388 4312 S 0.0 0.1 0:00.01 postgres: opennms opennms 127.0.0.1(56895) idle 22860 postgres 20 0 268m 10m 7452 S 0.0 0.1 0:00.11 postgres: opennms opennms 127.0.0.1(56896) idle 22863 postgres 20 0 268m 10m 7968 S 0.0 0.1 0:00.22 postgres: opennms opennms 127.0.0.1(56897) idle 22864 postgres 20 0 267m 7608 5368 S 0.0 0.1 0:00.02 postgres: opennms opennms 127.0.0.1(56898) idle 22866 postgres 20 0 268m 10m 7528 S 0.0 0.1 0:00.13 postgres: opennms opennms 127.0.0.1(56899) idle 29715 user1 16 -4 13052 1068 1064 S 0.0 0.0 0:00.11 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid 30719 mysql 16 -4 2936m 744m 6628 S 0.0 9.6 62:17.79 /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql -- 31064 postgres 20 0 265m 17m 16m S 0.0 0.2 0:32.35 /usr/pgsql-9.4/bin/postmaster -p 5432 -D /var/lib/pgsql/9.4 31072 postgres 20 0 119m 1728 868 S 0.0 0.0 0:03.28 postgres: logger process 31074 postgres 20 0 265m 42m 41m S 0.0 0.5 0:39.87 postgres: checkpointer process 31075 postgres 20 0 265m 7304 6368 S 0.0 0.1 0:05.45 postgres: writer process 31076 postgres 20 0 265m 6076 5156 S 0.0 0.1 0:53.84 postgres: wal writer process 31077 postgres 20 0 265m 2688 1496 S 0.0 0.0 0:12.46 postgres: autovacuum launcher process 31078 postgres 20 0 119m 2056 924 S 0.0 0.0 0:58.40 postgres: stats collector process 31304 user1 20 0 61220 768 664 S 0.0 0.0 0:00.00 egrep (mysqld|java|postgres)
- (Optional) To save job information about a node at a custom
output location, enter
./SystemLoadViewer.sh <output-file-path>
.
You can view information such as the memory on the nodes, number of root jobs and subjobs on each of the nodes, and so on.