Help us improve your experience.

Let us know what you think.

Do you have time for a two-minute survey?

Navigation

Troubleshooting: Software Upgrade Failure Due to Insufficient Free Disk Space on a Routing Matrix with a TX Matrix Plus Router

Problem

In a routing matrix with a TX Matrix Plus router, a successful upgrade of the Junos OS requires sufficient free disk space on all Routing Engines of the TX Matrix Plus router and connected T1600 or T4000 routers.

If the amount of free disk space on a Routing Engine is insufficient to install the Junos OS, you might receive a warning that the /var filesystem is low on free disk space, similar to the following message:

WARNING: The /var filesystem is low on free disk space.
WARNING: This package requires 1075136k free, but there
WARNING: is only 666502k available.

Cause

The amount of free disk space necessary to upgrade a Routing Engine with a new version of the Junos OS can vary from one release to another. Check the Junos OS Release Notes for the software version you are installing.

To determine the amount of free disk space on each Routing Engine in the routing matrix, issue the show system storage command on the master Routing Engine and also on the backup Routing Engine on the TX Matrix Plus router.

Note: When you issue the show system storage command on a master Routing Engine of a TX Matrix Plus router, the command is broadcast to the master Routing Engines (only) on all T1600 or T4000 routers in the routing matrix. Similarly, when you issue the show system storage command on a backup Routing Engine of a TX Matrix Plus router, the command is broadcast to the backup Routing Engines (only) on all T1600 or T4000 routers in the routing matrix.

The following example shows sample output for the show system storage command when it is issued on the master Routing Engine in slot 0 on the TX Matrix Plus router:

root@sfc0> show system storage 
sfc0-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                2.0G       8.0K       1.8G        0%  /tmp
/dev/md10               2.0G        11M       1.8G        1%  /mfs
/dev/ad0s1e             376M        26K       346M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              55G        27G        23G       54%  /var

lcc0-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                3.2G        14K       2.9G        0%  /tmp
/dev/md10               3.2G       438K       2.9G        0%  /mfs
---(more)---
                                        
/dev/ad0s1e             375M       3.9M       341M        1%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              52G        30G        18G       63%  /var

lcc2-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                3.2G        14K       2.9G        0%  /tmp
/dev/md10               3.2G       432K       2.9G        0%  /mfs
/dev/ad0s1e             376M        14K       346M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              53G        27G        22G       56%  /var

lcc4-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
---(more 60%)---
                                        
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                2.0G        14K       1.8G        0%  /tmp
/dev/md10               2.0G       432K       1.8G        0%  /mfs
/dev/ad0s1e             376M        14K       346M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              55G        27G        23G       54%  /var

lcc6-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                2.0G        14K       1.8G        0%  /tmp
/dev/md10               2.0G       432K       1.8G        0%  /mfs
/dev/ad0s1e             376M        14K       346M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              55G        25G        25G       50%  /var

lcc7-re0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             3.3G       370M       2.7G       12%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                 65M        65M         0B      100%  /packages/mnt/jbase
/dev/md1                 41M        41M         0B      100%  /packages/mnt/jkernel64-13.1R1.0
/dev/md2                147M       147M         0B      100%  /packages/mnt/jpfe-T-13.1R1.0
---(more 90%)---
                                        
/dev/md3                5.3M       5.3M         0B      100%  /packages/mnt/jdocs-13.1R1.0
/dev/md4                115M       115M         0B      100%  /packages/mnt/jroute-13.1R1.0
/dev/md5                 29M        29M         0B      100%  /packages/mnt/jcrypto64-13.1R1.0
/dev/md6                 65M        65M         0B      100%  /packages/mnt/jpfe-common-13.1R1.0
/dev/md7                428M       428M         0B      100%  /packages/mnt/jruntime-13.1R1.0
/dev/md8                 24M        24M         0B      100%  /packages/mnt/jruntime64-13.1R1.0
/dev/md9                2.0G        14K       1.8G        0%  /tmp
/dev/md10               2.0G       438K       1.8G        0%  /mfs
/dev/ad0s1e             376M        14K       346M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f              55G        26G        25G       51%  /var

Solution

To resolve the issue of insufficient free disk space on a Routing Engine to perform a Junos OS upgrade on that router, perform the following steps:

  1. To free up space in the file system, you can log in to a particular Routing Engine and then issue the request system storage cleanup command. This command deletes rotating log files in /var/log that are not current, temporary files in /var/tmp that have not been modified within the last two days, and all crash files in /var/crash. The command output first lists the files proposed for deletion and then prompts you to proceed to delete these storage cleanup candidates.

    Note: Although the command prompts you before deleting files, you can issue the command with the dry-run option, which causes the command to only list the cleanup candidates without offering to remove them.

    The following example shows sample output for the request system storage cleanup command with the dry-run option included:

    root@sfc0> request system storage cleanup dry-run
    Currently rotating log files, please wait.
    This operation can take up to a minute.
    
    List of files to delete:
    
    	 Size Date	   Name
    1287.5K Nov 21 23:57 /var/log/bgp.0.gz
    1060.6K Nov 21 23:37 /var/log/bgp.1.gz
    1061.9K Nov 21 23:37 /var/log/bgp.2.gz
    1053.0K Nov 21 23:10 /var/log/bgp.3.gz
     152.5K Dec  2 20:37 /var/log/chassisd.0.gz
     164.0K Nov 29 22:27 /var/log/chassisd.1.gz
     169.4K Nov 27 20:07 /var/log/chassisd.2.gz
     167.0K Nov 26 22:45 /var/log/chassisd.3.gz
      39.8K Dec  2 20:39 /var/log/dcd.0.gz
      47.4K Nov 24 19:27 /var/log/dcd.1.gz
      46.9K Jun  9 22:15 /var/log/filter.0.gz
      54.4K Jun  9 18:00 /var/log/filter.1.gz
      56.6K Jun  9 17:15 /var/log/filter.2.gz
      56.2K Jun  9 16:30 /var/log/filter.3.gz
      53.4K Jun  9 15:45 /var/log/filter.4.gz
      53.8K Jun  9 15:00 /var/log/filter.5.gz
      55.9K Jun  9 14:15 /var/log/filter.6.gz
      56.0K Jun  9 13:30 /var/log/filter.7.gz
      56.7K Jun  9 12:45 /var/log/filter.8.gz
      55.6K Jun  9 12:00 /var/log/filter.9.gz
       794B Nov 26 01:10 /var/log/install.0.gz
       786B Nov 25 02:27 /var/log/install.1.gz
       775B Oct 30 20:02 /var/log/install.10.gz
       790B Oct 30 08:02 /var/log/install.11.gz
       785B Oct 28 21:06 /var/log/install.12.gz
       775B Oct 26 10:30 /var/log/install.13.gz
       789B Oct 26 04:35 /var/log/install.14.gz
       786B Oct 19 07:32 /var/log/install.15.gz
       821B Oct 19 02:41 /var/log/install.16.gz
       887B Oct 18 07:27 /var/log/install.17.gz
       791B Oct 17 07:26 /var/log/install.18.gz
       791B Nov 22 19:18 /var/log/install.2.gz
                                            
                                            
      
      ...output_truncated...
      
    
    7157B May 31  2012 /var/tmp/pa4746.01/jinstall64-12.2-20120607.0-domestic-signed.x4746/jinstall64_pkg/packages/certs.pem
    231B  Jun  7 15:01 /var/tmp/pa4746.01/jinstall64-12.2R1.0-domestic-signed.x4746/jinstall64_pkg/pkg/manifest
    7157B Jun  7 12:21 /var/tmp/pa4746.01/jinstall64-12.2-20120607.0-domestic-signed.x4746/jinstall64_pkg/pkg/manifest.certs
    41B Jun  7 15:01 /var/tmp/pa4746.01/jinstall64-12.2-20120607.0-domestic-signed.x4746/jinstall64_pkg/pkg/manifest.sha1
    11.4M Jun  7 17:19 /var/tmp/pa4746.01/jinstall64-12.2-20120607.0-domestic-signed.x4746/jinstall64_pkg/bootstrap-install-12.2-20120607.0.tar
    Delete these files ? [yes,no] (no) 
  2. You can use the following file management operational commands to display information about the directories and files on a Routing Engine, but you must include the chassis and Routing Engine specifiers in the pathname:

    • To display detailed information about a file or directory contents, issue the file list command with the detail option.
    • To display the contents of a file, issue the file show command.
    • To compare the contents of two files, issue the file compare command.
  3. In addition to or instead of using the request system storage cleanup command to delete unnecessary files, you can use the following file management commands to delete or move files on a Routing Engine, but you must include the chassis and Routing Engine specifiers in the pathname:

    If you elect to use the request system storage cleanup command to automatically delete files, issue the command without the dry-run option, and then type yes when the CLI prompts to proceed with deleting the candidate files. The following truncated example shows sample output for the request system storage cleanup command:
  4. After sufficient free disk space is available on the Routing Engines on which you are upgrading the Junos OS, attempt the upgrade procedure again.

Published: 2013-03-13