Changes to use a new database in the app-id engine have introduced high memory utilization for the application-director.
Issue ID: I95-56484
Last Updated: 2024-05-29
Introduced in SSR Version: 6.2.3
Problem
The application-director has a race condition that can occur during app-id module updates, app-id database reloads, and configuration changes.
It is possible that stale connections to a database will utilize excessive memory over a long duration (application-director uptime).
The administrator might notice alarms for high system memory and an Application Director alarm with the message IP Database reload failed.
Release Notes
Resolve application-director stale database connection memory leak.
Severity
Details
The potential impact of a software defect if encountered. Severity levels are:
- Critical: Could severely affect service, capacity/traffic, and maintenance capabilities. May have a prolonged impact to the entire system.
- Major: Could seriously affect system operation, maintenance, administration and related tasks.
- Minor: Would not significantly impair the functioning or affect service.
Major
Status
Resolved
Resolved In
Product
SSR
Functional Area
System Health
Workaround
Details
Juniper may provide a method to temporarily circumvent a problem; workarounds do not exist for all issues.
Utilizing SSR metrics data, custom charts can be used to view memory usage over time to identify processes that are increasing. Also, the PCLI provides a way to view current process memory utilization and a way to compare the current value a previous point in time.
admin@node.router# show stats process memory process-name application-director
Thu 2024-05-30 20:38:03 UTC
✔ Retrieving statistics...
Process Memory Metrics
----------------------
======== ====== ====================== ============
Metric Node Process-name Value
======== ====== ====================== ============
rss node application-director 61681664
vsz node application-director 1589444608
The above values will appear as bytes. Shown below is comparing the current value to 10 days prior.
admin@node.router# show stats process memory process-name application-director since 10d
Thu 2024-05-30 20:52:39 UTC
✔ Retrieving statistics...
Since: 2024-05-20 20:52:39
Process Memory Metrics
----------------------
======== ====== ====================== =========
Metric Node Process-name Value
======== ====== ====================== =========
rss node application-director 1171456
vsz node application-director 0
In this case, the delta is 10Mb which is normal operating behavior.
However, if the deltas over time are increasing significantly such that the current utilization has doubled or tripled, this is a good indicator that this issue is occurring.
If you have confirmed the application-director has significant memory increases over time, the service can be restarted on the offending device using systemctl restart application-director as a privileged User.