Define Junos PyEZ Operational Tables for Parsing Unstructured Output
SUMMARY Create custom Tables that select data from CLI or vty command output to extract operational information from a Junos device.
Junos PyEZ operational (op) Tables for unstructured output extract data from the text output of a CLI command executed on a Junos device or a vty command executed on a given Flexible PIC Concentrator (FPC). The extracted data is then converted to JSON. This enables you to quickly retrieve and analyze operational state information for the device. Junos PyEZ op Tables for unstructured output are particularly useful when you need to parse command output that cannot be returned in a structured format such as XML.
This topic discusses the different components of the Table.
Summary of Parameters in Op Tables for Parsing Unstructured Output
Junos PyEZ Tables are formatted using YAML. Op Table definitions can include a number of required and optional parameters, which are summarized in Table 1.
Table Parameter Name |
Table Parameter |
Description |
---|---|---|
Table name |
– |
User-defined identifier for the Table. |
Command |
|
CLI or vty command to execute. |
Command arguments |
|
(Optional) When you define the command as a Jinja template, |
Target FPC |
|
(Optional) Flexible PIC Concentrator (FPC) on which to execute a vty command. |
Table item |
|
(Optional) String or regular expression that defines how to split the output into sections. These sections become the iterable reference for the associated View. Specify '*' to extract and match against the whole string rather than each line. |
Table item key |
|
(Optional) String or list of strings that define one or more keys that uniquely identify each Table item. |
Selected keys |
|
(Optional) List of one or more Table item keys for which to return data. |
Section title |
|
(Optional) String that selects the section of output containing the data to parse. |
Field delimiter |
|
(Optional) Delimiter that defines how to split the data in command output comprised of key-value pairs. The extracted data is stored as key-value pairs in a dictionary. |
Eval expression |
|
(Optional) Associative array, or dictionary, of one or
more key-value pairs that map a user-defined key to a string containing
a mathematical expression. When you retrieve the data, the expression
is evaluated using the Python |
Table View |
|
(Optional) View that is used to extract field data from the Table items. |
TextFSM template |
|
(Optional) Boolean that specifies whether a TextFSM template is used to parse the data. |
TextFSM template platform identifier |
|
(Optional) When a TextFSM template is used, specify the platform for the template. |
Table Name
The Table name is a user-defined identifier for the Table.
The YAML file or string can contain one or more Tables. The start
of the YAML document must be left justified. The following example
defines a Table named ChassisFanTable
:
--- ChassisFanTable: command: show chassis fan key: fan-name view: ChassisFanView
Command
A Junos PyEZ op Table for unstructured output extracts data
from the text output of a CLI or vty command. You must include the command
property in the Table definition to specify
the CLI command to execute on a device or the vty command to execute
on a given FPC. You can define the command as a simple string or a
Jinja template.
For example, the following Table executes the show
chassis fan
command on the device.
--- ChassisFanTable: command: show chassis fan key: fan-name view: ChassisFanView
The following Table executes the show cmerror module
brief
vty command on the target FPC.
--- CMErrorTable: command: show cmerror module brief target: fpc1 key: module view: CMErrorView
When you define the command as a Jinja template, you must also
supply the args
parameter with a dictionary
of key-value pairs that map the variables in the template to the values
used when the template is rendered. For information about defining
the command as a Jinja template, see Command Arguments (args).
Command Arguments (args)
You can define the CLI or vty command for the command
parameter using a Jinja template and substitute variables for the
command arguments. When you use a Jinja template, you must also define
the args
parameter, which is a dictionary
of key-value pairs that map the variable names in the Jinja template
to the values used when the template is rendered. You can provide
default values for the template variables in the Table definition,
and you can also define the values in the Junos PyEZ application.
To define default values for the template variables,
include the args
parameter in the Table
definition and map each template variable to its default value. The
following Table defines a command using a Jinja template that has
one variable, protocol
. The args
parameter defines the default value for protocol
, which is used when you call the get()
method in your script and do not provide an argument
that overrides that default.
--- DdosPolicerStatsTable: command: show ddos policer stats {{ protocol }} args: protocol: ospf target: Null title: "DDOS Policer Statistics:" key: location view: DdosPolicerStatsView
Additionally, you can define the args
argument as a dictionary in the Table’s get()
method to:
define a value for any template variable that does not have a default value defined in the Table
override the default value defined in the Table for one or more template variables
The following example executes the command in the previous Table using protocol 'bgp' instead of the default value of 'ospf'.
from jnpr.junos import Device from jnpr.junos.command.pfe_ddos_policer import DdosPolicerStatsTable from pprint import pprint import json with Device(host='router1.example.com') as dev: stats = DdosPolicerStatsTable(dev) stats.get(target='fpc0', args={'protocol':'bgp'}) pprint(json.loads(stats.to_json()))
Target FPC (fpc)
Junos PyEZ op Tables can execute vty commands on a specific
Flexible PIC Concentrator (FPC). When you use a vty command, the Table
must include the target
parameter to define
the target FPC. You can set target
to Null
and force the user to specify the target FPC in
the application, or you can set target
to
a default FPC, and the user can optionally override this value in
the application.
The following Table executes the show memory
vty command, but sets target: Null
, which
requires that the user supply the target FPC in the Junos PyEZ application:
--- FpcMemory: command: show memory target: Null key: ID key_items: - 0 - 1 view: FPCMemoryView
The following Table executes the show memory
vty command on FPC 1, unless the user overrides this value in the
Junos PyEZ application.
--- FpcMemory: command: show memory target: fpc1 key: ID key_items: - 0 - 1 view: FPCMemoryView
In the Junos PyEZ application, to define the target FPC
or override the default FPC defined in the Table, set the target
argument in the Table’s get()
method to the desired FPC, for example:
from jnpr.junos import Device from jnpr.junos.command.fpc_memory import FpcMemory from pprint import pprint import json with Device(host='router.example.com') as dev: stats = FpcMemory(dev) stats.get(target='fpc0') pprint(json.loads(stats.to_json()))
Table Item (item)
The optional Table item
property
is a string or regular expression that defines how to split the command
output for parsing. If the output has similar, repeating sets of data,
you can define item
to match on and extract
each iteration of the data. For example, show interfaces
returns a similar set of data for many interfaces. Alternatively,
you can define item: '*'
when you need
to extract the data as a single block of text.
Consider the following output for the show devices
local
vty command:
TSEC Ethernet Device Driver: .le1, Control 0x4296c218, (1000Mbit) HW reg base 0xff724000 [0]: TxBD base 0x7853ce20, RxBD Base 0x7853d640 [1]: TxBD base 0x7853d860, RxBD Base 0x7853e080 [2]: TxBD base 0x7853e2a0, RxBD Base 0x785422c0 [3]: TxBD base 0x785426e0, RxBD Base 0x78544700 Receive: 185584608 bytes, 2250212 packets, 0 FCS errors, 0 multicast packets 107271 broadcast packets, 0 control frame packets 0 PAUSE frame packets, 0 unknown OP codes 0 alignment errors, 0 frame length errors 0 code errors, 0 carrier sense errors 0 undersize packets, 0 oversize packets 0 fragments, 0 jabbers, 0 drops Receive per queue: [0]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [1]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [2]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [3]: 203586808 bytes, 2250219 packets, 0 dropped 0 jumbo, 0 truncated jumbo Transmit: 288184646 bytes, 2038370 packets, 0 multicast packets 106531 broadcast packets, 0 PAUSE control frames 0 deferral packets, 0 excessive deferral packets 0 single collision packets, 0 multiple collision packets 0 late collision packets, 0 excessive collision packets 0 total collisions, 0 drop frames, 0 jabber frames 0 FCS errors, 0 control frames, 0 oversize frames 0 undersize frames, 0 fragments frames Transmit per queue: [0]: 10300254 bytes, 72537 packets 0 dropped, 0 fifo errors [1]: 4474302 bytes, 106531 packets 0 dropped, 0 fifo errors [2]: 260203538 bytes, 1857137 packets 0 dropped, 0 fifo errors [3]: 199334 bytes, 2179 packets 0 dropped, 0 fifo errors TSEC status counters: kernel_dropped:0, rx_large:0 rx_short: 0 rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0 rx_bsy: 0,rx_babr:0, rx_trunc: 0 rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0 rx_errors: 0, rx_ints: 2250110, collisions: 0 eberr:0, tx_babt: 0, tx_underrun: 0 tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0 tx_aborted_errors: 0, tx_ints: 2038385, resets: 1 TSEC Ethernet Device Driver: .le3, Control 0x42979220, (1000Mbit) HW reg base 0xff726000 [0]: TxBD base 0x78545720, RxBD Base 0x78545f40 [1]: TxBD base 0x78546160, RxBD Base 0x78546980 [2]: TxBD base 0x78546ba0, RxBD Base 0x7854abc0 [3]: TxBD base 0x7854afe0, RxBD Base 0x7854d000 Receive: 0 bytes, 0 packets, 0 FCS errors, 0 multicast packets 0 broadcast packets, 0 control frame packets 0 PAUSE frame packets, 0 unknown OP codes 0 alignment errors, 0 frame length errors 0 code errors, 0 carrier sense errors 0 undersize packets, 0 oversize packets 0 fragments, 0 jabbers, 0 drops Receive per queue: [0]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [1]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [2]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo [3]: 0 bytes, 0 packets, 0 dropped 0 jumbo, 0 truncated jumbo Transmit: 6817984 bytes, 106531 packets, 0 multicast packets 106531 broadcast packets, 0 PAUSE control frames 0 deferral packets, 0 excessive deferral packets 0 single collision packets, 0 multiple collision packets 0 late collision packets, 0 excessive collision packets 0 total collisions, 0 drop frames, 0 jabber frames 0 FCS errors, 0 control frames, 0 oversize frames 0 undersize frames, 0 fragments frames Transmit per queue: [0]: 0 bytes, 0 packets 0 dropped, 0 fifo errors [1]: 4474302 bytes, 106531 packets 0 dropped, 0 fifo errors [2]: 0 bytes, 0 packets 0 dropped, 0 fifo errors [3]: 0 bytes, 0 packets 0 dropped, 0 fifo errors TSEC status counters: kernel_dropped:0, rx_large:0 rx_short: 0 rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0 rx_bsy: 0,rx_babr:0, rx_trunc: 0 rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0 rx_errors: 0, rx_ints: 0, collisions: 0 eberr:0, tx_babt: 0, tx_underrun: 0 tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0 tx_aborted_errors: 0, tx_ints: 106531, resets: 1
The following Table extracts each section of the output
that starts with TSEC Ethernet Device Driver:
. In this case, the value for key: name
is derived from the capturing group in the regular expression defined
in item
.
--- DevicesLocalTable: command: show devices local target: fpc1 item: 'TSEC Ethernet Device Driver: (\.?\w+),' key: name view: DevicesLocalView DevicesLocalView: fields: TSEC_status_counters: _TSECStatusCountersTable receive_counters: _ReceiveTable transmit_per_queue: _TransmitQueueTable
You can also define item
as an asterisk
('*') if you want to match against the entire section of output instead
of matching each line. When you include item: '*'
, in most cases, you must also include the title
parameter to specify the heading for the section of output to extract.
Extracting the output using item: '*'
is
useful when you include the regex
parameter
in a View and want each expression to match against the entire text
string. Otherwise, the regex
expressions
are combined and matched against each line.
The following Table extracts the text block under the
heading Receive:
and matches each regular
expression against the entire text string:
_ReceiveTable: item: '*' title: 'Receive:' view: _ReceiveView _ReceiveView: regex: bytes: '(\d+) bytes' packets: '(\d+) packets' FCS_errors: '(\d+) FCS errors' broadcast_packets: '(\d+) broadcast packets'
Table Item Key (key)
The optional key
property defines
the output fields that are used to uniquely identify a Table item.
You can identify a Table item using a single key or a list of keys.
If the Table and View return iterative data, the key
value must reference a variable or field name defined in the View.
Consider the following show chassis fan
output:
Item Status RPM Measurement Fan 1 OK 5280 Spinning at normal speed Fan 2 OK 5280 Spinning at normal speed Fan 3 OK 5280 Spinning at normal speed Fan 4 OK 5280 Spinning at normal speed Fan 5 OK 5280 Spinning at normal speed
The following Table defines the Table item key as fan-name
, which maps to the value under the Item
column in the output.
ChassisFanTable: command: show chassis fan key: fan-name view: ChassisFanView ChassisFanView: columns: fan-name: Item fan-status: Status fan-rpm: RPM fan-measurement: Measurement
When you retrieve and print the data in the Junos PyEZ application, each item in the resulting dictionary uses this field’s value as its key.
dict_keys(['Fan 1', 'Fan 2', 'Fan 3', 'Fan 4', 'Fan 5'])
You can also define key
as
a list to identify a Table item using a composite key. For example:
--- FPCIPV4AddressTable: command: show ipv4 address target: fpc1 key: - name - addr view: FPCIPV4AddressView FPCIPV4AddressView: columns: index: Index addr: Address name: Name
Selected Keys (key_items)
The key
parameter defines the output
fields that uniquely identify a Table item. When you include the key
parameter in a Table, you can use the optional key_items
parameter to only return data for certain
key values. key_items
defines the key or
list of keys of the Table items for which to retrieve data. You can
define the key_items
parameter in the Table
definition or in the Junos PyEZ application.
Consider the following show chassis fan
output:
Item Status RPM Measurement Fan 1 OK 5280 Spinning at normal speed Fan 2 OK 5280 Spinning at normal speed Fan 3 OK 5280 Spinning at normal speed Fan 4 OK 5280 Spinning at normal speed Fan 5 OK 5280 Spinning at normal speed
The following Table defines the Table item key as fan-name
and only retrieves the data for the Table
item with a key value equal to Fan 1
.
ChassisFanTable: command: show chassis fan key: fan-name key_items: - Fan 1 view: ChassisFanView ChassisFanView: columns: fan-name: Item fan-status: Status fan-rpm: RPM fan-measurement: Measurement
In the Junos PyEZ application, to define the key_items
to return or to override the key_items
defined in the Table, set the key_items
argument in the Table’s get()
method to a list or tuple of the desired items,
for example:
from jnpr.junos import Device from jnpr.junos.command.chassis_fan import ChassisFanTable from pprint import pprint import json with Device(host='router.example.com') as dev: fans = ChassisFanTable(dev) fans.get(key_items=['Fan 1', 'Fan 2']) pprint(json.loads(fans.to_json()))
user@host:~$ python3 junos-pyez-get-fan-data.py {'Fan 1': {'fan-measurement': 'Spinning at normal speed', 'fan-name': 'Fan 1', 'fan-rpm': 5280, 'fan-status': 'OK'}, 'Fan 2': {'fan-measurement': 'Spinning at normal speed', 'fan-name': 'Fan 2', 'fan-rpm': 5280, 'fan-status': 'OK'}}
Section Title (title)
Tables can include the optional title
parameter to define the starting point for a section in the command
output from which to extract and parse the data. When the Table defines item: '*'
, you must include title
to specify the heading for the section of output to extract
For example, consider the following command output, which is enclosed within a larger set of output:
... TSEC status counters: kernel_dropped:0, rx_large:0 rx_short: 0 rx_nonoctet: 0, rx_crcerr: 0, rx_overrun: 0 rx_bsy: 0,rx_babr:0, rx_trunc: 0 rx_length_errors: 0, rx_frame_errors: 0 rx_crc_errors: 0 rx_errors: 0, rx_ints: 2250110, collisions: 0 eberr:0, tx_babt: 0, tx_underrun: 0 tx_timeout: 0, tx_timeout: 0,tx_window_errors: 0 tx_aborted_errors: 0, tx_ints: 2038385, resets: 1 ...
The following Table uses the title
parameter to extract and parse data from the TSEC
status counters
section of the output. In this case,
the Table defines item
as '*', which considers
the data as a single text string.
_TSECStatusCountersTable: item: '*' title: 'TSEC status counters:' view: _TSECStatusCountersView _TSECStatusCountersView: regex: kernel_dropped: 'kernel_dropped:(\d+)' rx_large: 'rx_large:(\d+)'
Field Delimiter (delimiter)
Some operational commands return output comprised of only key-value
pairs. If you want to simply retrieve the entire set of data and extract
each key-value pair, you can use the delimiter
parameter to define how to split each data pair instead of defining
a separate View. Junos PyEZ uses the delimiter to split the data at
the specified boundary and stores each key-value pair in a dictionary.
Consider the following output for the show
link stats
vty command.
PPP LCP/NCP: 0 HDLC keepalives: 0 RSVP: 0 ISIS: 0 OSPF Hello: 539156 OAM: 0 BFD: 15 UBFD: 0 LMI: 0 LACP: 0 ETHOAM: 0 SYNCE: 0 PTP: 0 L2TP: 0 LNS-PPP: 0 ARP: 4292 ELMI: 0 VXLAN MRESOLVE: 0 Unknown protocol: 42
The following Table defines the delimiter as the colon (:) character:
--- FPCLinkStatTable: command: show link stats target: fpc1 delimiter: ":"
When you retrieve the data in the Junos PyEZ application, the Table splits each line of output at the delimiter and stores the key-value pairs in a dictionary.
{'ARP': 4292, 'ELMI': 0, 'SYNCE': 0, 'ISIS': 0, 'BFD': 15, 'PPP LCP/NCP': 0, 'OAM': 0, 'ETHOAM': 0, 'LACP': 0, 'LMI': 0, 'Unknown protocol': 42, 'UBFD': 0, 'L2TP': 0, 'HDLC keepalives': 0, 'LNS-PPP': 0, 'OSPF Hello': 539156, 'RSVP': 0, 'VXLAN MRESOLVE': 0, 'PTP': 0}
Eval Expression (eval)
You can use the optional eval
parameter
to add or modify key-value pairs in the final data returned by the
Table and View. The eval
parameter maps
a key name to a string containing a mathematical expression that gets
evaluated by the Python eval
function.
You can include the eval
parameter in both
Tables and Views, and eval
can define and
calculate multiple values.
When you use eval
in a Table, it
references the full data dictionary for the calculation, and the key
and calculated value are added as a single additional item to the
dictionary. When you use eval
in a View,
the expression is calculated on each iteration of the data, and the
calculated value is added to the data for that iteration. If the eval
key name matches a key defined in the View, eval
replaces the value of that key with the calculated
value. If the eval
key name does not match
a key defined in the View, eval
adds the
new key and calculated value to the data.
The eval
expression can reference
the data
dictionary returned by the View.
The expression must reference data
using
a Jinja template variable, so that Junos PyEZ can replace the variable
with the dictionary when the expression is evaluated.
The following example uses eval
in the Table definition to include a single additional entry in
the data dictionary. The added item’s key is cchip_errors_from_lkup_chip
, and its value is the sum of the number of interrupts.
--- CChipLiInterruptStatsTable: command: show xmchip {{ chip_instance }} li interrupt-stats target: NULL args: chip_instance: 0 key: - li_block - name eval: cchip_errors_from_lkup_chip: "reduce(lambda x,y: x+y, [v['interrupts'] for k,v in {{ data }}.items()])" view: CChipLiInterruptStatsView CChipLiInterruptStatsView: columns: li_block: LI Block name: Interrupt Name interrupts: Number of Interrupts last_occurance: Last Occurrence
You can also define eval
in the Table
to calculate and add multiple key-values pairs, for example:
--- CChipDRDErrTable: command: show xmchip {{ instance }} drd error-stats args: instance: 0 target: NULL key: Interrupt Name item: '*' eval: cchip_drd_wan_errors: sum([v['interrupt_count'] for k, v in {{ data }}.items() if k.endswith('_0')]) cchip_drd_fab_errors: sum([v['interrupt_count'] for k, v in {{ data }}.items() if k.endswith('_1')]) view: CChipDRDErrView CChipDRDErrView: regex: cchip_drd_wan_timeouts: 'Total WAN reorder ID timeout errors:\s+(\d+)' cchip_drd_fab_timeouts: 'Total fabric reorder ID timeout errors:\s+(\d+)' columns: interrupt_name: Interrupt Name interrupt_count: Number of Interrupts filters: - interrupt_count
For information about using eval
in
a View, see Eval Expression (eval).
Table View (view)
The view
property associates the
Table definition with a particular View. A View defines how the Table
output should be parsed and maps your user-defined Python variable
names to output fields in the selected Table items. You can customize
the View to only select certain fields from the Table items.
If the output consists of only key-value pairs, you can use
the Table’s delimiter
parameter to
extract the data and store the key-value pairs in a dictionary. In
this case, you do not need to define a separate View.
TextFSM Templates (platform and use_textfsm)
Junos PyEZ Tables can reference a TextFSM template to parse
command output from Juniper Networks devices or other vendors’
devices. You must install the ntc-templates
library on your Junos PyEZ server or virtual environment
in order to use TextFSM templates in your Tables.
To use a TextFSM template to parse the command output, include use_textfsm: True
in the Table. You can use a TextFSM
template by itself or in conjunction with a Junos PyEZ View. Junos
PyEZ uses the platform
and command
values to determine the template’s filename.
For example, the following Table uses the juniper_junos_show_arp_no-resolve.textfsm template to parse command output from a Juniper Networks Junos device:
--- ArpTableTextFSM: command: show arp no-resolve platform: juniper_junos key: MAC use_textfsm: True
For detailed information about using Junos PyEZ Tables with TextFSM templates, see Use Junos PyEZ Tables with TextFSM Templates.