Skip to content

Tutorial: Writing a custom yaml file for SNMP

Jump to bottom
Ian edited this page Aug 27, 2021 · 10 revisions

Tutorial: Write a custom yaml file describing a MIB.

This page will walk through creating a yaml file used for polling a custom set of oids. Ktranslate ships with a built in set of these at https://github.com/kentik/snmp-profiles/tree/main/profiles/kentik_snmp.

As a base, we will be using Palo Alto Network's PAN-COMMON-MIB.

Each yaml file has three top level sections: extends, sysobjectid, and metrics.

Extends

extends lists the other yaml files which this file inherits. List only the file name. A runtime error will occur if a listed file is not found. For example:

extends:
  - system-mib.yml
  - if-mib.yml

Will add in all mibs found in system-mib.yml and if-mib.yml

Sysobjectid

sysobjectid lists which system object ids will be matched to this profile. They can be specified either exactly:

sysobjectid: 1.3.6.1.4.1.9.1.111

As a list:

sysobjectid:
  - 1.3.6.1.4.1.9.1.111
  - 1.3.6.1.4.1.9.1.170

Or using a wildcard:

sysobjectid: 1.3.6.1.4.1.25461.*

Wildcards will match all oids with the value before the * as a prefix.

Metrics

metrics define the actual oids to poll. The simplest section here lists oids one by one with a name:

metrics:
  - MIB: PAN-COMMON-MIB
    symbol:
      OID: 1.3.6.1.4.1.25461.2.1.2.3.1.0
      name: panSessionUtilization
  - MIB: PAN-COMMON-MIB
    symbol:
      OID: 1.3.6.1.4.1.25461.2.1.2.3.2.0
      name: panSessionMax

Ktranslate will poll each of these oids and create metrics called kentik.snmp.panSessionUtilization and kentik.snmp.panSessionMax

Tables are supported as follows:

  - MIB: PAN-ENTITY-EXT-MIB
    table:
      OID: 1.3.6.1.4.1.25461.1.1.7.1.2.1
      name: panEntityFRUModuleTable
    symbols:
      - OID: 1.3.6.1.4.1.25461.1.1.7.1.2.1.1.1
        name: panEntryFRUModulePowerUsed
      - OID: 1.3.6.1.4.1.25461.1.1.7.1.2.1.1.2
        name: panEntryFRUModuleNumPorts
    metric_tags:
      - MIB: ENTITY-MIB
        column:
          OID: 1.3.6.1.2.1.47.1.1.1.1.2
          name: entPhysicalDescr
        table: entPhysicalTable
        tag: entity_description

This will create metrics kentik.snmp.panEntryFRUModulePowerUsed and kentik.snmp.panEntryFRUModuleNumPorts. Furthermore, the attribute entPhysicalDescr will be added to these metrics.

The final result for this mib is at https://github.com/kentik/snmp-profiles/blob/main/profiles/kentik_snmp/palo_alto/palo-alto.yml.

Enums

Some snmp values come back as enumerated integers. For example, look at http://oidref.com/1.3.6.1.4.1.318.1.1.12.2.3.1.1.3 which describes a PDU. From the description:

Getting this OID will return the phase/bank load state.

phaseLoadNormal(1) indicates that the phase/bank is
operating properly within the rPDULoadConfigLowLoadThreshold
and rPDULoadConfigNearOverloadThreshold OID values.

To encode this in the yaml file, add an enum block:

      - OID: 1.3.6.1.4.1.318.1.1.12.2.3.1.1.3
        name: rPDULoadStatusLoadState
        enum:
          phaseLoadNormal: 1
          phaseLoadLow: 2
          phaseLoadNearOverload: 3
          phaseLoadOverload: 4

This creates the metric kentik.snmp.rPDULoadStatusLoadState. It will be a gauge with a value of 1-4. There will also be an attribute rPDULoadStatusLoadState which is the string version of this value.

Interface or Device?

Metrics are separated into interface and device level one. Interface level metrics have names which start with if. As a user, these are treated the same.

Tags

Sometimes you will want to change the name of the metric. Use a tag: field to accomplish this. For example:

      - OID: 1.3.6.1.4.1.9.9.109.1.1.1.1.7
        name: cpmCPUTotal1minRev
        tag: CPU

Rather than creating a metric kentik.snmp.cpmCPUTotal1minRev, the metric will be kentik.snmp.CPU.

Hex Encoding

SNMP values are sometimes encoded as a hexadecimal string. These need to be explicitly converted. Set a conversion field to supply the needed information. Example:

      - column:
          OID: 1.3.6.1.2.1.75.1.2.1.1.1
          name: portIndex
        tag: port_index
        conversion: hextoint:BigEndian:uint16

The format is hextoint:BigEndian|LittleEndian:uint64|uint32|uint16.

Running

Now you have your awesome-custom-mib.yaml file. Add it to a directory my_custom_profiles and launch ktranslate with a new docker flag:

-v `pwd`/my_custom_profiles:/etc/profiles/my_custom_profiles

For example:

docker run -ti --name ktranslate --rm --net=host \
  -v `pwd`/my_custom_profiles:/etc/profiles/my_custom_profiles
  -v `pwd`/snmp-base.yaml:/snmp-base.yaml \
  kentik/ktranslate:v2 \
    -snmp=/snmp-base.yaml \
    -log_level=info \
    -format=json

Don't forget to add any new mibs into your snmp-base.yaml file in the mibs_enabled: section.

Verify that these are working by first looking for the log line like

2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate Found profile for 1.3.6.1.4.1.9.12.3.1.3.1812: cisco-nexus.yml

This should list both the sysoid used and yaml file matched. Next, you'll see a list of custom metadata:

2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Custom device metadata
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.1.2.0 -> sysObjectID
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.1.5.0 -> sysName
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.1.1.0 -> sysDescr
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Custom interface metadata
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.31.1.1.1.15 -> ifHighSpeed
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.31.1.1.1.1 -> ifName
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart    -> : 1.3.6.1.2.1.31.1.1.1.18 -> ifAlias

Metadata are the values defined in a metric_tags section.

Lastly, you will see:

2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding custom interface metadata oid: 1.3.6.1.2.1.31.1.1.1.1 -> ifName
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding interface metric .1.3.6.1.2.1.31.1.1.1.13 -> ifHCOutBroadcastPkts
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding interface metric .1.3.6.1.2.1.2.2.1.8 -> ifOperStatus
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding interface metric .1.3.6.1.2.1.2.2.1.13 -> ifInDiscards
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding interface metric .1.3.6.1.2.1.2.2.1.20 -> ifOutErrors
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Using custom interface metric set
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding device metric .1.3.6.1.4.1.9.9.305.1.1.1 -> CPU
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding device metric .1.3.6.1.4.1.9.9.305.1.1.2.0 -> MemoryUtilization
2021-07-20T15:29:31.542 ktranslate(294299) [Info] KTranslate>bart Adding device metric .1.3.6.1.2.1.7.8.0 -> udpHCInDatagrams

This is the applied set of metrics which will be polled.

yaml or yml?

Doesn't matter! Pick one and stick to it.

Auto-generation

If you have an existing .MIB file, ktranslate can automatically convert it to a yaml version. Follow these steps:

  • Install pysmi. (pip install pysmi)

  • Install librenms. (git clone https://github.com/librenms/librenms.git)

  • Download your MIB target file. Place into /tmp/snmp_in directory. Create /tmp/snmp_out directory also. Use ZSCALER-NSS-MIB as a test if you need one.

  • Run mibdump:

mibdump.py --mib-source=file:///path/to/src/librenms/mibs --mib-source=file:///tmp/snmp_in --generate-mib-texts --ignore-errors  --destination-format json --destination-directory=/tmp/snmp_out MY_MIB_NAME

For example, using mib ZSCALER-NSS-MIB and assuming that librenms is at /home/me/src/librenms:

mibdump.py --mib-source=file:///home/me/src/librenms/mibs --mib-source=file:///tmp/snmp_in --generate-mib-texts --ignore-errors  --destination-format json --destination-directory=/tmp/snmp_out ZSCALER-NSS-MIB

  • Verify that /tmp/snmp_out/ZSCALER-NSS-MIB.json exists

  • Convert to yaml

docker run --rm -v /tmp/snmp_out:/snmp_out kentik/ktranslate:v2 -snmp /etc/ktranslate/snmp-base.yaml -snmp_json2yaml /snmp_out/MY_MIB_NAME.json

For example, using ZSCALER-NSS-MIB:

docker run --rm -v /tmp/snmp_out:/snmp_out kentik/ktranslate:v2 -snmp /etc/ktranslate/snmp-base.yaml -snmp_json2yaml /snmp_out/ZSCALER-NSS-MIB.json
  • The final yaml file is at /tmp/snmp_out/ZSCALER-NSS-MIB.yaml.

NOTE: human checking is still needed. Notably, you will need to:

  1. Add an extends: section.
  2. Ensure that the sysobjectid: section is present and sane.

SNMP Overview

Flow Overview

Clone this wiki locally