New Relic Flow Collection Quickstart

Pre-reqs

Validate you have docker installed and running (docs)

docker version --format '{{.Server.Version}}'

Validate you have a non-root user available in the docker group (docs)

grep -e "docker" /etc/group

Get your New Relic Account ID and Insights Insert Key, replacing $NR_INSIGHTS_INSERT_KEY and $NR_ACCOUNT_ID in these commands, respectively.

Listen for Flow telemetry directly and send to New Relic:

Step 1

Download the ktranslate image from dockerhub:

docker pull kentik/ktranslate:v2

Step 2

Build a new snmp-base.yaml file with your flow devices annotated in the devices config block, setting the flow_only attribute to true in order to give ktranslate a device_name to use for your flow devices instead of using their source IP.

You can also add these devices to a pre-existing snmp-base.yaml file if you're running ktranslate in SNMP mode on the same host

If your device is already defined for SNMP collection, no further configuration is required and you do not need the flow_only attribute set.

Example:

devices:
  flowDevice:
    device_name: edge-router
    device_ip: 10.10.1.254
    flow_only: true
    user_tags:
      owning_team: net_eng
      environment: production

Step 3

Select which type of flow you want to listen to. Options are ipfix, netflow5, netflow9, sflow. (For Juniper's jFlow, use netflow5)

Start ktranslate to run in background and listen for flow. ktranslate containers can only listen for a single flow type. If you need to handle multiple types, each will require their own container

This will require you have your New Relic Account ID and Insights Insert Key, replacing $NR_INSIGHTS_INSERT_KEY and $NR_ACCOUNT_ID in this command, respectively.

For sizing, it is recommended to have 1 CPU core available for every 2,000 flows per second (fps) to be processed. Replace $CORE_COUNT with your preferred value.

• nf.source is the type of flow you are listening for
• nf.port is the port number you are sending flows to
• flow_only=true is the setting which prevents duplicate SNMP polling against devices in case they are already defined for a ktranslate container dedicated to SNMP collection

Example for sFlow:

docker run -d --name ktranslate-sflow --restart unless-stopped --net=host \
 -v `pwd`/snmp-base.yaml:/snmp-base.yaml \
 -e NEW_RELIC_API_KEY=$INSERT_API_KEY \
 kentik/ktranslate:v2 \
  -snmp /snmp-base.yaml \
  -nr_account_id=$NR_ACCOUNT_ID \
  -metrics=jchf \
  -log_level=info \
  -tee_logs=true \
  -flow_only=true \
  -nf.source=sflow \
  nr1.flow

Advanced Options:

• -max_threads -- set this to allow ktranslate to process higher volumes of flow. A rule of thumb is 1 thread for every 2k flows per second (fps) of flow sent.
• -sample_rate=100 -- set this to change the default sample rate at which flows are passed to New Relic events and/or set the sample rate for flow types that don't expose their rate by default
• -nf.workers=1 -- set this to override the number of workers used in processing udp packets. Use 1 worker for every 4k of flows per second (fps) of flow sent.
• -nf.port -- set this to override the default 9995 listening port for incoming flow packets
• -listen -- set this to override the default 8082 port used by ktranslate to expose health metrics (used when running multiple ktranslate containers on a single host)
• -metalisten -- set this to override the default 8083 port used by ktranslate to expose metadata (used when running multiple ktranslate containers on a single host)
• -metrics=jchf -- set this to forward health metrics from ktranslate into New Relic
• -log_level -- set this to override the default info log level for ktranslate; options: debug, info, warn, error
• -tee_logs -- set this to forward logs from ktranslate into New Relic Logs

Multiple Container Example:

docker run -d --name ktranslate-sflow --net=host \
 -e NEW_RELIC_API_KEY=$INSERT_API_KEY \
 kentik/ktranslate:v2 \
  -snmp /snmp-base.yaml \
  -nr_account_id=$NR_ACCOUNT_ID \
  -metrics=jchf \
  -log_level=info \
  -tee_logs=true \
  -nf.source=sflow \
  -nf.port=9996 \
  -max_threads=1 \
  -listen 0.0.0.0:8084 \
  -metalisten 0.0.0.0:8085 \
  nr1.flow


docker run -d --name ktranslate-netflow9 --net=host \
 -e NEW_RELIC_API_KEY=$INSERT_API_KEY \
 kentik/ktranslate:v2 \
  -snmp /snmp-base.yaml \
  -nr_account_id=$NR_ACCOUNT_ID \
  -metrics=jchf \
  -log_level=info \
  -tee_logs=true \
  -nf.source=netflow9 \
  -nf.port=2055 \
  -max_threads=2 \
  -listen 0.0.0.0:8084 \
  -metalisten 0.0.0.0:8085 \
  nr1.flow

A note on SNMP collection

There are various supported configurations available for running ktranslate against devices for both SNMP and Flow collection. The usage of the flow_only attribute on both a container level and in the devices section of the snmp-base.yaml config can be summarized as follows:

• If you have multiple containers collecting both SNMP and Flow with ktranslate, you'll define your devices in snmp-base.yaml following the standards for SNMP polling and they will be automatically matched to their respective flows based on their device_ip matching the sampling IP for the flow device.
  • Flow Container: -flow_only=true
  • snmp-base.yaml: No setting changes needed
• If you have multiple containers collecting both SNMP and Flow with ktranslate, and you have specific devices where you only want Flow collection and no SNMP polling, you can set them up in their respective device config snippet.
  • Flow Container: -flow_only=true
  • snmp-base.yaml: flow_only: true
• If you are only running containers for Flow collection with ktranslate and are not using any dedicated SNMP containers, you'll use the same configuration as above.
  • Flow Container: -flow_only=true
  • snmp-base.yaml: flow_only: true