Can not decode netflow data

[dbblackdiamond]

Hi,

I have just discovered goNfCollector and had high hopes to use it as my netflow collector. My router/firewall is a RPI4 running OpenWrt 21.02.1 r16325-88151b8303. I installed softflowd version 1.0.0.2 on it as my netflow exporter and configured it to send the flows to goNfCollector running in docker.
As soon as I start the goNfCollector docker container, I can see the following error messages in the log:

ERROR[2022-11-02T03:40:31Z] File: collector.go Function: github.com/goNfCollector/collector.(*Collector).collect Line: 282 [6]-Can not decode
 netflow data: (EOF)  debug="File: collector.go Function: github.com/goNfCollector/collector.(*Collector).collect Line: 282"

I followed the instructions given in a different issue about downloading the nfcollector-logger program and running it. I did and this is the output:

2022/11/02 10:16:31 received 904 bytes from 192.168.1.1:48779
2022/11/02 10:16:31 decoder error: EOF

Additional information is supposed to be contained in the /tmp/nfcollector-logger.log but it is empty, even after I CTRL-C the nfcollector-logger program.

I am not sure what it happening here and why goNfCollector can't decode the netflow data coming from softflowd.
Thanks a lot in advance for your help,
Bertrand.

[javadmohebbi]

Hi @dbblackdiamond

What is the version your trying to export using softflowd?
Please change the version to 5 and reproduce the problem and let me know if version 5 fixed your issue!

[dbblackdiamond]

I am currently using v9, so let me change it to v5 and report back.

[dbblackdiamond]

I have switched it to v5 and it seems to be working. I am going to let it run for a bit and if it works fine, I will close the issue. Any plan on supporting version 9?

[javadmohebbi]

Ok, let me know the result.

It already supports Version 9, getting the "can not decode" error message, for a short period of time after starting the solution, is a normal behavior, because it might be a flow records from the previous time from your softflowd. So just wait for a half of a minute or a minute to see if could collect v9 flows.

[dbblackdiamond]

Regarding getting the "can not decode" error message and getting it being normal for a short period after starting the solution, I thought it was normal, hence why I let it run for the whole night and was still seeing it after 12 hours of continuous running time and I was still getting it after that period of time.

[javadmohebbi]

Hi @dbblackdiamond

Thank you for submitting issue. To investigate the issue, we need to run the following commands to gather some logs from our application.

1- You need to stop the netflow collector docker images

cd $HOME/oi24/nfcollector
docker-compose down

2- Download the following logger tool and run int, wait form some minutes and after that, a file located in /tmp/nfcollector-dump.log (by default) will be created. You can change it using -out /path/to/your/log/file. Also if you are listening on different address other than default port :6859, you can change it using -addr command line option.

wget -O /tmp/nfcollector-logger https://github.com/javadmohebbi/goNfCollector/raw/main/build/linux/amd64/nfcollector-logger
chmod +x /tmp/nfcollector-logger
sudo /tmp/nfcollector-logger

This file might include some confidential information like IP addresses, You can replace the confidential IP Addresses and the share it with me.

Regards

[newpipe]

Hi,
I'm having exactly the same issue on an OpenWRT 22.03.2 with softflowd 1.0.0.
I followed the steps described in your most recent post to analyze the problem.
However, the file nfcollector-dump.log stays empty.

This is what you can see from the shell:

root@mailcow:~/oi24/nfcollector# /tmp/nfcollector-logger 2023/01/06 13:24:30 Creating log file in this path: /tmp/nfcollector-dump.log 2023/01/06 13:25:01 received 1024 bytes from 192.168.2.1:36574 2023/01/06 13:25:01 decoder error: EOF 2023/01/06 13:25:01 received 556 bytes from 192.168.2.1:54391 2023/01/06 13:25:01 decoder error: EOF 2023/01/06 13:26:01 received 1336 bytes from 192.168.2.1:54391 2023/01/06 13:26:01 decoder error: EOF 2023/01/06 13:26:01 received 580 bytes from 192.168.2.1:54391 2023/01/06 13:26:01 decoder error: EOF 2023/01/06 13:26:01 received 1320 bytes from 192.168.2.1:36574 2023/01/06 13:26:01 decoder error: EOF 2023/01/06 13:26:01 received 548 bytes from 192.168.2.1:36574 2023/01/06 13:26:01 decoder error: EOF 2023/01/06 13:27:01 received 724 bytes from 192.168.2.1:54391 2023/01/06 13:27:01 decoder error: EOF 2023/01/06 13:27:01 received 104 bytes from 192.168.2.1:36574 2023/01/06 13:27:01 decoder error: EOF 2023/01/06 13:28:01 received 328 bytes from 192.168.2.1:54391 2023/01/06 13:28:01 decoder error: EOF 2023/01/06 13:28:01 received 84 bytes from 192.168.2.1:36574 2023/01/06 13:28:01 decoder error: EOF ^C root@mailcow:~/oi24/nfcollector# cat /tmp/nfcollector-dump.log

You can clearly see, that nfcollector-logger is receiving data from softflowd, it also creates the file nfcollector-dump.log. But it doesn't write anything to it.

Edit:
Just to add: When switching to version 5 in softflowd everything is working. But I need version 9, because I need to track IPv6 traffic.

[jimbos]

I got the same issues

[jult]

All this is such a mess. I ended up going with nfdump and nfsen_ng and its IPFIX/Netflow input setup. Works like a charm, waaaay less buggy than all the docker overhead and dependency hell this creates. Hints:
https://github.com/mbolli/nfsen-ng
https://github.com/phaag/nfdump (this is even in debian 12 core repo)