Machine learning - options

[e101sg]

Dear friends,

Greetings! I'm working on an adaptive honeypot proof of concept project. Elastic tools is helpful on the T-Pot. Elastic Machine Learning, however, is only free for 30 days. When our team contacted the Elastic sales team, they informed us that it was possible to enable Elastic ML on its own. (Need to purchase an expensive cloud-based solution, which not suitable for us).

We require some sort of classification using ML based on the logs and network activity from Honeypot. Any suggestions on ML tools or any alternative to Elastic ML ? Any thoughts, useful. Thanks a lot :)

Cheers!
Chandra

[lcia-projects]

This is tough.. i'm not going to say it cant be done.. but i havent seen anyone do it well. ML in IDS's and in threat detection is kind of one of the holy grails of cyber securty. All the big vendors are fighting for the same thing. I worked with a team of researchers at a major university in the US for about 2 yrs to create baselines of businesses so we could find abnormal network traffic. There were just too many variables and network traffic changes pretty quickly.

i'm not trying to dishearten you, but its not a simple problem.

[e101sg]

Yes, indeed it is a challenge. agree with you.
I am working in Proof -of -concept (POC) projects, so, may not be need commercial grade features but with option to scale. Our research team to use SELKS distro for Machine Learning along with T-POT honeypots.
(https://www.stamus-networks.com/blog/selks-7-an-introduction)

[southwestflavorz]

@e101sg did you find anything yet? I'm currently studying at school, and I'm planning on doing some research around ML and honeypots as well. Would love to start from somewhere.

[e101sg]

As mentioned in the above reply plan to use SELKS distro. i.e T-POT and SELKS all in same local LAN network. SELKS will collect all the data passing through T-POT.

Mainly i am using ZAT which convert ZEEK logs (earlier BRO) into Pandas, scikit-learn data frame. I am using it for anomaly detection. Please see here more details -- https://github.com/SuperCowPowers/zat
Hope useful. :)

[southwestflavorz]

Sorry if this is a silly question. Is there any conversion required to go from T-POT logs to Zeek logs? Or are you converting from SELKS to zeek logs?

[e101sg]

I mean the logs files = network traffic files. (PCAP). My use case is anomaly detection using ML. Capture all the traffic goes T-pot system using like SPAN port or other method. The pcap file converted into Zeek logs using ZAT. the ML system and T-port in same network.
So, to answer your question: pcap files converted to Zeek logs. Not using any other T-pot logs. Hope this helps.

[southwestflavorz]

One last silly question, are selk logs = zeek logs?

[e101sg]

No.

[lcia-projects]

these are my thoughts.. i'm no expert.. so i could be totally talking out of my ass. but i do have a little experience in this.

A few things:

• TPOT Honeypots: logs activity of the honeypots and suricata, not all network communications. so you'll get external scans, and attacks.

• Suricata: logs activity based on indicators and rules set up, but not all traffic

• Zeek: logs different kinds of network traffic, malicious and non-malicious, it can also handle rules and indicators as part of what it logs.

• PCAP: a recording of all network activity over a time period

• Zeek isnt integrated into TPOT (there has been some talk of incorporating it)

• Unlike the current implementation of TPOT22, zeek captures and logs all network communication.

• you could always install it on top of a tpot implentation

An approach you could take.. or a thought would be..
set up 2 networks.. two domains/sub domains for a business.
(1) - tpot/honeypot network- it would be attacked from the outside/external network and you gather those logs and throw them into your ML model.
(2) normal network - you'd need real people on this network .. or real network traffic.. real users.. browsing the web, email, etc. you'd be logging this traffic too.. it would also be attacked from the outside/external network..

do some kind of analysis comparing the two networks. one network, network (1) will be all network scans and attacks.. the other (network 2) will have good traffic too.

somehow use that for your model .

[e101sg]

Useful points. Lately find out that is Malcolm (https://github.com/cisagov/Malcolm) can be useful as well in this regard. It has Opensearch, Arkime, Logtash, Filebeat and Zeek etc.,