Geolocation database outside the docker container i.e. in /data

[rocbr]

A possible improvement for the T-POTCE project would be to store the MaxMind Geolocation databases outside of the docker container, in the host machine's /data folder. This would make it easier to update these files, which are currently outdated by over a year.

Since IPv4 addresses are known to change location frequently, it is important to keep these databases up-to-date to ensure accurate geolocation data.

By storing them outside of the container, updates can be easily managed without having to rebuild the entire container or changing configuration files.

[t3chn0m4g3]

That is something to take into consideration. Does the Maxmind license model allow shipping their databases as part of an Open Source project?

[rocbr]

Reading the article available at "GeoLite2 Free Geolocation Data"
https://dev.maxmind.com/geoip/geolite2-free-geolocation-data, it seems that all you need to do is link to the MaxMind website to provide a product reference as described in https://dev.maxmind.com/geoip/geolite2-free-geolocation-data#license

The databases can be downloaded directly via a link with an API-KEY for each user. I'm not sure if it would be possible to automate this with the Docker build or through the installation shell script. So It can be an issue. Requiring each user to have an API-KEY might not be the best idea. It also doesn't seem ideal to embed your API key into the install script. The optimal approach would be to enable automatic downloading of the databases during the initial installation.

Database GeoLite2-ASN URL
https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-ASN&license_key=YOUR_LICENSE_KEY&suffix=tar.gz

GeoLite2-City
https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOUR_LICENSE_KEY&suffix=tar.gz

As an alternative, in this GitHub project there are direct download links without the need for keys. The databases appear to be up-to-date.
https://github.com/P3TERX/GeoLite.mmdb

[t3chn0m4g3]

Thank you, I will whip something up based on your research for the Tech Preview of the next version of T-Pot (currently dubbed T-Pot Light).
We have API keys already an place for Suricata so I will probably adapt this over.

[rocbr]

Awesome

[t3chn0m4g3]

So, turns out that logstash changed the inner works of the geoip plugin, so downloads of the geoip db can be automated by simply not setting the geoip db location 😁:

Seems to work fine on my end, will be pushing a new logstash image later ...

[INFO ] 2023-05-07 16:03:36.395 [[logstash]-pipeline-manager] downloadmanager - new database version detected? true
[INFO ] 2023-05-07 16:03:39.636 [[logstash]-pipeline-manager] databasemanager - By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[INFO ] 2023-05-07 16:03:39.636 [[logstash]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/data/plugins/filters/geoip/1683475415/GeoLite2-City.mmdb"}
[INFO ] 2023-05-07 16:03:41.648 [[logstash]-pipeline-manager] databasemanager - By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[INFO ] 2023-05-07 16:03:41.648 [[logstash]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/data/plugins/filters/geoip/1683475415/GeoLite2-City.mmdb"}
[INFO ] 2023-05-07 16:03:41.695 [[logstash]-pipeline-manager] databasemanager - By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[INFO ] 2023-05-07 16:03:41.695 [[logstash]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/data/plugins/filters/geoip/1683475415/GeoLite2-City.mmdb"}
[INFO ] 2023-05-07 16:03:41.736 [[logstash]-pipeline-manager] databasemanager - By not manually configuring a database path with `database =>`, you accepted and agreed MaxMind EULA. For more details please visit https://www.maxmind.com/en/geolite2/eula
[INFO ] 2023-05-07 16:03:41.736 [[logstash]-pipeline-manager] geoip - Using geoip database {:path=>"/usr/share/logstash/data/plugins/filters/geoip/1683475415/GeoLite2-City.mmdb"}

[t3chn0m4g3]

Fixed with f438be7 and new images have been pushed.