Describe Python virtualenv functionality

[bazsi]

We should have a chapter on generic Python manners (virtualenv, PYTHONPATH, pypi, etc). The closest I could find was

https://axoflow.com/docs/axosyslog-core/chapter-configuration-file/python-code-external-file/
and
https://axoflow.com/docs/axosyslog-core/chapter-configuration-file/python-code-logging/

Maybe we should collide these into a Python chapter with the title "Using Python to syslog-ng" or something like that, then the current two chapters could fit there nicely.

What I also wanted to add is a section about Python virtualenvs and dependencies, which came up in a discussion with @ryanfaircloth

Actually the topic could possibly split into AxoSyslog Core and AxoSyslog specific hints, the first being more general to distro related use-cases while AxoSyslog more about the specific use-case of using syslog-ng in a container.

With all that said here comes a braindump.

Intro

• Python code can be used to extend syslog-ng functionality (sources, destinations, parsers, filters, template functions)
• User supplied Python code can either be stored in the syslog-ng configuration file (enclosed in python { } blocks) or in Python modules stored outside of the configuration (and then referenced using the class() option). User supplied modules live either in /etc/syslog-ng/python or any directory on the PYTHONPATH, which needs to be set before syslog-ng is launched.
• syslog-ng also ships some built in functionality that is written in Python, these live in /usr/lib/syslog-ng/python. You should not add user supplied Python code into this directory.

External Dependencies

In simple cases syslog-ng will manage its own set of requirements so you are able to use its Python backed functionality without doing anything.

But in more advanced cases you might need to use additional dependencies that you need to make available for the Python code embedded into syslog-ng.

• Either user supplied or syslog-ng supplied Python code can use official Python modules published on PyPI or part of the Python distribution that you use. For example the kubernetes() source uses python code that in turn uses the "kubernetes" package from PyPI
• these dependencies need to be deployed before syslog-ng is started (at least if you intend to use these Python based components).
  • the deployment of these dependencies can be managed two ways
    • using the Linux/Python distribution's packaging tools (e.g. dpkg, dnf/yum, conda or pip), in which case the required python packages will be deployed into the system's Python package directories and thus be accessible for syslog-ng. OR
    • using a dedicated Python virtualenv for syslog-ng's own use, which is the recommended practice, unless you are a maintainer for a distribution.
• syslog-ng automatically initializes a virtualenv for its own use and for its own built-in functionality. This is usually initialized at postinst time (if you are using our upstream packaging) but you can also re-initialize it any time after installation by running the syslog-ng-update-virtualenv script. The list of modules deployed into this virtualenv is controlled by the /usr/lib/syslog-ng/python/requirements.txt file.
• if your own Python code requires dependencies in addition to what syslog-ng automatically adds to its own virtualenv, you have the following options:
  • use "/var/lib/syslog-ng/python-venv/bin/pip install " manually after syslog-ng finished its initialization to add your extra PyPI modules, OR
  • create your own dedicated virtualenv anywhere in the filesystem and activate the virtualenv using its activate script (e.g. ". $venv/bin/activate"
  • rely on syslog-ng to automatically activate the virtualenv by setting VIRTUAL_ENV environment variable to point this directory

watch for the following log message as syslog-ng starts up:

python: activating virtualenv; path='/install/var/python-venv', executable='/install/var/python-venv/bin/python'

Container specific notes

This could probably go to AxoSyslog instead of the lower level AxoSyslog Core.

If you are creating your container images based on AxoSyslog and your use-case relies on Python based functionality, the best practice to package your Python code along with syslog-ng is:

• copy your Python code as modules to /etc/syslog-ng/python
• create a virtualenv if your choice somewhere on the filesystem (could go to /etc/syslog-ng/python-venv), which you can initialize using standard python tools (e.g. python -m venv). Install the combination of your dependencies and syslog-ng's own set using pip: /etc/syslog-ng/python-venv/bin/pip install -r /usr/lib/syslog-ng/python/requirements.txt -r my-requirements.txt
• use the ENV command in your Dockerfile to specify the value of VIRTUAL_ENV: ENV VIRTUAL_ENV /etc/syslog-ng/python-venv