Develop a pluggable logging API for Cacti to be able to log to alternate logging formats

[TheWitness]

Feature Request

Is your feature request related to a problem? Please describe

Support extensible logging to Splunk, Kafka, Elastic, etc.

Describe the solution you'd like

We need to determine the minimum logging attributes that are required to encompass these various logging solutions and then define the JSON, XML, etc. structures that are the minimum to perform such logging, and finally an API and class library that extends the output format using a pluggable and extensible output class function.

Describe alternatives you've considered

Continuing one size fit's all logging format.

Additional context

This is an R&D project that should deliver a report on what should happen vs. simply coding an enhancement. So, it's a call for a design document.

[bernisys]

Thoughts about capabilities and formats

The local logging must be unified

• each log line must have the same structure
• each log line must have the same minimum information
• log lines need to be easily extendable with additional information (for example for debugging)
• adding verbosity should not alter the start of the lines, so that they can still be parsed by existing functions
• a common log line format must be defined, and several elements need a static position and a separator that can be clearly identified
• suggestion for a local-file log format: "<timestamp> <section/facility>:<subsection> - <log message> -- <key/value pairs> == <optional info>"
• for standard syslog, we should try to match RFC5424 conformity (logs with structured data) -> https://datatracker.ietf.org/doc/html/rfc5424#section-6.5 example 3

Considerations for configuration

• make it flexible
• keep it simple (too many options might confuse the users - maybe add "expert" section?)
• global options to enable/disable specific elements per section
• log levels should be configurable at least per facility or per code-section
• optional information needs to be configurable (for example: line numbers and code portions are usually only needed for debugging)

Additional considerations:

• there are several logging functions in different parts of cacti, this needs some unification
• allow more detailed logging in a specific section, for example when there is something with boost -> switch on boost logging with this and that severity
• transfer in output format should be done as final step (keep the logging item in a data structure as long as possible, separated by identifiers mentioned below)
• we might need to think about the general concept of logging into different files. Is that really necessary or wouldn't it be better to keep everything in one file? And if we want to keep supporting multiple files, which information shall go into which file?

log items/identifiers

Minimum/mandatory set of information (identifiers)

• timestamp in epoch or UTC (local time conversion later)
• facility/section inside cacti (i.e. Spine, cmdphp, Thresholds, Core, Automate, SQL, ...)
• subsection (TBD)
• severity (similar to syslog: debug=7, info, notice, warning, error, critical, alert, emergency=0)
• message

Optional items (where applicable / during debugging)

• php-file in which the log was produced
• line number
• function name
• object that produced the log
• The message can include additional key/value pairs
  • let's keep the old bracket-notation (like: host_id[42])
  • if possible, make the keys identical to the DB-column names they refer to

Collection of possible (sub)sections:

• [main] GUI operations
• [main] Boost
• [main] Polling (spine + cmdphp)
• [main] workflow/controlling
• [sub] database accesses
• [sub] command calls
• [sub] remote poller communication
  @TheWitness @netniV please contribute here, you know the code and its structure better

support for output plug-ins

The logging function should act as a hub for plug-ins that allow logging to not only a local file but also other destinations.
Possible output plug-ins:

• local file (default plug-in to keep current functionality)
  • Config: file location(s)m, rotation settings [hourly/daily/size-based, compression, archive folder structure]
  • log-file should immediately have a creation-timestamp in its name
• local syslog (default plug-in to keep current functionality)
  • Config: facility
• remote syslog (optional plug-in)
  • Config: target-server, protocol, port, facility, CA
• Elasticsearch (optional plug-in, TBD)
• kafka bus (optional plug-in, TBD)
• other plug-ins (community-provided)

Thoughts about implementation phases

prepare the refactoring

• create a new specific library-file: /lib/logging.php
• put all new logging-related functions into that file
• create new core-function which must be used in future for all logging purposes
• create a set of logging configuration in "settings" table: logging_ =
• create new GUI config elements for the new logging settings + code and sanity checks around them
• keep the current logging-function available (old plug-ins should be supported as long as possible)
  • use the current code as source for analyzing the logging internals
  • rebuild the basic functionality in the new function, add all planned features
  • make the old code call the new code
  • old code then acts as meta-code for the transition to new logging
  • think about expiring old logging at some point in future

reforming the logging concept

• create logging initialization procedure
  • if possible/applicable, called only once during start of the instance
  • initializes all logging-related variables
  • checks if log-files & folders are available and creates them if needed
  • long running processes might need a reload/re-init functionality to allow change of log levels (for example during a boost run)
• include the new core logging lib file everywhere, where logging is needed (which is basically everywhere)
• Start refactoring all "cacti_log" calls, replace them with the new core lib calls

[bernisys]

Whoever has ideas that need to be considered, please speak up! :)
This is an open discussion to take into account as many thoughts as possible from the start.
I will keep adding the information in compact form to my initial post.