Logging: combine structured logging with nice human-readable messages

[julianbrost]

Currently, we do logging a bit inconsistent across projects. For example, in Icinga DB, we often use format strings to add values to log messages, for example:

r.logger.Infof("Upserted %d %s items", count, s.Name())

On the other hand, in Icinga Notifications, we mostly have static messages and include everything dynamic in the additional key value fields, for example:

i.logger.Infow("Successfully sent a notification via channel plugin", zap.String("type", ch.Type),
	zap.String("contact", contact.FullName), zap.String("event_type", ev.Type))

Both ways have advantages and disadvantages: the first produces nice human-readable messages but doesn't really follow the idea of structured logging (what the underlying zap library we use is intended for). The latter is better for machine-readability but when viewed using for example journalctl, you currently might need workarounds to get a full picture of what the log message is trying to say.

Now comes the idea that's floating around in my head: why can't we have the good parts of both? In my head, that could look something like that (the "a" in Infoa stands for "amazing"):

logger.Infoa("Successfully notified {contact} via {channel}",
	zap.String("channel", ch.Type),
	zap.String("contact", contact.FullName),
	zap.String("event_type", ev.Type))

So the idea would be to still have everything that's dynamic in structured key value pairs but additionally have placeholders in the message that will be replaced with the corresponding values to get a nice human-readable message as well.

This is partially inspired by what the Windows Event Log can do, however, there it's very cumbersome to use so that it's not done frequently, but there you can pretty much log a message ID and values for placeholders and then provide a file with the message (which even allows you to view the logs in a different language when done properly):

MessageId=0x103
Severity=Warning
Facility=System
SymbolicName=MSG_RETRYS
Language=English
There have been %1 retrys with %2 success! Disconnect from
the server and retry later.
.

Note: This issue isn't supposed to say "please implement an Infoa function exactly as I suggested". It's a very broad suggestion, feedback and ideas how this could be implemented are welcome.