[pkg/stanza] Prevent data loss due to LogEmitter's buffer

[andrzej-stencel]

Component(s)

pkg/stanza

Is your feature request related to a problem? Please describe.

This issue is created as a result of discussion in #31074.

During an non-graceful shutdown of the collector, logs kept in the LogEmitter's buffer (the batch field) are lost. This is because they are not persisted and are already marked as sent by the File consumer.

Describe the solution you'd like

Remove the buffer in the LogEmitter and make LogEmitter synchronously emit every log/batch of logs received down the collector pipeline.

This will likely introduce a performance impact if implemented without other changes. This should only be done after:

• Measuring the possible performance impact: [pkg/stanza] Compare performance with and without buffer in LogEmitter #35454
• Possibly implementing buffering earlier in the Stanza pipeline (in File consumer in case of Filelog receiver): [pkg/stanza/fileconsumer] Emit logs in batches #35455
• Considering buffering earlier in the Stanza pipeline for receivers other than those relying on file consumer (especially Windows Event Log receiver, Syslog receiver) - issues to be created after validating the path in the two issues above

Describe alternatives you've considered

• Instead of removing the buffering, have it persisted so that the data is not lost during non-graceful shutdown.
• Change the logic of marking logs as sent, so that logs are only marked as sent when they are actually successfully sent out to next consumer in the collector pipeline.

Additional context

See discussion in #31074 (comment) and further comments.

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.