Pod logs are a major performance bottleneck, both due to enormous data transfers from the Kubernetes API and due to formatting being expensive. Before this patch, if multiple clients requested the same log simultaneously, these expensive operations were done multiple times in parallel. With this patch, mutexes are added around log retrieval and log formatting such that only one such operation can execute at any time.

In the multi-client scenario, the client arriving first will lock the mutex and retrieve the logs. When this client unlocks the mutex and another client is allowed through, the logs will already be cached and the request returns immediately.

Performance tests using the ab CLI tool confirm this: One of the requests still takes the same time as before, but all other requests are much faster.

Memory requirements are reduced from O(client count) to O(1).

In case multiple clients request different logs each, in the worst case, the response time may be slightly worse, but probably not by much on average:
Requesting logs in parallel from Kubernetes is likely not faster than requesting in series due to storage or network bottlenecks, and since Foreman is single-threaded, formatting in parallel is also not faster.

Additional Context

N/A

Checklist

• The pull request title meets the Conventional Commits specification and optionally includes the scope, for example: feat: Add social login