runtime benchmark suite

[onur-ozkan]

Motivation

KDF supports quite numerous of different targets and platforms. While this is cool, it's also introduces significant challenges from the development perspective. Identifying platform-specific performance regressions on dev branch is nearly impossible because unless we manually test the project, we cannot identify the performance impacts.

Solution

As a solution, we can use conditional compilation (e.g., adding the rt-benchmark compilation feature) without adding runtime overhead on prod releases to store detailed benchmark reports for nearly all core areas of the project. These reports would include metrics such as the time t (a function) was called, the number of times t called, the execution time spent on t and the resources utilized by t. Ideally we can use JSON format for storing these metrics, so it will be easy to parse and compare it from any environment.

This would enable convenient use of these reports both in CI environments (e.g., generating these reports in PR pipelines and comparing them with the target branch) and by developers.