Rust telemetry to JSON files, instead of OTLP/gRPC

[graebm]

Lots going on here, apologies...

• Write telemetry to file, instead of exporting via OTLP/gRPC via opentelemetry-otlp crate.
  • Reasons for change:
    • Exporting via OTLP/gRPC was impacting noticeably performance. I tried to improve this by delaying any actual work until the benchmark run was complete (increasing buffer sizes, manually flushing buffers), but it still slowed max throughput from ~70 -> ~50 Gb/s.
    • It's just much simpler to spit out a file
  • Add our own telemetry exporter code, that writes OpenTelemetry Protocol (OTLP) in JSON format.
    • We started with code from an old version of the opentelemetry-stdout crate. Specifically, we took the commit just before the Simplify LogExporter::Export interface change. This change removed JSON export, so the code would be simpler for others to copy/paste. But we wanted JSON export, so copy/pasting from that point.
    • Modify the exporter so it doesn't do ANY work (buffering up all spans) until flush_to_file(filename) is called.
      • Hopefully, this eliminates the performance impact of gathering telemetry
      • This gives us a unique file per run. It's much nicer to analyze a run on its own, vs trying to isolate run 5 of 10 within an enormous file.
      • File is named like trace_20241009T185957Z_download-30GiB-1x-ram_run01.json
• Add python scripts to visualize the telemetry data
  • Start with allspans.py which graphs ALL tracing spans, each in its own row, similar to tools like Jaeger
    • Draw with plotly.express.timeline().
      • plotly was recommended by ChatGPT, I asked for a python graphing library where you could hide information until you mouse-over it. I also specified that files should be easy to share, and plotly generates a single HTML file.
    • The figure could use more work, but it's a start
  • More visualizations coming...

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[graebm]

Screenshot of small workload: 24MiB (3 parts) download:

Screenshot of large workload: 30GiB (3840 parts) download:

[waahm7]