[Merged by Bors] - Add demo data-warehouse-iceberg-trino-spark

[sbernauer]

Description

Needs a larger k8s cluster! I use IONOS k8s with 9x 4 cores (8 threads), 20GB ram and 30GB hdd disk
Maybe we can also offer a smaller variant later on.

Otherwise business as usual. From feature-branch run stackablectl --additional-stacks-file stacks/stacks-v1.yaml --additional-releases-file releases.yaml --additional-demos-file demos/demos-v1.yaml demo install data-warehouse-iceberg-trino-spark

I'm not happy with some parts but i think an iterative approach is best:

• Shared bikes are currently not streamed into Kafka (instead on-time job)
• Some high-volume real-time datasource would be great. Currently we use the water levels and duplicate them to get higher volumes.
• Some sort of Upsert or Deletion usecases would be great. But probably not on the large datasets as for our wallet ^^
• Better Dashboards. The current one were thrown together quickly
• I would like to partitions the water_level measurements by day but run into Structured Streaming writes to iceberg table with non-identity partition spec breaks with spark extensions enabled apache/iceberg#5625. There might be ways around by using a dedicated Spark context for compaction but we can easily adopt the partitioning after the issue gets fixed. Sorting during rewrites did cost performance during compaction and did not provide real benefits for my intial measurements. Disabled for now.
• As always tracked my findings in [Tracker] Findings of demos demos#15

To get to the Spark UI kubectl port-forward $(kubectl get pod -o name | grep 'spark-ingest-into-warehouse-.*-driver') 4040

Review Checklist

• Code contains useful comments
• (Integration-)Test cases added (or not applicable)
• Documentation added (or not applicable)
• Changelog updated (or not applicable)
• Cargo.toml only contains references to git tags (not specific commits or branches)

Once the review is done, comment bors r+ (or bors merge) to merge. Further information

[maltesander]

Nice. LGTM!

[sbernauer]

It kept running over the weekend. We are now at

|files|total_size     |avg_size   |total_records |avg_records|smallest_file|
|-----|---------------|-----------|--------------|-----------|-------------|
|298  |150,318,197,593|504,423,481|36,430,087,583|122,248,616|7,101,325    |

[sbernauer]

@maltesander could you please do another round reviewing the docs?

[maltesander]

Just english nitpicking!

[maltesander]

LGTM!

[sbernauer]

Many many thanks!

[sbernauer]

bors r+

[bors]

Pull request successfully merged into main.

Build succeeded:

• Run Rustfmt