Benchmark Utils for Structured streaming

we wanted to have performance benchmarks for various scenarios as part of the RocksDb State Storage implementation SPARK-28120

Jungtaek Lim pointed me to his github project which he has used for his benchmarks. I have created this project in similar lines for streaming performance scenarios.

Build the project

    sbt assemebly

Use RocksDB State Storage

    /usr/lib/spark/bin/spark-submit \
    --class com.qubole.spark.benchmark.streaming.states.StateStoreBenchmarkRunner \
    --driver-memory 2g \
    --executor-memory 7g \
    --num-executors 1 \
    --max-executors 1 \
    --executor-cores 4 \
    --conf spark.executor.memoryOverhead=3g \
    ./build/spark-benchmark.jar \
    --query-status-file "/tmp/queryStatus-rocksdb" \
    --rate-row-per-second "20000" \
    --output-mode "append" \
    --run-time-in-sec 1800 \
    --shuffle-partition 8 \
    --use-rocks-db

Use Memory State Storage

    /usr/lib/spark/bin/spark-submit \
    --class com.qubole.spark.benchmark.streaming.states.StateStoreBenchmarkRunner \
    --driver-memory 2g \
    --executor-memory 7g \
    --num-executors 1 \
    --max-executors 1 \
    --executor-cores 4 \
    --conf spark.executor.memoryOverhead=3g \
    ./build/spark-benchmark.jar \
    --query-status-file "/tmp/queryStatus-memory" \
    --rate-row-per-second "20000" \
    --output-mode "append" \
    --run-time-in-sec 1800 \
    --shuffle-partition 8 \

Analyze the progress

    /usr/lib/spark/bin/spark-submit \
    --class com.qubole.spark.benchmark.streaming.states.AnalyzeProgress \
    --driver-memory 2g \
    --executor-memory 5g \
    --num-executors 2 \
    --max-executors 2 \
    --conf spark.executor.memoryOverhead=1g \
    ./build/spark-benchmark.jar \
    "/tmp/queryStatus-rocksdb"

Sample Output

    |runId                |maxBatchId|TotalProcessedRecordsInMillion|TotalExecutionTimeInSec|AvgExecutionTimeInSec|MaxStateRowsInMillion|maxStateSizeInGB|
    | 3fc89a39-1cc3-46b...|        33|                         30.62|                1638.42|                48.19|                 8.57|            0.68| 

Acknowledgement

Jungtaek Lim for the original work for the structured streaming benchmarking.