Enhance Throughput Efficiency by Modifying GSI Sort Key in DynamoMQ

[vvatanabe]

Describe the solution you'd like:
I propose changing the Global Secondary Index (GSI) sort key from sent_at to sequence_key in DynamoMQ. This modification aims to improve throughput efficiency. When FIFO is disabled, the sequence_key will hold a random string (e.g., UUID), which will enable partition splitting in DynamoDB and thus enhance throughput. When FIFO is enabled, the sequence_key will continue to hold an ISO 8601 formatted date-time string. Although this retains the throughput limitation inherent to FIFO, it is an acceptable constraint given the requirement for ordered processing.

Describe alternatives you've considered:
An alternative considered was maintaining the current design with queue_type as the partition key and sent_at as the sort key in the GSI. However, this approach limits the available throughput to 1000 WCUs and 3000 RCUs due to all data being stored in the same GSI partition. Using Scan for fetching messages was also considered but discarded due to its inefficiency.

Additional context:

• Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 1: Loading)
• Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 2: Querying)
• Scaling DynamoDB: How partitions, hot keys, and split for heat impact performance (Part 3: Summary and best practices)
• Feedbacks in Reddit

[max-pv]

Hi @vvatanabe, thanks for this great library. I'm considering using it in my project. Could you please tell me if this would be a breaking change?

[vvatanabe]

@br4in3x Thank you for considering DynamoMQ for your project! To improve performance, I'm planning some breaking changes to the table definition. I intend to release new version (v0.12.0) within this month.