Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Version 4.0.0 lacks for support of native DynamoDB Streams #205

Open
olbat opened this issue Sep 6, 2023 · 2 comments
Open

Version 4.0.0 lacks for support of native DynamoDB Streams #205

olbat opened this issue Sep 6, 2023 · 2 comments

Comments

@olbat
Copy link

olbat commented Sep 6, 2023

It seems that the version 4.0.0 of the library removes support of native DynamoDB Streams.
My understanding is that this is caused by the migration to KCL v2 that doesn't support it either.

The release notes are suggesting:

The Kinesis Binder does not support DynamoDB Streams Adapter any more. There is no its implementation for SDK v2 and general AWS recommendation is to use Kinesis enabled on the table to captures data changes.

Will it be possible to elaborate a bit on where these recommendations are coming from and what are the alternatives for applications that are strongly dependent on native DynamoDB Streams core features? Thanks in advance 👍


For a bit of extra context, here are a few key differences between native DynamoDB Streams and DynamoDB integration with Kinesis to implement change streams that are important for some use-cases and might have be overlooked when deprecating the support of native DynamoDB Streams:

Change data capture for DynamoDB Streams:

DynamoDB Streams helps ensure the following:

  • Each stream record appears exactly once in the stream.
  • For each item that is modified in a DynamoDB table, the stream records appear in the same sequence as the actual modifications to the item.

Kinesis Data Streams to capture changes to DynamoDB:

The Kinesis data stream records might appear in a different sequence than the item changes occurred. The same item notifications might also appear more than once in the stream. You can check the ApproximateCreationDateTime attribute to identify the approximate order that the item modifications occurred, and to identify duplicate records. ApproximateCreationDateTime indicates the time of the modification in milliseconds.

@artembilan
Copy link
Contributor

We will be happy to bring that adapter support back when its version for AWS SDK v2 is released.
And that is exactly a matter of the issue on AWS Lab side: awslabs/dynamodb-streams-kinesis-adapter#22.

Until then we just cannot have both AWS SDK v1 & v2 on classpath.
Plus as you know the KCL v2 cannot interact with that DynamoDB Streams v1.

@olbat
Copy link
Author

olbat commented Sep 7, 2023

I understand, let's hope a version of the adapter will be released in AWS SDK v2 soon 🤞
Thanks for your support 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants