Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FLINK-36558] Fix column metadata parsing compatibility with MySQL 8.0.17 / 8.0.18 #3647

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

yuxiqian
Copy link
Contributor

@yuxiqian yuxiqian commented Oct 17, 2024

This closes FLINK-36558.

It is known that MySQL 8.0.17 and 8.0.18 writes Array-typed key index TableMap data into Binlog differently.

  • MySQL 8.0.17 used a different internal enum value for TYPED_ARRAY column type
  • MySQL 8.0.17 and 8.0.18 writes extraneous bytes after expected column metadata

The "extra byte" issue has been fixed since MySQL 8.0.19. Since MySQL takes Binlog format as an internal implementation detail[1] and doesn't guarantee its backwards compatibility, MySQL CDC has to deal with the compatibility problem itself.

The actual code change is limited to TableMapEventDataDeserializer.java where the encoded "column metadata bytes size" is used to determine how many bytes should be sent to readMetadata function for parsing. Any unused bytes will be ignored and will not affect parsing the following fields.

[1]   https://bugs.mysql.com/bug.php?id=105545

Copy link
Contributor

@lvyanquan lvyanquan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @yuxiqian for this detailed description, and this change looks good to me.

Signed-off-by: yuxiqian <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants