Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integrate CDC data from MySQL/Postgres/MongoDb data source #3313

Merged
merged 30 commits into from
Oct 12, 2023
Merged

Integrate CDC data from MySQL/Postgres/MongoDb data source #3313

merged 30 commits into from
Oct 12, 2023

Commits on Sep 7, 2023

  1. Add KafkaClusterExtensionWithConfig plugin 

    Signed-off-by: Haidong <[email protected]>
    @wanghd89
    wanghd89 authored and Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    7ac1e3c View commit details
    Browse the repository at this point in the history

  2. add unit tests to kafka cluster config extension 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    3b4e25e View commit details
    Browse the repository at this point in the history

  3. integrate with kafka connect 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    7286de8 View commit details
    Browse the repository at this point in the history

  4. Modify Config, waiting for final confirm of Config files' change 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5e373ed View commit details
    Browse the repository at this point in the history

  5. update config to make mysql/postgres/mongodb as source 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    587a3d5 View commit details
    Browse the repository at this point in the history

  6. add ExtractNewRecordState transforms to simplify the CDC data 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    eabc079 View commit details
    Browse the repository at this point in the history

  7. fix mysql postgres field 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    e4d77d7 View commit details
    Browse the repository at this point in the history

  8. update readme 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    cd8ce61 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. add missing unit test for PluginMetrics. 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    74102d2 View commit details
    Browse the repository at this point in the history

Commits on Sep 11, 2023

  1. fix unit test and add snapshot_fetch_size for mongoDB to increase sna… 

    …pshot performance.

Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 11, 2023
    Configuration menu
    Copy the full SHA
    2c0adf1 View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2023

  1. address PR comments to use Duration 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    8219819 View commit details
    Browse the repository at this point in the history

  2. fix checkstyle 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 12, 2023
    Configuration menu
    Copy the full SHA
    5f7f502 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. rename config and exclude zookeeper according to PR review 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    27d1693 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2023

  1. Merge remote-tracking branch 'upstream/main' into feat-cdc

    Haidong committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    f5b66c9 View commit details
    Browse the repository at this point in the history

  2. expose connector_rebalance_max_delay config to customers. The default… 

    … delay is 5 mins. Also increase the default connector_start_timeout to 6 mins.

Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 15, 2023
    Configuration menu
    Copy the full SHA
    71ed291 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2023

  1. address comments from PR 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 18, 2023
    Configuration menu
    Copy the full SHA
    dc9ada6 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2023

  1. Merge branch 'main' of github.com:opensearch-project/data-prepper int… 

    …o feat-cdc

# Conflicts:
#	build.gradle
    Haidong committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    fcd1708 View commit details
    Browse the repository at this point in the history

  2. fix merge conflicts 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    1b20224 View commit details
    Browse the repository at this point in the history

  3. minimize affects the jetty-boom version 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    74f0f1d View commit details
    Browse the repository at this point in the history

  4. revert the change 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Sep 28, 2023
    Configuration menu
    Copy the full SHA
    af526c9 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2023

  1. Merge remote-tracking branch 'upstream/main' into feat-cdc 

    # Conflicts:
#	data-prepper-plugins/kafka-plugins/src/integrationTest/java/org/opensearch/dataprepper/plugins/kafka/source/KafkaSourceJsonTypeIT.java
#	data-prepper-plugins/kafka-plugins/src/integrationTest/java/org/opensearch/dataprepper/plugins/kafka/source/KafkaSourceMultipleAuthTypeIT.java
#	data-prepper-plugins/kafka-plugins/src/integrationTest/java/org/opensearch/dataprepper/plugins/kafka/source/MskGlueRegistryMultiTypeIT.java
#	data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/configuration/EncryptionConfig.java
#	data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/configuration/KafkaSourceConfig.java
#	data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/source/KafkaSource.java
#	data-prepper-plugins/kafka-plugins/src/main/java/org/opensearch/dataprepper/plugins/kafka/util/KafkaSecurityConfigurer.java
#	data-prepper-plugins/kafka-plugins/src/test/java/org/opensearch/dataprepper/plugins/kafka/source/KafkaSourceTest.java
#	data-prepper-plugins/kafka-plugins/src/test/resources/sample-pipelines.yaml
    Haidong committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    b6d1ad0 View commit details
    Browse the repository at this point in the history

  2. merge mainline into feat-cdc-kafkaconnect branch. Resolved conflicts 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    926dd1a View commit details
    Browse the repository at this point in the history

  3. resolve jetty CVE check by using latest 9.x version. revert using pla… 

    …tform

Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    45d97ae View commit details
    Browse the repository at this point in the history

  4. downgrade scoped Jetty dependency to use 9.4.52 version for Kafka-Con… 

    …nect-Plugin, which minimize the impact to other plugins.

Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 7, 2023
    Configuration menu
    Copy the full SHA
    401acb4 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2023

  1. remove kotlin-compiler to fix service_map plugin 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    3dcb858 View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/main' into feat-cdc 

    # Conflicts:
#	settings.gradle
    Haidong committed Oct 9, 2023
    Configuration menu
    Copy the full SHA
    845949e View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2023

  1. fix checkstyle caused by KK 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    0323ded View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Merge remote-tracking branch 'upstream/main' into feat-cdc 

    # Conflicts:
#	build.gradle
#	data-prepper-plugins/kafka-plugins/build.gradle
#	settings.gradle
    Haidong committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    8f51c45 View commit details
    Browse the repository at this point in the history

  2. fix CVE 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    1a1dff5 View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2023

  1. update CVE to use latest jetty9.4.53 version 

    Signed-off-by: Haidong <[email protected]>
    Haidong committed Oct 12, 2023
    Configuration menu
    Copy the full SHA
    a9ebf15 View commit details
    Browse the repository at this point in the history