Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds an auto_schema flag to Avro/Parquet codecs #3206

Merged

Conversation

dlvenable
Copy link
Member

@dlvenable dlvenable commented Aug 22, 2023

Description

The current design of the auto-schema generation for the Avro/Parquet codecs will make the default behavior automatically generate a schema. Users may not be aware that this is happening.

This PR adds a new field auto_schema to enable the feature.

Thus, the user can configure a codec as such:

s3:
  codec:
    parquet:
      auto_schema: true

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Collaborator

@asifsmohammed asifsmohammed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@asifsmohammed
Copy link
Collaborator

Do we need to update any documentation for this change?

@dlvenable
Copy link
Member Author

Yes, I will update the existing PR.

@dlvenable dlvenable merged commit 3673d02 into opensearch-project:main Aug 22, 2023
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants