Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] opentelemetry KVLIST_VALUE is not readable in open search #3123

Open
topikachu opened this issue Aug 9, 2023 · 0 comments
Open

[BUG] opentelemetry KVLIST_VALUE is not readable in open search #3123

topikachu opened this issue Aug 9, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@topikachu
Copy link
Contributor

Describe the bug
The opentelemetry KVLIST_VALUE is recursively converted to one non-readable string
To Reproduce
Steps to reproduce the behavior:
Use opentelemetry k8sobjectsreceiver source to collect kubernetes events, and export to data prepper.
The body field in opensearch is something like

{"type":"ADDED","object":"{\"reason\":\"Created\",\"metadata\":\"{\\\"uid\\\":\\\"69a08c03-9043-4194-9e2a-a7994686232f\\\",\\\"managedFields\\\":\\\"[\\\\\\\"{\\\\\\\\\\\\\\\"apiVersion\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"v1\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"manager\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"kubelet\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"fieldsV1\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"{\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"f:firstTimestamp\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"{}\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"f:message\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"{}\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"f:count\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"{}\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"f:type\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\

Expected behavior
It's better to parse KVLIST_VALUE correctly and insert into OpenSearch as separated field key.
If no, parsing to a simple json string so the next pipeline can parse back
Screenshots
image

Environment (please complete the following information):
data-prepper image opensearchproject/data-prepper:2.3.2
Additional context
data-prepper is installed in an AWS eks and use AWS managed opensearch service
@topikachu topikachu added bug Something isn't working untriaged labels Aug 9, 2023
topikachu added a commit to topikachu/data-prepper that referenced this issue Aug 9, 2023
@topikachu
Fix otel log format issue opensearch-project#3123
dd8dc3d 
Covert KVList into a regular json string format.

Signed-off-by: Gong Yi <[email protected]>
@topikachu topikachu mentioned this issue Aug 9, 2023
Open
4 tasks
topikachu added a commit to topikachu/data-prepper that referenced this issue Oct 18, 2023
@topikachu
Fix otel log format issue opensearch-project#3123 for otel_logs_sourc…
a3da9a1 
…e plugin

1. Add a new arguments "recursiveJsonDisabled" whose default value is false and compatible with previouse behavior.

2. Once recursiveJsonDisabled sets to true,covert KVList into a regular json string format.

Signed-off-by: Gong Yi <[email protected]>
topikachu added a commit to topikachu/data-prepper that referenced this issue Oct 18, 2023
@topikachu
Fix otel log format issue opensearch-project#3123 for otel_logs_sourc…
bc5cd7e 
…e plugin

1. Add a new arguments "kvListAsRecursiveJson" whose default value is true and compatible with previous behavior.

2. Once kvListAsRecursiveJson sets to false,covert KVList into a regular json string format.

Signed-off-by: Gong Yi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Data Prepper Tracking Board
Status: Unplanned
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dlvenable @topikachu