Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate importer to ingestion takes <2s with Kafka #29

Open
mgoelswirlds opened this issue Aug 26, 2022 · 2 comments
Open

Validate importer to ingestion takes <2s with Kafka #29

mgoelswirlds opened this issue Aug 26, 2022 · 2 comments
Assignees
@mgoelswirlds
Labels
BigDataPOC Mirror Node POC

Comments

@mgoelswirlds
Copy link
Collaborator

Import a 20k+ record file data through the new importer into Pinot.
Verify the roundtrip time <2s
@mgoelswirlds mgoelswirlds added the BigDataPOC Mirror Node POC label Aug 26, 2022
@mgoelswirlds mgoelswirlds self-assigned this Aug 29, 2022
@mgoelswirlds
Copy link
Collaborator Author

mgoelswirlds commented Aug 31, 2022 •
edited
Loading

Tested this with a records file from Performance environment containing 24667 records.
For the entire file ingestion took about 2.6 seconds.
Testing just one transaction took 665 ms

@mgoelswirlds mgoelswirlds changed the title Validate importer to ingestion takes <2s Validate importer to ingestion takes <2s with Kafka Sep 1, 2022
@mgoelswirlds
Copy link
Collaborator Author

So After further Investigation the new importer is taking most of the time and actual Kafka->Pinot->Rest Query takes about 290ms.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mgoelswirlds mgoelswirlds

Labels
BigDataPOC Mirror Node POC
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mgoelswirlds