Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance issue when batch fetch bars #29

Open
zillionare opened this issue Jul 14, 2022 · 0 comments
Open

performance issue when batch fetch bars #29

zillionare opened this issue Jul 14, 2022 · 0 comments
Assignees

Comments

@zillionare
Copy link
Owner

omicron >= 2.0.0.a35

when fetch one day's minute bars of all stock/index, _batch_get_persisted_bars will cost almost 120 secs in debug mode (53 seconds almost if not in debug).

Time consumed at:

  1. read content from http request and unzip, almost 20 seconds
  2. convert csv to dataframe, 3.7 secs if using 'c' engine and 1.x secs if 'pyarrow' engine is picked
  3. group and convert dataframe to numpy structured array, 50 secs.
  4. influxdb will cost almost 20 secs.

we need decrease time cost by both 1 and 3 to seconds.

for 1, maybe we can try:

  1. use httpx instead of aiohttpclient
  2. let pandas to do the unzip

for 3, no idea at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants