Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: generator-based chunky Table data iteration #5595

Merged
merged 23 commits into from
Jul 10, 2024

Conversation

jmao-denver
Copy link
Contributor

@jmao-denver jmao-denver commented Jun 10, 2024

Fixes #5186

@jmao-denver jmao-denver self-assigned this Jun 10, 2024
@jmao-denver jmao-denver added this to the 3. May 2024 milestone Jun 10, 2024
@jmao-denver jmao-denver changed the title Add a table data reader generator New table data reader generator Jun 10, 2024
@jmao-denver jmao-denver marked this pull request as ready for review June 11, 2024 21:42
@jmao-denver
Copy link
Contributor Author

jmao-denver commented Jun 11, 2024

I have looked at the new ColumnVectors and discussed with @rcaudy about utilizing it to achieve the same goal. I am still not very clear about the advantages/disadvantages between the two approach. @rcaudy could you weigh on this with your thoughts?

Another approach @rcaudy mentioned is something similar to ConstructSnaphot, again I am not sure if it has definitive advantages in performance/consistency.

@rcaudy
Copy link
Member

rcaudy commented Jun 12, 2024

I have looked at the new ColumnVectors and discussed with @rcaudy about utilizing it to achieve the same goal. I am still not very clear about the advantages/disadvantages between the two approach. @rcaudy could you weigh on this with your thoughts?

Another approach @rcaudy mentioned is something similar to ConstructSnaphot, again I am not sure if it has definitive advantages in performance/consistency.

The issue with using ColumnVectors is that the iteration pattern is cell-oriented, even if the data retrieval is chunk-oriented. There's no pattern implemented that would let you hand data over to Python in an efficient pattern. I think the Python generator pattern wrapping a getChunk or fillChunk pattern in Java is probably your best bet.

py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_reader.py Outdated Show resolved Hide resolved
@jmao-denver jmao-denver changed the title New table data reader generator New table data generator Jun 21, 2024
@jmao-denver jmao-denver changed the title New table data generator New table data generator-based iteration Jun 21, 2024
@jmao-denver jmao-denver changed the title New table data generator-based iteration Generator-based Table data iteration Jun 23, 2024
@jmao-denver jmao-denver changed the title Generator-based Table data iteration Generator-based chunky Table data iteration Jun 23, 2024
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
@jmao-denver jmao-denver changed the title Generator-based chunky Table data iteration feat: generator-based chunky Table data iteration Jun 27, 2024
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/table_listener.py Outdated Show resolved Hide resolved
py/server/deephaven/table_listener.py Outdated Show resolved Hide resolved
py/server/deephaven/table_listener.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/table.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
py/server/deephaven/_table_reader.py Outdated Show resolved Hide resolved
@jmao-denver jmao-denver dismissed devinrsmith’s stale review July 10, 2024 16:36

Already implemented what's requested.

@jmao-denver jmao-denver merged commit b1feeeb into deephaven:main Jul 10, 2024
16 checks passed
@jmao-denver jmao-denver deleted the 5186-data-iteration-python branch July 10, 2024 16:36
@github-actions github-actions bot locked and limited conversation to collaborators Jul 10, 2024
@deephaven-internal
Copy link
Contributor

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#260

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Native data iteration from python
5 participants