Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CHORE] Enable lancedb reads for native executor #2925

Open
wants to merge 16 commits into
base: main
Choose a base branch
from

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Sep 25, 2024

Enables Lance reads for swordfish.

The existing lance read mechanism reads from python iterators. This PR abstracts out that logic to a separate function.

Both the python and streaming executors will call this function, the python executor will do a .collect on the iterator, while the native runner will wrap the iterator in a stream.

@github-actions github-actions bot added the chore label Sep 25, 2024
Copy link

codspeed-hq bot commented Sep 25, 2024

CodSpeed Performance Report

Merging #2925 will not alter performance

Comparing colin/swordfish-lance-reads (2e23e47) with main (73ff3f3)

Summary

✅ 17 untouched benchmarks

Copy link

codecov bot commented Sep 25, 2024

Codecov Report

Attention: Patch coverage is 95.06173% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.48%. Comparing base (73ff3f3) to head (2e23e47).

Files with missing lines Patch % Lines
src/daft-micropartition/src/python.rs 94.73% 4 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2925      +/-   ##
==========================================
+ Coverage   78.14%   78.48%   +0.33%     
==========================================
  Files         610      610              
  Lines       72146    71752     -394     
==========================================
- Hits        56381    56315      -66     
+ Misses      15765    15437     -328     
Files with missing lines Coverage Δ
src/daft-local-execution/src/sources/scan_task.rs 83.41% <100.00%> (+1.46%) ⬆️
src/daft-micropartition/src/micropartition.rs 90.47% <100.00%> (-0.33%) ⬇️
src/daft-micropartition/src/python.rs 78.29% <94.73%> (+1.55%) ⬆️

... and 15 files with indirect coverage changes

@colin-ho colin-ho marked this pull request as ready for review October 8, 2024 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant