Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEAT] IO Morsel Size #3004

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[FEAT] IO Morsel Size #3004

wants to merge 3 commits into from

Conversation

colin-ho
Copy link
Contributor

@colin-ho colin-ho commented Oct 6, 2024

Change morsel size for IO (url download / upload) to be small. Users can still change it up by setting morsel_size in the execution config.

Todos:

  • Ideally I'd like users to be able to configure morsel size at a per-op level, but that will likely be a more involved API change.

@github-actions github-actions bot added the enhancement New feature or request label Oct 6, 2024
Copy link

codspeed-hq bot commented Oct 6, 2024

CodSpeed Performance Report

Merging #3004 will not alter performance

Comparing colin/morsel-size (753ac24) with main (73ff3f3)

Summary

✅ 17 untouched benchmarks

Copy link

codecov bot commented Oct 6, 2024

Codecov Report

Attention: Patch coverage is 90.24390% with 4 lines in your changes missing coverage. Please review.

Project coverage is 78.52%. Comparing base (73ff3f3) to head (753ac24).

Files with missing lines Patch % Lines
src/common/daft-config/src/python.rs 40.00% 3 Missing ⚠️
...-execution/src/intermediate_ops/intermediate_op.rs 87.50% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3004      +/-   ##
==========================================
+ Coverage   78.14%   78.52%   +0.37%     
==========================================
  Files         610      610              
  Lines       72146    71769     -377     
==========================================
- Hits        56381    56354      -27     
+ Misses      15765    15415     -350     
Files with missing lines Coverage Δ
daft/context.py 79.48% <ø> (ø)
src/common/daft-config/src/lib.rs 87.50% <100.00%> (ø)
src/daft-dsl/src/expr/mod.rs 75.66% <100.00%> (+0.10%) ⬆️
src/daft-dsl/src/functions/scalar.rs 94.00% <100.00%> (+0.81%) ⬆️
src/daft-dsl/src/lib.rs 100.00% <ø> (ø)
src/daft-functions/src/uri/download.rs 83.84% <100.00%> (+0.38%) ⬆️
src/daft-functions/src/uri/upload.rs 65.26% <100.00%> (+0.63%) ⬆️
...ft-local-execution/src/intermediate_ops/project.rs 100.00% <100.00%> (ø)
src/daft-local-execution/src/lib.rs 89.74% <100.00%> (ø)
src/daft-local-execution/src/run.rs 89.84% <100.00%> (ø)
... and 2 more

... and 16 files with indirect coverage changes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant