Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEAT] Ray shuffle experiment #2883

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open

[FEAT] Ray shuffle experiment #2883

wants to merge 16 commits into from

Commits on Sep 29, 2024

  1. Added a Ray-based shuffle service that uses an actor-per-node, no ide… 

    …a if this works though
    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    2bdb0ce View commit details
    Browse the repository at this point in the history

  2. Naive forwarding to physical plan

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    0e91bb2 View commit details
    Browse the repository at this point in the history

  3. yoloswag

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    fb1fde4 View commit details
    Browse the repository at this point in the history

  4. Who knows if this works

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    f137172 View commit details
    Browse the repository at this point in the history

  5. fix

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    b90435c View commit details
    Browse the repository at this point in the history

  6. fix

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    0ae0d71 View commit details
    Browse the repository at this point in the history

  7. only request for little bit of CPU on actor to match placement group

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    626e952 View commit details
    Browse the repository at this point in the history

  8. rename to fully materializing

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    a9bfb4f View commit details
    Browse the repository at this point in the history

  9. Add another streaming push variant

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    cef8843 View commit details
    Browse the repository at this point in the history

  10. stub out the thing

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    005c804 View commit details
    Browse the repository at this point in the history

  11. Add simple pull-based shuffle

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    7e0f88d View commit details
    Browse the repository at this point in the history

  12. Who knows if this works

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    0bc057d View commit details
    Browse the repository at this point in the history

  13. naming

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    37faaa0 View commit details
    Browse the repository at this point in the history

  14. Fixes it works locally

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    1e8b87d View commit details
    Browse the repository at this point in the history

  15. Add node affinity for reducer and merger

    Jay Chia committed Sep 29, 2024
    Configuration menu
    Copy the full SHA
    37b1d8e View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Fix math for number of returns from mapper

    Jay Chia committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    ade5e2f View commit details
    Browse the repository at this point in the history