Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stateful Notebooks #9298

Open
wants to merge 32 commits into
base: dev
Choose a base branch
from
Open

Stateful Notebooks #9298

wants to merge 32 commits into from
+736 −242

Commits on Sep 17, 2024

  1. initial implementation of checkpoint state

    @shubham3121
    shubham3121 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    1c33ce5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    6ab466e View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    3ea39c7 View commit details
    Browse the repository at this point in the history

  2. clean up code

    @shubham3121
    shubham3121 committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    78f3cdb View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2024

  1. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    c8ee04f View commit details
    Browse the repository at this point in the history

  2. move checkpoint to syft util test_helpers 

    fix notebooks
    @shubham3121
    shubham3121 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    2db9c38 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 20, 2024
    Configuration menu
    Copy the full SHA
    ab13b9b View commit details
    Browse the repository at this point in the history

Commits on Sep 23, 2024

  1. add an API for database reset 

    - move env getters to new file to prevent circular imports
- move create admin method to user module
    @shubham3121
    shubham3121 committed Sep 23, 2024
    Configuration menu
    Copy the full SHA
    972649a View commit details
    Browse the repository at this point in the history

Commits on Sep 24, 2024

  1. disable reset database API 

    - add a stash method to filter by unique attrs
- overwrite objs that are init at startup with the incoming migrated obj
    @shubham3121
    shubham3121 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    1b5b083 View commit details
    Browse the repository at this point in the history

  2. update scenario notebooks with checkpoint

    @shubham3121
    shubham3121 committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    ec25150 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. update load checkpoint to prune initial pools and images 

    - restore and spin woker image and pools from checkpoint
- fix generation of UID from seed of worker name for inmem workers

Co-authored-by: rasswanth-s <[email protected]>
    @shubham3121 @rasswanth-s
    shubham3121 and rasswanth-s committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    66552da View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    c4501e2 View commit details
    Browse the repository at this point in the history

  3. fix argument name in load_checkpoint

    @shubham3121
    shubham3121 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    859a602 View commit details
    Browse the repository at this point in the history

  4. fix checkpoint path if not running from notebook

    @shubham3121
    shubham3121 committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    c7a24e0 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. extract notebook name from pytest environment if running in pytest 

    - fix imports in 0.9.1_helpers
    @shubham3121
    shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    ed74096 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    ce02a8d View commit details
    Browse the repository at this point in the history

  3. add ability to reset and load migrated data with custom worker pools

    @rasswanth-s
    rasswanth-s committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    c7e12ec View commit details
    Browse the repository at this point in the history

  4. fix worker pool list

    @rasswanth-s
    rasswanth-s committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    8aa464a View commit details
    Browse the repository at this point in the history

  5. Change env imports paths in grid/server.py 

    modified purge_worker_pool to purge_workers

Co-authored-by: Shubham Gupta <[email protected]>
    @rasswanth-s @shubham3121
    rasswanth-s and shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    a43ef8a View commit details
    Browse the repository at this point in the history

  6. Fix load_migration_data to separate out reset db 

    Co-authored-by: Shubham Gupta <[email protected]>
    @rasswanth-s @shubham3121
    rasswanth-s and shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    580805f View commit details
    Browse the repository at this point in the history

  7. Merge pull request #9320 from OpenMined/rasswanth/exp-reset-checkpoint 

    [WIP] Reset Stateful Migrations
    @rasswanth-s
    rasswanth-s committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    3d4350b View commit details
    Browse the repository at this point in the history

  8. pass root email and pwd to load checkpoint in scenario notebooks

    @shubham3121
    shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    0bed0a1 View commit details
    Browse the repository at this point in the history

  9. fix security bugs

    @shubham3121
    shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    b81affb View commit details
    Browse the repository at this point in the history

  10. fix uid mismatch when external uid is passed via values.yml in k8s

    @shubham3121
    shubham3121 committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    44751d5 View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. update tags to latest in devspace

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    79c2271 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    7c461ff View commit details
    Browse the repository at this point in the history

  3. fix worker image.get_all API not returning all images 

    - add check for pool exists in Kubernetes before scaling it down
- fix images.get_all in load checkpoint to rebuilt images
    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    43a70ad View commit details
    Browse the repository at this point in the history

  4. add few more prints

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    17327d8 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'dev' into shubham/notebooks-with-state

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    e429570 View commit details
    Browse the repository at this point in the history

  6. fix lint

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    69de9a7 View commit details
    Browse the repository at this point in the history

  7. fix pytest

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    97c5cd1 View commit details
    Browse the repository at this point in the history

  8. fix api/notebooks 11-container

    @shubham3121
    shubham3121 committed Sep 27, 2024
    Configuration menu
    Copy the full SHA
    5771b49 View commit details
    Browse the repository at this point in the history