This repository has been archived by the owner on Oct 9, 2023. It is now read-only.
Instrument ArrayNode #550
Merged
Instrument ArrayNode #550
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
kumare3
reviewed
Jul 27, 2023
There was a problem hiding this comment.
To be honest this is a crazy big code change and it looks pretty good. Boy though, when you look at some of the repeated verbosity in golang haha. But, mostly looks great to me, couple small comments.
Since this is behind a feature gate, i think this is safe to go
Signed-off-by: Daniel Rammer <[email protected]>
Signed-off-by: Daniel Rammer <[email protected]>
kumare3
approved these changes
Jul 28, 2023
eapolinario
pushed a commit
to eapolinario/flytepropeller
that referenced
this pull request
Aug 9, 2023
* updated flyteidl to local to get ArrayNode Signed-off-by: Daniel Rammer <[email protected]> * added boilerplate to support ArrayNode Signed-off-by: Daniel Rammer <[email protected]> * pushing forward Signed-off-by: Daniel Rammer <[email protected]> * refactored node executor interfaces to fix dependency cycle Signed-off-by: Daniel Rammer <[email protected]> * refactoring almost complete Signed-off-by: Daniel Rammer <[email protected]> * refactor complete Signed-off-by: Daniel Rammer <[email protected]> * supporting environment variables Signed-off-by: Daniel Rammer <[email protected]> * minimum viable product Signed-off-by: Daniel Rammer <[email protected]> * update print statements for debugging Signed-off-by: Daniel Rammer <[email protected]> * massive refactor fixing NodeExecutionContext override for ArrayNode Signed-off-by: Daniel Rammer <[email protected]> * refactoring TODOs Signed-off-by: Daniel Rammer <[email protected]> * subnode retries working Signed-off-by: Daniel Rammer <[email protected]> * parallelism working Signed-off-by: Daniel Rammer <[email protected]> * cache and cache_serialize working - first new functionality in maptask Signed-off-by: Daniel Rammer <[email protected]> * adding implementation notes Signed-off-by: Daniel Rammer <[email protected]> * removed eventing from subtasks Signed-off-by: Daniel Rammer <[email protected]> * adding correct requirements Signed-off-by: Daniel Rammer <[email protected]> * working end-2-end with flytekit Signed-off-by: Daniel Rammer <[email protected]> * reporting output directory on success Signed-off-by: Daniel Rammer <[email protected]> * fixed output directory append Signed-off-by: Daniel Rammer <[email protected]> * mocking TaskTemplate interface to enable caching Signed-off-by: Daniel Rammer <[email protected]> * capture failure reasons Signed-off-by: Daniel Rammer <[email protected]> * wrapped up abort and finalize functionality Signed-off-by: Daniel Rammer <[email protected]> * mocking initialization events Signed-off-by: Daniel Rammer <[email protected]> * sending all events Signed-off-by: Daniel Rammer <[email protected]> * minor refactoring of debug prints and formatting Signed-off-by: Daniel Rammer <[email protected]> * intratask checkpointing working Signed-off-by: Daniel Rammer <[email protected]> * support for and Signed-off-by: Daniel Rammer <[email protected]> * setting node log ids correctly Signed-off-by: Daniel Rammer <[email protected]> * reporting cache status Signed-off-by: Daniel Rammer <[email protected]> * correctly setting subnode abort phase Signed-off-by: Daniel Rammer <[email protected]> * removing dead code Signed-off-by: Daniel Rammer <[email protected]> * cleaned up most random TODO items Signed-off-by: Daniel Rammer <[email protected]> * refactored into new files Signed-off-by: Daniel Rammer <[email protected]> * refactoring for ArrayNode unit tests Signed-off-by: Daniel Rammer <[email protected]> * refactored for unit testing to allow creation of NodeExecutor in array package Signed-off-by: Daniel Rammer <[email protected]> * first unit test for handling ArrayNodePhaseNone Signed-off-by: Daniel Rammer <[email protected]> * most of executing unit tests completed Signed-off-by: Daniel Rammer <[email protected]> * finished executing unit tests Signed-off-by: Daniel Rammer <[email protected]> * finished succeeding unit tests Signed-off-by: Daniel Rammer <[email protected]> * wrote failing phase unit tests Signed-off-by: Daniel Rammer <[email protected]> * moving towards complete unit_test success Signed-off-by: Daniel Rammer <[email protected]> * unit tests passing Signed-off-by: Daniel Rammer <[email protected]> * fixed lint issues Signed-off-by: Daniel Rammer <[email protected]> * updated flyteidl dep Signed-off-by: Daniel Rammer <[email protected]> * added unit tests for Abort Signed-off-by: Daniel Rammer <[email protected]> * adding unit test for Finalize Signed-off-by: Daniel Rammer <[email protected]> * added utils unit tests Signed-off-by: Daniel Rammer <[email protected]> * moved state structs to handler package Signed-off-by: Daniel Rammer <[email protected]> * added docs Signed-off-by: Daniel Rammer <[email protected]> * cleaned up abort event reporting Signed-off-by: Daniel Rammer <[email protected]> * fixed RecordNodeEvent unit tests Signed-off-by: Daniel Rammer <[email protected]> * removed taskEventRecorder from nodes package Signed-off-by: Daniel Rammer <[email protected]> * adding interface checking for arraynode Signed-off-by: Daniel Rammer <[email protected]> * added transform unit test Signed-off-by: Daniel Rammer <[email protected]> * fixed input bindings issue Signed-off-by: Daniel Rammer <[email protected]> * fixed unit tests Signed-off-by: Daniel Rammer <[email protected]> * fixed unit tests Signed-off-by: Daniel Rammer <[email protected]> * go generate Signed-off-by: Daniel Rammer <[email protected]> * addressing random TODO Signed-off-by: Daniel Rammer <[email protected]> * fixed unit tests Signed-off-by: Daniel Rammer <[email protected]> * addressing pr comments Signed-off-by: Daniel Rammer <[email protected]> --------- Signed-off-by: Daniel Rammer <[email protected]>
8 tasks
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Instrumenting support for ArrayNodes to provide a functionally complete map task experience.
Type
Are all requirements met?
Complete description
^^^
Tracking Issue
flyteorg/flyte#1131
Follow-up issue
NA