-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add executed_with_root_target
condition to handle partial runs / failures
#25253
base: 10-10-add_error_for_user_code_sensors_with_too_many_entities_targeted
Are you sure you want to change the base?
Conversation
2e98c95
to
dce5c4f
Compare
b9dbe0c
to
4e4b437
Compare
dce5c4f
to
0059e10
Compare
4e4b437
to
2850300
Compare
0059e10
to
87bf450
Compare
2850300
to
840bb0e
Compare
87bf450
to
2025cb6
Compare
840bb0e
to
230df04
Compare
2025cb6
to
8e94ee3
Compare
230df04
to
0ddac7d
Compare
8e94ee3
to
f84471d
Compare
0ddac7d
to
46b1a7c
Compare
f84471d
to
07c15fb
Compare
46b1a7c
to
dfc460c
Compare
07c15fb
to
b029615
Compare
dfc460c
to
528031d
Compare
check_selection = run_record.dagster_run.asset_check_selection or set() | ||
return target_key in (asset_selection | check_selection) | ||
|
||
def _compute_latest_check_run_executed_with_target( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fact that these two implementations are so different is brutal
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are there performance implications of this given that it is in the eager
condition and ends up invoking expensively_compute_partition_keys
? Are we saved by latest_time_window
here?
b029615
to
caca0a8
Compare
528031d
to
5aa93dd
Compare
caca0a8
to
da8082f
Compare
5aa93dd
to
7ad38c3
Compare
da8082f
to
f8c18ab
Compare
7ad38c3
to
e802a7c
Compare
f8c18ab
to
e3fc146
Compare
e802a7c
to
fbf6ea6
Compare
We're actually saved by the fact that this is and'd together with the |
Summary & Motivation
Resolves: #24389
To put it simply, this ensures the
any_deps_updated
condition does not "count" an update from a parent if the run that materialized that parent also planned to materialize the child.This guards against a variety of situations where a run can be launched with the intention of updating many assets, but fail to do so for whatever reason (intentional skip, failure event). In those cases, the assumption is that the child should NOT be attempted again (even though the parent did successfully materialize).
This is a fairly niche implementation detail that users should not have to think about when crafting their own policies.
How I Tested These Changes
I added a parameterized test that failed before this change (for the skip and fail cases) and passes now.
Changelog
Fixed an issue with
AutomationCondition.eager()
that could cause it to attempt to launch a second attempt of an asset in cases where it was skipped or failed during a run where one of its parents successfully materialized.