Is Prefect suitable for managing fully dynamic & async workflow

[namoshizun]

Hii Prefect team,

I am currently looking for a workflow management tool for an internal app that my company is building, and Prefect looks like the most assuring one compared to other alternatives I researched :D! To give a bit of background:

• We want to build a SaaS-like app, where the user can design a workflow of computation tasks, trigger the workflow to execute, and watch how it goes. Each computation task corresponds to a configurable algorithm.
• The user can pause the workflow to re-configure a future task, then resume the workflow execution. He can also append new tasks.
• The user can leave a workflow half-completed, and come back to it anytime he feels like to.
• Our algorithms are running behind a Celery app.

From the description, I gather that such workflow requires such features:

1. Non-static, evolving DAG
2. DAG serialization & de-serialization

• so that I can snapshot a flow, save it into database, and resume from where it was at
• more importantly, a flow can be created entirely dynamically from some DAG representation created by frontend user.

3. Hooks to react to external events:

• which means during the lifetime of a flow, it cannot be a closed system immune to external events.
• ideally it can somehow be influenced by those off-schedule signals
  • user's pause/resume action
  • algorithm celery's message saying a computation request has been accepted and started.

While going through Prefect docs, it occurs to me that Prefect can tackle 2 and (kinda) 3. But I am not entirely sure... As for 3, I suspect I am actually looking for an event-driven workflow. It'd be extremely appreciated if you can help me understand whether Prefect is the best candidate for our use case. 😃

[namoshizun]

Hello...? anyone could help?

[i3v]

@namoshizun , so, after all, have you managed to adapt Prefect for this your idea?

[namoshizun]

Nope :). We gave up trying to implement what we wanted using Prefect and went to develop our own workflow tool instead.

[i3v]

@namoshizun OK, thanks for sharing!

I feel about the same... Looks like Prefect is even less suitable for this than the Apache Airflow.

Surprisingly, there seem to be no "workflow manager" that supports this seemingly simple and basic requirement of "realtime" insertion/removal of DAG vertices at any DAG execution stage...

[zanieb]

https://www.prefect.io/blog/announcing-prefect-orion/ basically meets the dynamicism requirements. The rest is quite complicated to address in the broad/general use-case but isn't outside the scope of our long-term goals.

[namoshizun]

@i3v
ikr, good to know I am not the only one having that feeling! I spent lots of time trying to find an existing workflow tool that handles the "realtime" DAG but couldn't find one... @madkinsz is there a time table for the fully dynamic DAG? I believe it is a very needed infrastructure and I am more than willing to pay $$$ for a good tool that handles well.

[zanieb]

We've released the technical preview. Refer to the FAQ for our current timetable / roadmap https://orion-docs.prefect.io/faq/