[Feature] internal k8s error messages are actionable for general users

[slai]

Motivation: Why do you think this is important?
When an error occurs with k8s starting or running a task, the resulting error message in Flyte Console is often cryptic and unactionable.

Some examples -

• Pod reported success despite being OOMKilled, when a process in the task pod was OOMKilled but not pyflyte so task still completes. Reporting success despite a bad condition like OOMKilled is confusing - did the task succeed or not? What were the memory request/limits set? Which process was OOMKilled?
• containers with unready status: [execution_id]|context deadline exceeded, generally when a container image cannot be pulled from the registry. This is a combination of an unclear k8s message (why is the container unready?) and a Go specific error that a general user wouldn't understand (context deadline exceeded, i.e. timed out waiting). What should the user do?
• [3/3] currentAttempt done. Last Error: SYSTEM::object [execution_id] terminated in the background, manually, generally when the task pod was on an instance that was spot pre-empted or otherwise removed. What does 'currentAttempt done' mean, what does 'terminated in the background, manually' mean - what manually terminated it? This error is actually benign in most cases because the retry should succeed, but the message gives no indication of that

Goal: What should the final outcome look like, ideally?
The error message should strike a balance between -

• being understandable for the general user (little k8s knowledge)
• contain enough info for an engineer to troubleshoot
• be actionable without being too prescriptive and eliminating other causes

For example, the first error message above could be -

Task ended with one or more processes killed due to lack of memory. Current memory request is X, limit is Y. Process killed was Z. Consider increasing the memory request/limit of this task.

Alternatively, maybe an error code that a user can look up elsewhere with more info would be a better way to keep the necessary k8s detail in one place, and the general user explanation in another.

Flyte component

• Overall
• Flyte Setup and Installation scripts
• Flyte Documentation
• Flyte communication (slack/email etc)
• FlytePropeller
• FlyteIDL (Flyte specification language)
• Flytekit (Python SDK)
• FlyteAdmin (Control Plane service)
• FlytePlugins
• DataCatalog
• FlyteStdlib (common libraries)
• FlyteConsole (UI)
• Other

Additional context
See https://flyte-org.slack.com/archives/CNMKCU6FR/p1611329101012800 for some further discussion. Also seems like #512 and #535 are similar issues.

Is this a blocker for you to adopt Flyte
Nope, already a user :)

[kumare3]

@slai thank you for this. I think the aim of Flyte should be to simplify the messaging so that a complicated technology is more approachable. I really appreciate this feedback, but I think this is a feature not a bug?

[slai]

@kumare3 good point, I've updated the title and description to a feature request.

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏

[hamersaw]

k8s event reporting is now available in the UI as of flyteorg/flytepropeller#600