Scheduler was dead when database service failover. #38279
Replies: 6 comments 11 replies
-
|
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Beta Was this translation helpful? Give feedback.
-
|
@robintian001 could you share some logs for the same. Also if my understanding is right when your mysql service is failing over it causes the scheduler to exit, if so that seems reasonable since it expects an active connection to be available for it to run? cc: @potiuk |
Beta Was this translation helpful? Give feedback.
-
|
The Scheduler's logs 2024-03-16 17:11:08,194 INFO - Adopting or resetting orphaned tasks for active dag runs The above exception was the direct cause of the following exception: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): this message from microsoft support team It can be seen from the background connection log that your standby server restarts at 17:00 UTC on 3/16 for regular maintenance, and restarts at 17:07 UTC on 3/16 for about 7 minutes. In this case, your standby server will be failover to the new primary server for you to connect. At the same time, your original primary server restarts at 17:11 UTC on 3/16 and completes the restart at 17:19 UTC on 3/16, which takes about 8 minutes. From the connection log, your initial standby server restarts before the initial primary server, so your standby server that has restarted can connect before your initial primary server restarts.
From the figure below about the conversion of HA server, you can see that at 17:12 UTC on 3/16, your HA server has been switched, which means that at this time your original standby server has become your new primary server available for connection. As you can see from the following figure, your total failover time is about 2 minutes. If you feel that you are not connected for about 20 minutes, it may be related to the retry mechanism of your client.
|
Beta Was this translation helpful? Give feedback.
-
|
Yes. It's entirely expected for airflow to go down in such case. We hae no resilience for database missing. If soeon @robintian001 would like to add such resilance that would be an interesting AIP (Airflow Improvement Proposal) to write. |
Beta Was this translation helpful? Give feedback.
-
|
When we troubleshoot the problem, we observed the entire process of Airflow. First, the Web Server restored normal after the database was reachable, but the Scheduler process crashed. |
Beta Was this translation helpful? Give feedback.
-
|
Yes. that's expected. And the right way to recover is to restart airflow scheduler when it crashes. Ths is a perfectly valid way of dealing with the problem. But if your team wants to increase the robustness, you are most welcome to design, propose and implement proposal on how to do such failover more robust. Airflow is developed by > 2800 contributors, so if your team is interested in it, it's a great way to contribute back for the free software you have, to contribute such feature. It's a lot of work, but if you are committed to do it, and would like to dedicate some engineering time, that would be fantastic. |
Beta Was this translation helpful? Give feedback.
-
|
After my thinking and reading the airflow source code, I thought we could leave max in the
|
Beta Was this translation helpful? Give feedback.
-
|
It will not help in all cases. We only do DB retries in selected cases when we know it will not be problematic because it will potentially have problems with state stored in memory. There is a big number of SQL queries run by scheduler that do not have retries - precisely because of the potentially impacting state that is stored in memory. Generally implementing (and testing) resilience against potentially longer time of DB unavailability will likely require analysis and handling of all potential queries run by scheduler and implementing specific solution to make sure that it can survive such cases. Plus it will require extensive test harness to test such cases if it were to be maintainable in the future as well. Implementing and designing such an approach should be documented in AIP and very thoroughly examined and tested. You have to remember that Airflow is an open-source project that is developed by volunteers, and any case where you implement something like that need to be thorouhgly and automatically tested using test suite in CI - because we have no team of testers that would take every future release of Airflow through a set of manual testing those kind of automatic failover scenarios. Another option (other than contributing automatic test harness to test all the cases) is for company like yours to commit such manual testing time and effort - but it would be extremely difficult to coordinate and it would hold us back from making releases as frequently as we woudl like to, so it's rather impossible. As opposed to thet - exiting Airflow when DB connection is lost and letting deployment to monitor it and restart is far simpler solution that does not require extensive testing, because the state stored in memory is discarded automatically when scheduler is shutdown. So you have two choices:
That's basically the choice you have. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for your suggestion. We will test our scheme in our own environment |
Beta Was this translation helpful? Give feedback.
-
|
Converted to discussion. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @robintian001, do you have a solution for this case now? We are encountering the same issue, where our internal MySQL might fail over and cause the scheduler job to exit. Would it be possible to introduce a DAO layer or something like a state backend to provide a unified interface for DB access? This layer could handle various scenarios, such as switchover, etc. cc @potiuk |
Beta Was this translation helpful? Give feedback.
-
|
Hi @TakawaAkirayo . |
Beta Was this translation helpful? Give feedback.
-
Surely. Usually such systems are order of magnitude more complex than when they don't do it - to handle various cases. So if you someone (your company) can sponsor development of it, and get the team dedicated for it for a few months of work, then yes, surely it could be done. Alternative is to restart failed process when the monitoring of that process detects that it fails. Which is a standard approach for managing running applications for as long as I am doing an IT work - and pretty much all standard ways of running long-time running applicaiton do it - starting from systemd and ending with Kubernetes that even has liveness probes that not only check if process is running but also whether it is "alive" - and you can working it out of the box if you are choosing to run one of those 20 and 10 years old solutions. |
Beta Was this translation helpful? Give feedback.
-
|
We are currently deployed in VM, we have also studied deployment in K8S, but found that one component has a memory leak bug, which often triggers our monitoring alarms, so we have not tried this again for the time being, if the probe can be used in the container to complete the survival check and complete the restart, for now, I think using a process (or probe) to restart the scheduler process can meet the current availability requirements. |
Beta Was this translation helpful? Give feedback.
-
|
@robintian001 Thanks for the response. We did something similar. We now use supervisord to start the Airflow scheduler along with other critical components. It monitors the processes and restarts them if they fail. |
Beta Was this translation helpful? Give feedback.
-
|
@potiuk Thank you for the response! We are planning to migrate a significant amount of heavy workloads from other scheduler systems to Airflow. There are critical workflows with strict SLA/SLE requirements, so stability is paramount. We've decided to invest resources into this, such as introducing a DAO layer(Or internal API in the Airflow's roadmap), and other improvements are underway. We don't want to diverge too much from the community, so it's good to know that some of our ideas align with the community's direction. |
Beta Was this translation helpful? Give feedback.
-
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.7.1
What happened?
Scheduler was dead when database service failover.
What you think should happen instead?
No response
How to reproduce
The mysql service failover is triggered manually.
Operating System
Linux (rhel 9.2) RedHat
Versions of Apache Airflow Providers
Airflow 2.7.1
Deployment
Other
Deployment details
deploment on azure vm, the database is MySQL Flexible Server of Azure.
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Beta Was this translation helpful? Give feedback.
All reactions