From 099f87a2904d631a2c22d6976efbf2c1e6bb879f Mon Sep 17 00:00:00 2001 From: Kyle McCormick Date: Wed, 25 Oct 2023 12:21:25 -0400 Subject: [PATCH] fix: run Celery tasks asynchronously in dev mode In `tutor dev`, Celery tasks were running eagerly (in-process) rather than being delegated to the worker containers for asynchronous processing. This difference in behavior between `tutor dev` and `tutor local` meant that certain code paths and race-condition bugs would not surface when testing in dev. The cause is that CELERY_ALWAYS_EAGER is set to True in the upstream devstack settings file. That's because Devstack does not support async LMS/CMS workers; it runs all those tasks in-process. The fix is to simply override CELERY_ALWAYS_EAGER to False. We do this in the common settings file for simplicity, but the change should not affect production mode, which has always had CELERY_ALWAYS_EAGER equal False. For unit tests, we override CELERY_ALAYS_EAGER back to True, because many of them are written to assume in-process tasks. --- changelog.d/20231025_122158_kyle_celery_async.md | 1 + tutor/templates/apps/openedx/settings/partials/common_all.py | 3 +++ tutor/templates/apps/openedx/settings/partials/common_test.py | 4 ++++ 3 files changed, 8 insertions(+) create mode 100644 changelog.d/20231025_122158_kyle_celery_async.md diff --git a/changelog.d/20231025_122158_kyle_celery_async.md b/changelog.d/20231025_122158_kyle_celery_async.md new file mode 100644 index 0000000000..b53b2d2b11 --- /dev/null +++ b/changelog.d/20231025_122158_kyle_celery_async.md @@ -0,0 +1 @@ +- [Bugfix] In development mode, Celery tasks will now be delegated to worker containers for asynchronous execution rather than eagerly exected by the lms/cms app processes. This matches the behavior of production mode and should enable more robust testing & debugging of Celery tasks. Unit tests, however, will still eagerly execute Celery tasks in-process (by @kdmccormick). diff --git a/tutor/templates/apps/openedx/settings/partials/common_all.py b/tutor/templates/apps/openedx/settings/partials/common_all.py index b1d5a9f82a..f0984bde85 100644 --- a/tutor/templates/apps/openedx/settings/partials/common_all.py +++ b/tutor/templates/apps/openedx/settings/partials/common_all.py @@ -235,5 +235,8 @@ "user": None, } +# Run Celery tasks asynchronously on the workers (rather than blocking the app process). +CELERY_ALWAYS_EAGER = False + {{ patch("openedx-common-settings") }} ######## End of settings common to LMS and CMS diff --git a/tutor/templates/apps/openedx/settings/partials/common_test.py b/tutor/templates/apps/openedx/settings/partials/common_test.py index 290ac2929c..a16d5c5268 100644 --- a/tutor/templates/apps/openedx/settings/partials/common_test.py +++ b/tutor/templates/apps/openedx/settings/partials/common_test.py @@ -1,3 +1,7 @@ # Fix MongoDb connection credentials DOC_STORE_CONFIG["user"] = None DOC_STORE_CONFIG["password"] = None + +# In edx-platform CI, unit tests are run in-process, so many of them break if run +# async on the workers. So, we default to testing them in-process as well. +CELERY_ALWAYS_EAGER = True