Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

try to use NullPool to fix idle connection issue #121

Merged
merged 5 commits into from
Oct 17, 2024
Merged

Conversation

meksor
Copy link
Contributor

@meksor meksor commented Oct 16, 2024

Currently, we keep running out of available connections on the postgres cluster because of how the scaling and connection pooling works. NullPool will probably slow us down a bit for each request but fix the issue for now until I can find a way to make this uncompromisigly scalable: https://docs.sqlalchemy.org/en/20/core/pooling.html#sqlalchemy.pool.NullPool

Copy link
Member

@glatterf42 glatterf42 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for hijacking and stalling this PR.

Just for posterity: the Code Quality check was broken because ubuntu-latest (the GitHub runner image) switched to 24.04, which comes with issues such as an externally managed Python, which does not allow you to pip install things unless you're in a venv.
The pre-commit GitHub Action is in maintenance mode and unlikely to respond to that, so I fiddled around with running pre-commit the recommended way. This needs to be enabled for the IIASA organization.
Pre-commit install a separate venv for every tool and is opinionated enough to never change that, regardless of how many people complain. Thus, for type checkers like mypy, workarounds need to be found to make them run in our poetry-managed venv. For this, https://jaredkhan.com/blog/mypy-pre-commit#option-1-use-language-system-to-run-mypy-in-an-existing-environment and python-poetry/poetry#7091 were very useful, but try as I may, I couldn't get it to work.

In the meantime, enough push back has happened for Github to revert ubuntu-latest to 22.04, which works fine with the pre-commit action. Nevertheless, we should keep an eye open and try to find a more sustainable way of running mypy in our CI pipeline.

(I disabled the pre-commit bot for the organization again, so please ignore the failed check.)

@meksor meksor merged commit e3ee9c8 into main Oct 17, 2024
10 of 11 checks passed
@meksor meksor deleted the bugfix/db-connections branch October 17, 2024 06:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants