Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-45007: Memory leak in Prompt Processing service #3492

Merged
merged 3 commits into from
Jul 9, 2024
Merged

DM-45007: Memory leak in Prompt Processing service #3492

merged 3 commits into from
Jul 9, 2024

Commits on Jul 9, 2024

  1. Move all int->string conversion to Helm template. 

    Doing any string conversions in the template, instead of in the values
files, makes for a more user-friendly interface and avoids human error.
The use of toString allows compatibility with values that really are
strings in the first place.
    @kfindeisen
    kfindeisen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    e18ee2a View commit details
    Browse the repository at this point in the history

  2. Add config flag for forcing worker reboots. 

    This flag is useful for working around memory leaks and other aging
problems. Since this *is* a workaround, reboots are turned off
by default.
    @kfindeisen
    kfindeisen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    7bb5fe7 View commit details
    Browse the repository at this point in the history

  3. Enable restarts on Prompt Processing workers with fixed pods. 

    The production systems that disable autoscaling are the only ones where
a pod is likely to last long enough for memory leaks to be an issue.
Even ComCamSim-dev doesn't hit the limit.
    @kfindeisen
    kfindeisen committed Jul 9, 2024
    Configuration menu
    Copy the full SHA
    366fdb0 View commit details
    Browse the repository at this point in the history