-
Notifications
You must be signed in to change notification settings - Fork 149
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory Retention After Executing Python Function in SharedInterpreter #555
Comments
Thank you for providing clear instructions to easily recreate the problem. The problem is that jep relies on Python reference counting to clear memory during The good news is that the memory is not lost forever, if you have other SharedInterpreters running or if you open new SharedInterpreters then the Python garbage collection(gc) will eventually run and free the memory. I was able to free the memory by opening a new SharedInterpreter and running gc manually like this: try (SharedInterpreter interpreter = new SharedInterpreter()) {
interpreter.exec("def foo():\n\tpass\nfoo()");
interpreter.exec("x=[10000]*100000000");
}
Thread.sleep(30000); // observe the memory
try (SharedInterpreter interpreter = new SharedInterpreter()) {
interpreter.exec("import gc");
interpreter.exec("gc.collect()");
} Another workaround for existing versions of jep is to run gc manually before close, this worked for me also: try (SharedInterpreter interpreter = new SharedInterpreter()) {
interpreter.exec("def foo():\n\tpass\nfoo()");
interpreter.exec("x=[10000]*100000000");
interpreter.exec("globals().clear()\nimport gc\ngc.collect()");
} I agree this is something that Jep should handle better so I have opened #556 to have jep run gc on close(). That will resolve the problem in future versions of Jep. If you have the ability to build Jep with the change in the PR and test it then I would appreciate confirmation that this fixes the problem in any more complex use cases you might have. |
Thanks a lot for the quick turnaround! I had to replace |
In this case that would be enough to break the reference cycle so reference counting can clear the memory without gc.
I have not seen this error in my tests. I tried it on both python3.10 and python3.9. I am just using the 5 lines from my last comment to test. If you have any idea how to recreate this error it would be helpful because I would like to make sure the fix in the PR is not causing the same error.
It should work regardless of whether there is one or multiple interpreters. I have only tested it with a single interpreter. the collect() might take slightly longer with more SharedInterpreters because there are more objects to traverse but I wouldn't expect it to be noticeable in most cases. |
This reason for this failure was code-specific. We were doing something that basically was calling a gettattr which would have been subsequently deleted due globals().clear() |
Hey @bsteffensmeier . I found another memory leak. If you have a class method that is cached ( in my case functools.lru_cache) it leaks memory. This probably extends to the fact that static class variables and methods are not gc'ed. I will create a minimal reproduction and post here, was curious if you have seen something like this before. |
Describe the bug
When using the Jep library, it appears that defining and executing a Python function within a SharedInterpreter instance causes memory to be retained, even after closing the interpreter.
To Reproduce
Observe a memory spike in the JVM.
Observe that the memory used by the interpreter is not released.
Expected behavior
The memory used by the interpreter should be released after the interpreter is closed.
Environment :
Additional context
Observed that this issue also happens on
jep==4.2.0
If the line defining and invoking the function
jepInterpreter.exec("def foo():\n\tpass\nfoo()");
is removed, the memory is released as expected after closing the interpreter.The text was updated successfully, but these errors were encountered: