Deadlock when holding dpg.mutex() a long time in a frame callback

[axeldavy]

Version of Dear PyGui

Version: 1.11.1
Operating System: Arch Linux

My Issue/Question

When doing heavy processing inside a dpg.mutex in a callback called from set_frame_callback, the application hangs.

To Reproduce

Run:

import dearpygui.dearpygui as dpg
import time

dpg.create_context()
dpg.create_viewport()
dpg.setup_dearpygui()

with dpg.window(tag="Primary Window"):
    dpg.add_text("Hello", tag="hello")

def hanging_handler(unused):
    with dpg.mutex():
        time.sleep(0.01)

dpg.show_viewport()
dpg.set_primary_window("Primary Window", True)

t0 = time.time()
while(dpg.is_dearpygui_running()):
    t1 = time.time()
    print("Frame rendered in:", t1-t0)
    t0 = t1
    dpg.set_frame_callback(dpg.get_frame_count()+1, hanging_handler)
    dpg.render_dearpygui_frame()
dpg.destroy_context()

This doesn't occur without the dpg.mutex or from a mouse or keyboard callback. The following code doesn't hang:

import dearpygui.dearpygui as dpg
import time

dpg.create_context()
dpg.create_viewport()
dpg.setup_dearpygui()

def key_handler(sender, data):
    print("In key handler")
    with dpg.mutex():
        time.sleep(0.1)

def mouse_handler(sender, data):
    print("In mouse handler")
    with dpg.mutex():
        time.sleep(0.1)

with dpg.window(tag="Primary Window"):
    dpg.add_text("Hello", tag="hello")
    
    with dpg.handler_registry(show=True):
        k_down = dpg.add_key_down_handler(key=dpg.mvKey_A)
        dpg.set_item_callback(k_down, key_handler)
        m_down = dpg.add_mouse_down_handler(button=dpg.mvMouseButton_Left)
        dpg.set_item_callback(m_down, mouse_handler)

dpg.show_viewport()
dpg.set_primary_window("Primary Window", True)
t0 = time.time()
while(dpg.is_dearpygui_running()):
    t1 = time.time()
    print("Frame rendered in:", t1-t0)
    t0 = t1
    dpg.render_dearpygui_frame()
dpg.destroy_context()

[axeldavy]

The following variant doesn't hang:

import dearpygui.dearpygui as dpg
import time

dpg.create_context()
dpg.create_viewport()
dpg.setup_dearpygui()


with dpg.window(tag="Primary Window"):
    dpg.add_text("Hello", tag="hello")

def hanging_handler(unused):
    with dpg.mutex():
        time.sleep(0.01)

dpg.show_viewport()
dpg.set_primary_window("Primary Window", True)
for i in range(20):
    dpg.set_frame_callback(i+1, hanging_handler)
t0 = time.time()
while(dpg.is_dearpygui_running()):
    t1 = time.time()
    print("Frame rendered in:", t1-t0)
    t0 = t1
    dpg.render_dearpygui_frame()
dpg.destroy_context()

Thus it might be possible that the hang calls from set_frame_callback not being thread-safe.

[v-ein]

The first script hangs up on dpg.get_frame_count(). Here's why.

1. set_frame_callback schedules the callback to run on the handlers thread, as it should.
2. Eventually the callback gets started. It locks the mutex and goes into time.sleep, at which point Python releases the GIL and lets another thread take over. The mutex remains locked.
3. The main thread takes over and continues to run the rendering loop. When it goes into get_frame_count, that function attempts to lock the mutex. Note: among the API functions called in the rendering loop, only render_dearpygui_frame and get_frame_count actually lock the mutex; the other two calls don't do it.
4. Since the mutex is being held by the handlers thread, the main threads enters waiting state, still holding the GIL.
5. As time passes and sleep is over, the handlers thread wakes up and needs to lock the GIL, which is held by the main thread.
6. We've got a nice classic deadlock: the handlers thread is waiting for the GIL, and the main thread is waiting on the mutex.

I think I've fixed this kind of deadlocks in my copy of DPG and will eventually push the fix to the main repo, but this won't happen soon. Until that point, avoid making extra API calls from the rendering loop. The default rendering loop (consisting of is_dearpygui_running and render_dearpygui_frame) does not deadlock.