Segfault if attempting to feed too quickly to a stream

[tazz4843]

Hey there! I've been trying out Coqui in production for a little while now, and I've been hitting a segfault that seems extremely similar to #1995, with the same source (get_history_helper). However, switching feeding and decoding to the same thread doesn't seem to work, and still reliably, 100% of the time on streams longer than about 7 seconds (rarer but still possible on shorter ones), segfaults in the exact same location as well. It seems to be a null pointer deref according to gdb.

Bit of details about my situation:

• I'm using the Rust bindings I wrote myself (https://github.com/tazz4843/coqui-stt).
• The host hardware is a Ryzen 9 3900X with 128GB RAM (idling at 10GB used), so I seriously doubt it's any sort of OOM error.
• I'm streaming in 20ms audio packets 50x per second, but even if I ensure they're in order, it still segfaults.
• I'm using this in async code that calls to the stream from multiple threads, but I've been sure that the stream always stays on the same thread no matter what (using a mpsc queue that passes in things to be called on the target thread).

I'm not sure what more situational info I could give, so any help is appreciated.

[reuben]

I'm not sure what more situational info I could give, so any help is appreciated.

Code to reproduce this would be great!

[tazz4843]

I just pulled latest git master and am no longer able to reproduce either issue which is very weird. I'm instead hitting a different one:

scripty_v2: native_client/ctcdecode/ctc_beam_search_decoder.cpp:111: void DecoderState::next(const double*, int, int): Assertion `prefix->timesteps != nullptr' failed.

There seems to be no consistency to when this gets thrown though. It seems entirely random.
GDB shows backtrace as

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff5f00537 in __GI_abort () at abort.c:79
#2  0x00007ffff5f0040f in __assert_fail_base (fmt=0x7ffff6069128 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7ffff7830e64 "prefix->timesteps != nullptr", file=0x7ffff7830e30 "native_client/ctcdecode/ctc_beam_search_decoder.cpp", line=111, 
    function=<optimized out>) at assert.c:92
#3  0x00007ffff5f0f662 in __GI___assert_fail (assertion=0x7ffff7830e64 "prefix->timesteps != nullptr", file=0x7ffff7830e30 "native_client/ctcdecode/ctc_beam_search_decoder.cpp", line=111, function=0x7ffff7830df8 "void DecoderState::next(const double*, int, int)")
    at assert.c:101
#4  0x00007ffff6e9cc5b in DecoderState::next (this=0x7fff98000c40, probs=0x7fff9800e3d0, time_dim=10, class_dim=29) at native_client/ctcdecode/ctc_beam_search_decoder.cpp:111
#5  0x00007ffff6e85dfa in StreamingState::processBatch (this=0x7fff98000bc0, buf=std::vector of length 4940, capacity 7904 = {...}, n_steps=10) at native_client/stt.cc:257
#6  0x00007ffff6e85931 in StreamingState::flushBuffers (this=0x7fff98000bc0, addZeroMfccVectors=true) at native_client/stt.cc:179
#7  0x00007ffff6e8575d in StreamingState::finishStream (this=0x7fff98000bc0) at native_client/stt.cc:143
#8  0x00007ffff6e868b9 in STT_FinishStream (aSctx=0x7fff98000bc0) at native_client/stt.cc:477
#9  0x000055555737955b in coqui_stt::stream::Stream::finish_stream (self=...) at /home/zero/.cargo/git/checkouts/coqui-stt-f6918436735af20a/2cb789b/src/stream.rs:221
#10 0x0000555555b25756 in core::ops::function::FnOnce::call_once<fn(coqui_stt::stream::Stream) -> core::result::Result<alloc::string::String, coqui_stt::errors::Error>, (coqui_stt::stream::Stream)> ()
    at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/core/src/ops/function.rs:227
#11 0x0000555555b2546a in core::ops::function::FnOnce::call_once<fn(coqui_stt::stream::Stream) -> core::result::Result<alloc::string::String, coqui_stt::errors::Error>, (coqui_stt::stream::Stream)> ()
    at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/core/src/ops/function.rs:227
#12 0x000055555737907c in alloc::boxed::{impl#44}::call_once<(coqui_stt::stream::Stream), (dyn core::ops::function::FnOnce<(coqui_stt::stream::Stream), Output=core::result::Result<alloc::string::String, coqui_stt::errors::Error>> + core::marker::Send + core::marker::Sync), alloc::alloc::Global> (self=..., args=...) at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/alloc/src/boxed.rs:1811
#13 0x000055555737ef10 in coqui_stt::threadsafe_stream::{impl#0}::send_and_get_recv_move::{closure#0}<core::result::Result<alloc::string::String, coqui_stt::errors::Error>> (stream=...)
    at /home/zero/.cargo/git/checkouts/coqui-stt-f6918436735af20a/2cb789b/src/threadsafe_stream.rs:76
#14 0x0000555557382de0 in core::ops::function::FnOnce::call_once<coqui_stt::threadsafe_stream::{impl#0}::send_and_get_recv_move::{closure#0}, (coqui_stt::stream::Stream)> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/core/src/ops/function.rs:227
#15 0x0000555557378ff5 in alloc::boxed::{impl#44}::call_once<(coqui_stt::stream::Stream), (dyn core::ops::function::FnOnce<(coqui_stt::stream::Stream), Output=()> + core::marker::Send + core::marker::Sync), alloc::alloc::Global> (self=..., args=...)
    at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/alloc/src/boxed.rs:1811
#16 0x000055555737e353 in coqui_stt::threadsafe_stream::{impl#0}::new::{closure#0} () at /home/zero/.cargo/git/checkouts/coqui-stt-f6918436735af20a/2cb789b/src/threadsafe_stream.rs:40
#17 0x00005555573b854e in std::sys_common::backtrace::__rust_begin_short_backtrace<coqui_stt::threadsafe_stream::{impl#0}::new::{closure#0}, ()> (f=...) at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/sys_common/backtrace.rs:123
#18 0x0000555557381952 in std::thread::{impl#0}::spawn_unchecked::{closure#1}::{closure#0}<coqui_stt::threadsafe_stream::{impl#0}::new::{closure#0}, ()> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/thread/mod.rs:477
#19 0x00005555573b8512 in core::panic::unwind_safe::{impl#23}::call_once<(), std::thread::{impl#0}::spawn_unchecked::{closure#1}::{closure#0}> (self=..., _args=()) at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/core/src/panic/unwind_safe.rs:271
#20 0x00005555573b9044 in std::panicking::try::do_call<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#1}::{closure#0}>, ()> (data=0x7fffb37fa968)
    at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/panicking.rs:406
#21 0x00005555573ba9cb in __rust_try ()
#22 0x00005555573b8f80 in std::panicking::try<(), core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#1}::{closure#0}>> (f=...) at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/panicking.rs:370
#23 0x00005555573acf02 in std::panic::catch_unwind<core::panic::unwind_safe::AssertUnwindSafe<std::thread::{impl#0}::spawn_unchecked::{closure#1}::{closure#0}>, ()> (f=...) at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/panic.rs:133
#24 0x0000555557381761 in std::thread::{impl#0}::spawn_unchecked::{closure#1}<coqui_stt::threadsafe_stream::{impl#0}::new::{closure#0}, ()> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/std/src/thread/mod.rs:476
#25 0x0000555557382dff in core::ops::function::FnOnce::call_once<std::thread::{impl#0}::spawn_unchecked::{closure#1}, ()> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/core/src/ops/function.rs:227
#26 0x00005555575bb943 in alloc::boxed::{impl#44}::call_once<(), dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/alloc/src/boxed.rs:1811
#27 alloc::boxed::{impl#44}::call_once<(), alloc::boxed::Box<dyn core::ops::function::FnOnce<(), Output=()>, alloc::alloc::Global>, alloc::alloc::Global> () at /rustc/928783de663bd855a96f14b2d38c1061603587c6/library/alloc/src/boxed.rs:1811
#28 std::sys::unix::thread::{impl#2}::new::thread_start () at library/std/src/sys/unix/thread.rs:108
#29 0x00007ffff61f4ea7 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00007ffff5fd8def in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I would be willing to upload a rr session privately somehow though if that's possible.

[reuben]

You could upload the session to https://pernos.co/ and then send me the recording URL privately over email. The URL is not guessable so. My email is my github username at coqui dot ai

[tazz4843]

I tried recording this with rr but perf tanked so severely it was essentially impossible to reproduce just from how long it took to run a single transcription. I'm really not sure what else to try at this point

[tazz4843]

I know this is rather old at this point, but I was finally able to reproduce this (on latest git master, with a debug build of the lib), and can send you a link if you're still able to try and figure this out.

[tazz4843]

I finally did fix this out. See scripty-bot/scripty#4 for details. Mentioning in the docs that a Model can only be used by one StreamingState at a time would really have helped here.