Using opentelemetry-otlp without an async runtime

[V0ldek]

I'm trying to configure a tracing provider for opentelemetry without having any async runtime, but I constantly get "there is no reactor running, must be called from the context of a Tokio 1.x runtime" errors even with install_simple.

At first I tried this code:

let exporter = opentelemetry_otlp::new_exporter()
    .tonic()
    .with_endpoint(oltp_endpoint);
opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(exporter)
    .with_trace_config(trace_config)
    .install_simple()?

expecting install_simple instead of install_batch(Tokio) to work without a runtime. It doesn't, tonic requires Tokio somewhere internally. I found the http-proto feature that is supposed to allow you to use something else than tonic, pulled in opentelementry-http with reqwest enabled and reqwest itself, and changed the exporter configuration to:

let exporter = opentelemetry_otlp::new_exporter()
    .http()
    .with_http_client(reqwest::Client::new())
    .with_endpoint(oltp_endpoint);

This allows the pipeline to build, but as soon as I run actual logic code and report a span I get the reactor error again. Here's the full stacktrace (omitting my application stack):

thread '<unnamed>' panicked at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/dns.rs:122:24:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/panicking.rs:74:14
   2: core::panicking::panic_display
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/panicking.rs:264:5
   3: tokio::runtime::scheduler::Handle::current::panic_cold_display
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/panic.rs:100:13
   4: tokio::runtime::scheduler::Handle::current
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/scheduler/mod.rs:106:27
   5: tokio::runtime::handle::Handle::current
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/handle.rs:139:20
   6: tokio::runtime::blocking::pool::spawn_blocking
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/runtime/blocking/pool.rs:182:14
   7: tokio::task::blocking::spawn_blocking
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.38.0/src/task/blocking.rs:207:9
   8: <hyper_util::client::legacy::connect::dns::GaiResolver as tower_service::Service<hyper_util::client::legacy::connect::dns::Name>>::call
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/dns.rs:122:24
   9: <reqwest::dns::gai::GaiResolver as reqwest::dns::resolve::Resolve>::resolve
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/dns/gai.rs:26:18
  10: <reqwest::dns::resolve::DynResolver as tower_service::Service<hyper_util::client::legacy::connect::dns::Name>>::call
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/dns/resolve.rs:75:9
  11: <S as hyper_util::client::legacy::connect::dns::sealed::Resolve>::resolve
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/dns.rs:290:13
  12: hyper_util::client::legacy::connect::dns::resolve::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/dns.rs:300:5
  13: hyper_util::client::legacy::connect::http::HttpConnector<R>::call_async::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/http.rs:432:18
  14: <hyper_util::client::legacy::connect::http::HttpConnector<R> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/http.rs:365:62
  15: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/future/future.rs:123:9
  16: <hyper_util::client::legacy::connect::http::HttpConnecting<R> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/connect/http.rs:514:9
  17: <hyper_tls::client::HttpsConnector<T> as tower_service::Service<http::uri::Uri>>::call::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-tls-0.6.0/src/client.rs:150:34
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/future/future.rs:123:9
  19: <hyper_tls::client::HttpsConnecting<T> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-tls-0.6.0/src/client.rs:179:9
  20: reqwest::connect::Connector::connect_with_maybe_proxy::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/connect.rs:298:41
  21: reqwest::connect::with_timeout::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/connect.rs:483:11
  22: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/future/future.rs:123:9
  23: <tower::util::oneshot::Oneshot<S,Req> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tower-0.4.13/src/util/oneshot.rs:97:38
  24: <F as futures_core::future::TryFuture>::try_poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  25: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  26: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  27: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  28: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  29: <F as futures_core::future::TryFuture>::try_poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  30: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
  31: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
  32: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  33: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  34: <F as futures_core::future::TryFuture>::try_poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
  35: <futures_util::future::try_future::try_flatten::TryFlatten<Fut,<Fut as futures_core::future::TryFuture>::Ok> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/try_flatten.rs:49:61
  36: <futures_util::future::try_future::TryFlatten<Fut1,Fut2> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  37: <futures_util::future::try_future::AndThen<Fut1,Fut2,F> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
  38: <futures_util::future::either::Either<A,B> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/either.rs:109:32
  39: <hyper_util::common::lazy::Lazy<F,R> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/common/lazy.rs:71:28
  40: futures_util::future::future::FutureExt::poll_unpin
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/mod.rs:558:9
  41: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/select.rs:118:35
  42: hyper_util::client::legacy::client::Client<C,B>::one_connection_for::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/client.rs:438:49
  43: hyper_util::client::legacy::client::Client<C,B>::connection_for::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/client.rs:389:61
  44: hyper_util::client::legacy::client::Client<C,B>::try_send_request::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/client.rs:272:14
  45: hyper_util::client::legacy::client::Client<C,B>::send_request::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/client.rs:244:70
  46: <hyper_util::client::legacy::client::ResponseFuture as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hyper-util-0.1.8/src/client/legacy/client.rs:707:9
  47: <reqwest::async_impl::client::PendingRequest as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/async_impl/client.rs:2552:53
  48: <reqwest::async_impl::client::Pending as core::future::future::Future>::poll
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/reqwest-0.12.7/src/async_impl/client.rs:2522:51
  49: opentelemetry_http::reqwest::<impl opentelemetry_http::HttpClient for reqwest::async_impl::client::Client>::send::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry-http-0.25.0/src/lib.rs:75:54
  50: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/future/future.rs:123:9
  51: opentelemetry_otlp::exporter::http::trace::<impl opentelemetry_sdk::export::trace::SpanExporter for opentelemetry_otlp::exporter::http::OtlpHttpClient>::export::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry-otlp-0.25.0/src/exporter/http/trace.rs:50:49
  52: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/future/future.rs:123:9
  53: futures_executor::local_pool::block_on::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:23
  54: futures_executor::local_pool::run_executor::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:90:37
  55: std::thread::local::LocalKey<T>::try_with
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/std/src/thread/local.rs:283:12
  56: std::thread::local::LocalKey<T>::with
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/std/src/thread/local.rs:260:9
  57: futures_executor::local_pool::run_executor
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:86:5
  58: futures_executor::local_pool::block_on
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-executor-0.3.30/src/local_pool.rs:317:5
  59: <opentelemetry_sdk::trace::span_processor::SimpleSpanProcessor as opentelemetry_sdk::trace::span_processor::SpanProcessor>::on_end::{{closure}}
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry_sdk-0.25.0/src/trace/span_processor.rs:133:38
  60: core::result::Result<T,E>::and_then
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/result.rs:1346:22
  61: <opentelemetry_sdk::trace::span_processor::SimpleSpanProcessor as opentelemetry_sdk::trace::span_processor::SpanProcessor>::on_end
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry_sdk-0.25.0/src/trace/span_processor.rs:129:22
  62: opentelemetry_sdk::trace::span::Span::ensure_ended_and_exported
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry_sdk-0.25.0/src/trace/span.rs:223:17
  63: <opentelemetry_sdk::trace::span::Span as core::ops::drop::Drop>::drop
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/opentelemetry_sdk-0.25.0/src/trace/span.rs:245:9
  64: core::ptr::drop_in_place<opentelemetry_sdk::trace::span::Span>
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/ptr/mod.rs:574:1
  65: <tracing_opentelemetry::layer::OpenTelemetryLayer<S,T> as tracing_subscriber::layer::Layer<S>>::on_close
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-opentelemetry-0.26.0/src/layer.rs:1104:65
  66: <tracing_subscriber::layer::layered::Layered<L,S> as tracing_core::subscriber::Subscriber>::try_close
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-subscriber-0.3.18/src/layer/layered.rs:196:13
  67: tracing_core::dispatcher::Dispatch::try_close
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-core-0.1.32/src/dispatcher.rs:705:9
  68: <tracing::span::Span as core::ops::drop::Drop>::drop
             at /home/mat/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/span.rs:1463:13
  69: core::ptr::drop_in_place<tracing::span::Span>
             at /rustc/f79a912d9edc3ad4db910c0e93672ed5c65133fa/library/core/src/ptr/mod.rs:574:1

What do I need to do to run opentelemetry collection from my app without any async runtime?

[cijothomas]

Interesting. The http call is done from a futures_executor::block_on().... Wondering if this is not supported by Reqwest/other crates.

@TommyCpp Could use your help here, as I am unsure of the details here.

@V0ldek Separately, can you share why are you not using an async time? i am curious to know the details, as we are currently working to make OTel work without any async runtimes, and instead leveraging a dedicated thread spun up by OTel itself.

[cijothomas]

This looks like same/similar to #2071
with a plausible root cause from: seanmonstar/reqwest#1215 (comment)

@V0ldek
Do you see the same issue when you use OTel Tracing API to create a span? Or is it only when using tracing macros on async functions like below?

#[tracing::instrument]
async fn some_function() {
    info!("some_function()");
    inner_function().await;
    info!("inner function done.");
}

[V0ldek]

The entire execution path that results in the stacktrace I've shown is synchronous (no async functions at all), however I do use tracing::instrument. I can check what happens if I remove all of them...

[cijothomas]

@V0ldek checking if you had a chance to try without tracing::instrument..

[V0ldek]

@cijothomas

Hey, sorry, I was on sick leave the entire week!

Removing instrument doesn't help, the error still gets thrown when a span is dropped. In particular in a function that looks like this:

// stuff
tracing::info_span!("my name").in_scope(|| {
  // stuff
})?;
// more stuff

you get the same stacktrace starting from core::ptr::drop_in_place<tracing::span::Span>.

[cijothomas]

Thanks @V0ldek We now have a good understanding of these issues. Working on resolving them one by one.
#2189 is related.