Parachain Collator Node Panic Due to Missing set_validation_data Inherent

[asiniscalchi]

As a developer/operator of the parachain node, I want the node to run without panics and include all necessary inherents so that it maintains consensus and functions correctly within the network.

Our parachain collator node is experiencing a critical panic during block production, caused by a missing set_validation_data inherent in some blocks. This issue does not halts the node's operation.

Steps to Reproduce

1. Connect to a collator node
2. Observe the node's logs for any panic messages or errors.
3. Notice the panic occurring with messages related to missing set_validation_data inherent.

Expected Behavior

• Blocks should be produced without panics or critical errors.

Actual Behavior

• The node panics during block production with the following errors:

panicked at ... parachain-system/src/lib.rs:266:18: set_validation_data inherent needs to be present in every block!

Possible Causes

1. Runtime Code Issues

• Bugs or omissions in custom runtime code or pallets.
• Failure to correctly implement or include essential inherents like set_validation_data.

Notes

• This issue may affect consensus and the overall health of the parachain network if not addressed promptly.

[asiniscalchi]

2024-09-20 06:18:20 [Relaychain] 💤 Idle (9 peers), best: #22614030 (0x8f26…024d), finalized #22614027 (0x5870…68fc), ⬇ 272.6kiB/s ⬆ 315.5kiB/s
2024-09-20 06:18:20 [Parachain] 💤 Idle (7 peers), best: #553955 (0xf1b5…09eb), finalized #553952 (0x8fb2…6b48), ⬇ 8.8kiB/s ⬆ 3.7kiB/s
2024-09-20 06:18:24 [Relaychain] 🏆 Imported #22614031 (0x8f26…024d → 0xe0ff…123b)
2024-09-20 06:18:24 [Parachain] 🙌 Starting consensus session on top of parent 0xf1b547c185ff8dfd0698fed64eb2ba012304c3b023ad3f40b8d99741a03e09eb
2024-09-20 06:18:24 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/8c8edac/cumulus/pallets/parachain-system/src/lib.rs:1281:17:
assertion `left == right` failed: expected parent to be included
  left: 0x3f1075864e87e9da1fb19efc5d06811a4f7be789796fbc6f2f35e570782e37ff
 right: 0xf1b547c185ff8dfd0698fed64eb2ba012304c3b023ad3f40b8d99741a03e09eb
2024-09-20 06:18:24 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-09-20 06:18:24 [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
2024-09-20 06:18:24 [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x3f6437 - <unknown>!rust_begin_unwind
    1: 0xbc29 - <unknown>!core::panicking::panic_fmt::h7a368385936888dc
    2: 0xcb86 - <unknown>!core::panicking::assert_failed_inner::h8608b548c883ea8f
    3: 0x2f735f - <unknown>!core::panicking::assert_failed::hc1bd59472f668bd1
    4: 0x1a952a - <unknown>!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors::h771d663d983a5c6f
    5: 0x294757 - <unknown>!frame_support::storage::transactional::with_transaction::h6053a02eea4a3ab4
    6: 0x39656e - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h09cc640031ce2828
    7: 0x397e2b - <unknown>!environmental::local_key::LocalKey<T>::with::h1093f7d8f2027179
    8: 0xc3899 - <unknown>!<laos_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::he1f5a24e17db6cb3
    9: 0xbf24a - <unknown>!<laos_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h52fe072214ee0bd0
   10: 0x100624 - <unknown>!<fp_self_contained::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra,SelfContainedSignedInfo> as sp_runtime::traits::Applyable>::apply::h7ff24cb636aac3e4
   11: 0x258080 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h4d301b1e51ecbe7f
   12: 0x26bfe9 - <unknown>!BlockBuilder_apply_extrinsic. Dropping.
2024-09-20 06:18:24 [Parachain] panicked at /home/builder/cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/8c8edac/cumulus/pallets/parachain-system/src/lib.rs:266:18:
set_validation_data inherent needs to be present in every block!
2024-09-20 06:18:24 [Parachain] err=Error { inner: Proposing

Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x3f6437 - <unknown>!rust_begin_unwind
           1: 0xbc29 - <unknown>!core::panicking::panic_fmt::h7a368385936888dc
           2: 0xc7cc - <unknown>!core::option::expect_failed::h6f4ce1bb06a99e85
           3: 0x1a3ea7 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::hbaa5c6bc09c4d26e
           4: 0x2588fc - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h2e7c9ca9b6bc59c4
           5: 0x259013 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h3989f09d94ae47eb
           6: 0x26c0a6 - <unknown>!BlockBuilder_finalize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x3f6437 - <unknown>!rust_begin_unwind
           1: 0xbc29 - <unknown>!core::panicking::panic_fmt::h7a368385936888dc
           2: 0xc7cc - <unknown>!core::option::expect_failed::h6f4ce1bb06a99e85
           3: 0x1a3ea7 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::hbaa5c6bc09c4d26e
           4: 0x2588fc - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h2e7c9ca9b6bc59c4
           5: 0x259013 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h3989f09d94ae47eb
           6: 0x26c0a6 - <unknown>!BlockBuilder_finalize_block

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::new
   1: <cumulus_client_consensus_proposer::Proposer<B,T> as cumulus_client_consensus_proposer::ProposerInterface<B>>::propose::{{closure}}
   2: cumulus_client_consensus_aura::collators::basic::run::{{closure}}
   3: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
   4: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
   5: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
   6: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
   7: tokio::runtime::task::core::Core<T,S>::poll
   8: tokio::runtime::task::harness::Harness<T,S>::poll
   9: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run
  11: tokio::runtime::context::set_scheduler
  12: tokio::runtime::context::runtime::enter_runtime
  13: tokio::runtime::scheduler::multi_thread::worker::run
  14: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  15: tokio::runtime::task::core::Core<T,S>::poll
  16: tokio::runtime::task::harness::Harness<T,S>::poll
  17: std::sys_common::backtrace::__rust_begin_short_backtrace
  18: core::ops::function::FnOnce::call_once{{vtable.shim}}
  19: std::sys::pal::unix::thread::Thread::new::thread_start
  20: <unknown>
  21: __clone }
2024-09-20 06:18:25 [Parachain] 💤 Idle (7 peers), best: #553955 (0xf1b5…09eb), finalized #553953 (0x7a73…de8a), ⬇ 6 B/s ⬆ 6 B/s
2024-09-20 06:18:25 [Relaychain] 💤 Idle (9 peers), best: #22614031 (0xe0ff…123b), finalized #22614028 (0x15de…8b27), ⬇ 107.7kiB/s ⬆ 250.3kiB/s
2024-09-20 06:18:30 [Relaychain] 💤 Idle (9 peers), best: #22614031 (0xe0ff…123b), finalized #22614029 (0x8a33…b3a6), ⬇ 73.7kiB/s ⬆ 196.3kiB/s
2024-09-20 06:18:30 [Parachain] 💤 Idle (7 peers), best: #553955 (0xf1b5…09eb), finalized #553953 (0x7a73…de8a), ⬇ 0.2kiB/s ⬆ 0.2kiB/s
2024-09-20 06:18:30 [Relaychain] 🏆 Imported #22614032 (0xe0ff…123b → 0x61fc…dcaf)
2024-09-20 06:18:30 [Parachain] 🙌 Starting consensus session on top of parent 0xf1b547c185ff8dfd0698fed64eb2ba012304c3b023ad3f40b8d99741a03e09eb
2024-09-20 06:18:30 [Parachain] 🎁 Prepared block for proposing at 553956 (1 ms) [hash: 0x0ffcca80b75b7cbb164f8897b3d8658dd937bc923651120572062c4de934793e; parent_hash: 0xf1b5…09eb; extrinsics (2): [0x8ff0…d819, 0x8c88…26f6]
2024-09-20 06:18:30 [Parachain] 🔖 Pre-sealed block for proposal at 553956. Hash now 0x09640e7f90e8cb84b344e2db89bf1b3ca09a1cd7c3c1335e1d67509ca04db1c7, previously 0x0ffcca80b75b7cbb164f8897b3d8658dd937bc923651120572062c4de934793e.
2024-09-20 06:18:30 [Parachain] 🏆 Imported #553956 (0xf1b5…09eb → 0x0964…b1c7)

[asiniscalchi]

deployed in Mercury with runtime - 2192 .... waiting to see if the runtime panic disappears