Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate why the Chainhook chain segment can become outdated #623

Open
csgui opened this issue Jul 16, 2024 · 1 comment
Open

Investigate why the Chainhook chain segment can become outdated #623

csgui opened this issue Jul 16, 2024 · 1 comment
Assignees
Labels
bug Something isn't working

Comments

@csgui
Copy link
Contributor

csgui commented Jul 16, 2024

An issue has happened, in production, where the Chainhook’s chain segment became outdated.

And since the code base is not prepared to handle that, only restoring the Chainhook's chain state solves this issue.

https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L255
https://github.com/hirosystems/chainhook/blob/main/components/chainhook-sdk/src/indexer/mod.rs#L414

The following log messages where triggered in the case of that situation.

Will have to fork: OutdatedSegment
Unable to process Stacks Block #158000 (0x089f...445b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157999 (0x2ee0...087b) - inboxed for later
unable to infer chain progress
...
Will have to fork: OutdatedSegment
Unable to process Stacks Block #157998 (0x4353...70af) - inboxed for later
unable to infer chain progress

The work here is to find the root cause for why this can happen.

@smcclellan smcclellan added the bug Something isn't working label Jul 29, 2024
@smcclellan
Copy link
Contributor

smcclellan commented Jul 29, 2024

I see two issues here:

  1. A spike to investigate why the chain state became outdated, ideally reproducing the issue
  2. Make the system more resilient so it can recover from an outdated chain state

Let's have this ticket be the first one, and I'll create a separate ticket for the second one.

@smcclellan smcclellan changed the title Issue when Chainhook chain segment became outdated Investigate why the Chainhook chain segment can become outdated Jul 29, 2024
@rafaelcr rafaelcr self-assigned this Oct 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📋 Backlog
Development

No branches or pull requests

3 participants