[Question]: Rpc error "could not find results for height #21835191"

[abotau]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Two Gaia nodes running v18.1.0, directly with the binaries. I upgraded them to v19.1.0 from release files, with a restart from snapshot (version 20240822-0310) Nodes both started and are running almost perfectly. Except for this error :

ERR failed to LoadFinalizeBlockResponse err="could not find results for height #21835191" module=rpc   

I can fetch the block directly with curl and the APIs, e.g. with

$ curl --header "Content-Type: application/json" --request POST --data '{"method": "block", "params": ["21835191"], "id": 1}' localhost:26657 > block.json

but it looks like there is a problem with the gaia rpc calls.

Gaia Version

v19.1.0

How to reproduce?

No response

[MSalopek]

Thanks for opening the issue!

Unfortunately this is not a bug but expected behaviour.

The block that you are trying to query was created using v18.1.0 and not v19.1.0.

v19.1.0 is creating blocks starting from 21,835,200, but your block is 9 heights before the upgrade (21,835,191) - this means you can query that block using v18.1.0.

This is due to the large data infrastructure changes between cometbft v0.37.x and v0.38.x.

Gaia v18.1.0 is using cometbft v0.37.x -> that has BeginBlockResponse and EndBlockResponse.
Gaia v19.1.0 is using cometbft v0.38.x -> that has FinalizeBlockResponse.

The 2 responses are not backward compatible, they simply have data organized in a non-backward compatible way.

How this is usually addressed:

You can index the block data and events using the old binary and have it available for your services that rely on historic data.

This is unfortunate, but the backward compatibility is not preserved during this upgrade due to the upgrades to dependencies.

Please let us know if you need further assistance.