Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loss transactions in full node database #2889

Open
tdnguyenND opened this issue Jan 16, 2024 · 9 comments
Open

Loss transactions in full node database #2889

tdnguyenND opened this issue Jan 16, 2024 · 9 comments
Assignees
Labels
other: decayed Stale issues that need follow up from commentators. Were closed for inactivity scope: comet-bft

Comments

@tdnguyenND
Copy link

tdnguyenND commented Jan 16, 2024

Problem

Some transactions are missing (not from latest block)

Closing criteria

Deploy full node successfully with no missing transactions

Problem details

When I deploy a full node cosmos, I found that some transactions are missing from my database (i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D - which I found exists in mintscan. For data analyzing purpose, I expect my database to have all the neccessary transaction.
So the question is how can I download the missing transaction?

How to reproduce this problems?

  1. Install gaiad & cosmovisor. I'm using version 13.0.2
    image
  2. Download archieved data source from quicksync. At the moment I downloaded, it was 8.5TB, sadly I don't remember the latest block when downloading it
  3. Unzip downloaded file and put in into the correct database folder
  4. Start gaiad app
  5. Run search for transaction
curl http://localhost:26657/tx?hash=0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

Expected result: Return transaction detail.
Actual result: {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (AA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD) not found"}}

In fact, all 3 transactions from block 7412749 are missed. Although they're all searchable at mintscan, I can't find it from any other explorer platform (maybe mintscan just simply decode from block data, i guess?)

@tdnguyenND tdnguyenND added the status: waiting-triage This issue/PR has not yet been triaged by the team. label Jan 16, 2024
@mmulji-ic
Copy link
Contributor

Thanks @tdnguyenND for reporting, I'll check with the Comet team why this has occurred, they're supporting the db and p2p layers. I'm assuming that you have transactions before and after this transaction. Also how did you stumble across this transaction that was missing?

@andynog
Copy link

andynog commented Jan 17, 2024

Hi @tdnguyenND, I'm Andy from the CometBFT team, in order to help @mmulji-ic troubleshoot this, could you please let us know how you're querying for this transaction using your full node ?

@tdnguyenND
Copy link
Author

@andynog I have provide more detail in problem description. Hope it help

@andynog
Copy link

andynog commented Jan 22, 2024

Thanks @tdnguyenND, what do you get when you query the block ?

curl http://localhost:26657/block?height=7412749

@tdnguyenND
Copy link
Author

tdnguyenND commented Jan 29, 2024

The block is still fine with all txns included
7412749.json

@mmulji-ic
Copy link
Contributor

@andynog any further thoughts?

@mmulji-ic mmulji-ic removed the status: waiting-triage This issue/PR has not yet been triaged by the team. label Feb 29, 2024
@andynog
Copy link

andynog commented Apr 1, 2024

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

@tdnguyenND
Copy link
Author

tdnguyenND commented Apr 5, 2024

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

Maybe I was wrongly referring the command, but I'm sure that some transaction were somehow removed from quicksync's archieved file - i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D (I have a list of transactions that are missed)

Unfortunately this happened quite long and I decided to ignore them, my fullnode no longer exists so I can't recreate the scenario.

@a26nine
Copy link

a26nine commented May 16, 2024

I am seeing this as well. For example, 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6 exisats on https://www.mintscan.io/cosmos/tx/8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6?height=15213800 , but the node gives : {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6) not found"}}

@mpoke mpoke added the other: decayed Stale issues that need follow up from commentators. Were closed for inactivity label Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
other: decayed Stale issues that need follow up from commentators. Were closed for inactivity scope: comet-bft
Projects
Status: 🤔 F1: Investigate
Development

No branches or pull requests

6 participants