Mode to sync chain first + index later

[tzarebczan]

I've been playing around with Rosetta for LBRY, and I've noticed I have lots of trouble on the initial sync + index when we start getting some larger blocks come in. I did see a recent discussion around improvements, but I haven't tried them yet (just saw them today).

Logs when this happens - almost looks like it fails to process a block in time, and then logs that block on shutdown along with the http request errors:

ss=0.005344 cache=2073.6MiB(5280827txo) IBD
2020-12-07T12:29:34Z UpdateTip: new best=2e51cbf8dfbd8820dd3bbdd55d1097c625414b7b8cb53190890eddfb135e1ab5 height=837104 version=0x20000000 log2_work=73.933872 txb=602 tx=20026401 date='2020-09-26T17:06:41Z' progress=0.005351 cache=2075.2MiB(5285550txo) IBD
2020-12-07T12:29:50Z UpdateTip: new best=6d6bfe8ca9db93626e88cc19e8ace45738730679995a0cf99bdba003b6cf19bb height=837117 version=0x20000000 log2_work=73.933907 txb=177 tx=20028012 date='2020-09-26T17:39:32Z' progress=0.005353 cache=2075.6MiB(5287003txo) IBD
2020-12-07T12:40:53Z tor: Thread interrupt
2020-12-07T12:40:53Z addcon thread exit
2020-12-07T12:40:53Z torcontrol thread exit
2020-12-07T12:40:53Z opencon thread exit
2020-12-07T12:40:53Z Shutdown: In progress...
2020-12-07T12:40:53Z net thread exit
2020-12-07T13:05:39Z UpdateTip: new best=7bfa4f19d91c3ed3dfa304ee3e82e2d7765bd8335d7802bcc24ad0e19ac02945 height=837138 version=0x20000000 log2_work=73.933963 txb=120 tx=20029990 date='2020-09-26T18:35:35Z' progress=0.005355 cache=57.7MiB(0txo) IBD
2020-12-07T13:05:39Z msghand thread exit
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z ~HTTPRequest: Unhandled request
2020-12-07T13:05:39Z scheduler thread interrupt
2020-12-07T13:05:39Z Dumped mempool: 0.001684s to copy, 0.000714s to dump
2020-12-07T13:05:40Z Shutdown: done

Would it make sense to have a mode that lets the blockchain sync up fully before processing the blocks?

[patrick-ogrady]

I've been playing around with Rosetta for LBRY

🚀 Very cool!

I did see a recent discussion around improvements, but I haven't tried them yet (just saw them today).

We just released a big upgrade yesterday (unfortunately requires you to re-sync) but should make things a lot faster!

Would it make sense to have a mode that lets the blockchain sync up fully before processing the blocks?

I think that is definitely an interesting idea (especially for folks running things on smaller boxes). I'd definitely review a PR that added this functionality.

Just curious, what type of box are you using (CPU, RAM, SSD/HDD)?