Firehose Cursor Reset for bsky.network

[bnewbold]

The Firehose from the Relay at bsky.network will have the cursor reset on Wednesday, February 21st at 10AM PST. The DNS record for bsky.network will be updated to point at a new host which will have a new cursor.

Behind the scenes, we have multiple Relay servers running in different regions and service providers. We are switching which instance is the primary public relay ahead of enabling federation. The newer instance has a different IP address, operates from our own IP space and hardware, and has a different cursor sequence.

(h/t @ericvolp12 for this write-up)

Who is Affected?

Firehose consumers who depend on PERFECT data consistency from the firehose, or consumers who will fail if the sequence number returned from the firehose is less than a previously seen sequence number.

What should I do?

If you're running a feed generator or other firehose-consuming infrastructure where missing an event or two isn't the end of the world, you don't need to do anything. Reconnecting to the Relay with your existing cursor will tell the new Relay to restart your stream from a live tail of events.

If you rely on PERFECT data consistency from the firehose, you can update your consumer to do the following when it reconnects to the firehose and gets a new, lower seq number cursor:

• Ensure event processing for your consumer is idempotent
• Rewind the returned seq by ~10,000 events
• Reconnect to the firehose and re-process events to make sure nothing was missed

[mackuba]

So I'm assuming that when the switch happens, my client will stay connected and streaming from the old server, until it loses connection at some point or I disconnect it, and at that moment it will reconnect to the new one instead? So I don't have to do anything precisely at 10:00:00?

[ericvolp12]

Yeah, we'll probably restart the old relay at some point in that 24 hour period to force people to reconnect to the new one after DNS changes have a chance to propagate.

[bnewbold]

yup, that is the expectation. at some time after 10am, either naturally, or if we do a restart of the old instance, clients would re-connect at the same hostname/endpoint and get the "new" instance.

[mackuba]

@ericvolp12 can you tell me if this sounds like a good strategy?

• shortly after 10AM PST, connect to the new firehose for a moment, write down the cursor seq (confirm that it's a different one) (or just ask someone else - maybe you can just post it in #general ?) and disconnect
• wait a moment just to be sure
• stop your firehose client
• reset the saved cursor to what you've got in point 1
• connect to the new firehose
• it should now catch up from that saved seq, which means you should have just a little bit of overlap between what was saved from the old one and what you get from the new one now

[mackuba]

Ok, second strategy :D

• run a firehose client and log events of your own posts
• after 10AM, start a second firehose client in parallel
• post something on Bluesky and write down the seq printed on both clients (seq1 and seq2)
• stop both clients
• update the saved cursor by the difference between seq1 and seq2 (cursor := cursor - (seq1 - seq2))
• restart the firehose and you're catching up with 0 overlap and 0 gap

[ericvolp12]

The way I've been doing it is using the Gosky read-stream command to grab a sequence number from the new Firehose, then wait like 30 seconds, shutdown my existing consumer, update the cursor to the one I just grabbed, then bring it back up.

[redsolver]

How to make sure my consumer connects to the new instance on restart?