in-memory realtime sync #1115

kyscott18 · 2024-09-20T01:12:02Z

The "sync" portion of ponder handling extracting and ordering events is entirely in-memory, removing the need for database queries. Another property is that only finalized data is added to the sync-store. This is a combination of #1092 and #1103. Closes #1091.

This involves:

storing child address information in memory
filtering, formatting, and ordering events (replacing getEvents())

Child address methodology

separate unfinalizedChildAddresses and finalizedChildAddresses
track factoryLogsPerBlock, so that unfinalizedChildAddresses can always be recomputed
on reorg, evict reorged blocks from factoryLogsPerBlock and recompute unfinalizedChildAddresses
on finalization, add child addresses from finalized blocks to finalizedChildAddresses, evict finalized blocks from factoryLogsPerBlock and recompute unfinalizedChildAddresses

Sync ordering methodology

store unindexed events in unindexedEvents
store unfinalized event data in unfinalizedEventData
when a new block is received, format with buildEvents() and add to unindexedEvents
when the overall checkpoint is moved forward, select events that are now able to be indexed, remove them from unindexedEvents, and pass them to the indexing functions
on reorg, removed events with reorged block hashes from unindexedEvents and unfinalizedEventData
on finalization, print a warning if the chain's finalized checkpoint is farther along than the overall indexing checkpoint
on finalization, removed newly finalized data from unfinalizedEventData and insert it into sync-store

Other

A metric to track how far away from tip the indexing progress is (known block - indexed block) might be helpful. However, it also may be unintuitive for many consumers when using omnichain ordering as the metric will appear far behind tip quite consistently

0xOlias

Looking great, couple questions.

packages/core/src/sync-realtime/index.ts

0xOlias · 2024-10-08T18:28:20Z

packages/core/src/sync-realtime/index.ts

-      ) as SyncCallTrace[];
+      callTraces = traces
+        .filter((trace) => trace.type === "call")
+        .filter((trace) => trace.result !== null) as SyncCallTrace[];


What's the story behind this change?

Previously, we were relying on the getEvents query to filter out traces with error. Now it happens here

Makese sense. Though, this means we'll technically be storing different data in the database in historical vs realtime, correct? In realtime, we're filtering before insertion but the opposite in historical.

0xOlias · 2024-10-08T18:33:33Z

packages/core/src/sync-store/index.ts

-              transactionIndex: eb.ref("excluded.transactionIndex"),
-            })),
-          )
+          .onConflict((oc) => oc.column("hash").doNothing())


What's the context behind this change?

This previous logic was to handle the case when a reorg occurred. It no longer applies because all data in the sync-store is finalized.

0xOlias · 2024-10-08T19:04:46Z

packages/core/src/sync/index.ts

+          const events: RawEvent[] = unindexedEvents.filter(
+            ({ checkpoint }) => checkpoint <= to,
+          );
+          unindexedEvents = unindexedEvents.filter(


I find it a bit confusing that this is called unindexedEvents, because this service does not actually do the indexing. Am I correct that this is the list of events (across all chains) that the sync service is holding onto, but can't pass up to the runtime because there is another network with a lagging checkpoint? Essentially a buffer?

Yes, exactly. Is eventBuffer more clear?

Maybe bufferedEvents or pendingEvents ?

0xOlias · 2024-10-08T19:06:24Z

packages/core/src/sync/index.ts

+          ...args.sources
+            .filter(({ filter }) => filter.chainId === network.chainId)
+            .map(({ filter }) =>
+              args.syncStore.insertInterval({ filter, interval }),
+            ),


Really nice that this is now happening in the same spot - not necessary for this PR, but ultimately I think it would be great to use a database transaction here.

0xOlias · 2024-10-08T19:08:30Z

packages/core/src/sync/index.ts

+        localSyncContext.get(network)!.unfinalizedEventData =
+          unfinalizedEventData.filter(
+            (led) =>
+              hexToNumber(led.block.number) <= hexToNumber(event.block.number),
+          );

-          break;
+        const reorgedHashes = new Set<Hash>();
+        for (const b of event.reorgedBlocks) {
+          reorgedHashes.add(b.hash);
        }

-        default:
-          never(event);
+        unindexedEvents = unindexedEvents.filter(
+          (e) => reorgedHashes.has(e.block.hash) === false,
+        );


I'm getting a bit confused by the difference between unfinalizedEventData and unindexedEvents here. It seems like they contain the same information, why do we need both?

They do contain the same information, where unindexedEvents is derived from unfinalizedEventData. However, when building events, point-in-time child addresses are needed. Also, the lifetime of each of these variables is different.

0xOlias · 2024-10-08T19:09:11Z

packages/core/src/sync/index.ts

-      eventQueue.pause();
-      eventQueue.clear();
-      promises.push(eventQueue.onIdle());


Why'd you remove these?

The sync module no longer contains a queue. Instead, it is synchronous with sync-realtime. This is primarily to maintain consistency for factory addresses. Without this change, it would be almost impossible to maintain unfinalizedChildAddresses and pass the correct value for the correct block height to buildEvents.

0xOlias · 2024-10-08T19:10:17Z

packages/core/src/sync-realtime/index.ts

+    // @ts-ignore
+    block.transactions = undefined;
+
+    await args.onEvent({


Why did you add the await here?

#1115 (comment)

0xOlias · 2024-10-09T20:45:38Z

packages/core/src/sync/index.ts

+        ]);
+
+        // Add corresponding intervals to the sync-store
+        // Note: this should happen after so the database doesn't become corrupted


Nice catch, I bet this would have bit us.

0xOlias

LGTM

kyscott18 added 8 commits September 19, 2024 20:07

start in-memory realtime sync

f8ede93

fix realtime log

3468b61

notes

c0fcca4

no queue for onRealtimeSyncEvent

7f4e80c

pass factory information to buildEvents

5e067e5

cleanup

8ffa323

backpressure log

ec0a05c

Merge branch 'main' into kjs/mem

e2921c7

railway-app bot temporarily deployed to production September 24, 2024 00:04 Inactive

kyscott18 added 8 commits September 24, 2024 15:41

remove reorged events

b4e1b1e

on conflict do nothing

5d835cf

fix reorg

f35b9b2

filter out reverted call traces

1785257

fix finalize interval

cc7d1ed

fix reorg handling

d76cab5

filter call traces with function selector

c55168a

Merge branch 'main' into kjs/mem

473e7b2

railway-app bot temporarily deployed to production October 4, 2024 22:14 Inactive

vercel bot deployed to Preview October 4, 2024 22:15 View deployment

cleanup

5506b08

railway-app bot temporarily deployed to production October 7, 2024 23:57 Inactive

kyscott18 marked this pull request as ready for review October 8, 2024 00:24

kyscott18 requested a review from 0xOlias October 8, 2024 00:24

fix test

2d20879

railway-app bot temporarily deployed to production October 8, 2024 00:33 Inactive

0xOlias reviewed Oct 8, 2024

View reviewed changes

kyscott18 added 3 commits October 9, 2024 10:44

nits

b0c2dc0

chore: changeset

3292804

Merge branch 'main' into kjs/mem

a1a333f

railway-app bot temporarily deployed to production October 9, 2024 19:21 Inactive

0xOlias reviewed Oct 9, 2024

View reviewed changes

0xOlias approved these changes Oct 9, 2024

View reviewed changes

cleanup

fd8d59f

0xOlias merged commit 1a7d1f8 into main Oct 10, 2024
8 of 9 checks passed

0xOlias deleted the kjs/mem branch October 10, 2024 00:26

github-actions bot mentioned this pull request Oct 10, 2024

chore: version packages #1153

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

in-memory realtime sync #1115

in-memory realtime sync #1115

kyscott18 commented Sep 20, 2024 •

edited

Loading

0xOlias left a comment

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024

0xOlias Oct 9, 2024

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024

0xOlias Oct 9, 2024 •

edited

Loading

0xOlias Oct 8, 2024

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024

0xOlias Oct 8, 2024

kyscott18 Oct 9, 2024 •

edited

Loading

0xOlias Oct 9, 2024

0xOlias left a comment

in-memory realtime sync #1115

in-memory realtime sync #1115

Conversation

kyscott18 commented Sep 20, 2024 • edited Loading

Child address methodology

Sync ordering methodology

Other

0xOlias left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xOlias Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kyscott18 Oct 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xOlias left a comment

Choose a reason for hiding this comment

kyscott18 commented Sep 20, 2024 •

edited

Loading

0xOlias Oct 9, 2024 •

edited

Loading

kyscott18 Oct 9, 2024 •

edited

Loading