Archival bucketlist #4403

SirTyson · 2024-07-30T18:38:13Z

Description

Resolves #4394

Depends on XDR changes in stellar/stellar-xdr#200.

This change refactors the BucketList so that we can have multiple Bucket Lists simultaneously. This is necessary for State Archival, where validators will maintain a live BucketList, hot archive BucketList, and cold archive BucketList. This change does not yet support writing the new archival BucketLists to history,

Each BucketList has small differences wrt to the entry types it stores, merge logic, and how lookups are done, but the overall BucketList level logic is the same. Because of this, it seemed easiest to template the Bucket related classes and specialize a few functions. The difference are as follows (I'll be updating the Archival CAP with this info soon)

Entry Types:
Live BucketList: BucketEntry
Hot Archive BucketList: HotArchiveBucketEntry.

This change was necessary due to how LedgerKeys are stored. In the Live BucketList, only DEADENTRY store plain LedgerKey. DEADENTRY is equivalent to null, so these LedgerKeys are dropped at the bottom level bucket.

Hot Archives require two types that store LedgerKeys: HA_LIVE and HA_DELETED. HA_LIVE is equivalent to the "null" type in the Live BucketList, and is dropped in the bottom bucket. However, HA_DELETED needs to be persisted.

Merging:
In the hot archive, LedgerEntry are never updated. They may be overwritten by another HotArchiveBucketEntry type, but the LedgerEntry contents itself never change. This means that the INITENTRY, LIVEENTRY, and DEADENTRY abstraction doesn't really make sense. This makes the merge logic for Hot Archive buckets very simple, the newer entry always overwrites the older entry.

This PR adds the concept of a "Tombstone" entry. A tombstone entry essentially represents null in the given BucketList and can be dropped once it reaches the bottom bucket. On the live BucketList, DEADENTRY is the tombstone type. On the Hot Archive BucketList, HA_LIVE is the tombstone.

BucketListDB lookup:
The live BucketList only ever returns LedgerEntry types when queried. Any DEADENTRY LedgerKey is simply omitted from the return, as this is the "null" tombstone type.

Hot archive lookups must return both LedgerKey and LedgerEntry types. HA_LIVE entries are of type LedgerKey and can be omitted from the return since they are the tombstone type. However, HA_ARCHIVED entries are not tombstone entries and must be returned when found. Do to this, Hot Archive lookups return HotArchiveBucketEntry instead of LedgerEntry when queried.

Future updates will add more functionality to Hot Archives, further distinguishing it from the live BucketList, and add a third BucketList type called the cold archive. This cold archive will add a third BucketEntry type and have different merge logic as well, so I think the templated Bucket classes, while verbose, are warranted.

This is currently a draft until the XDR changes are merged.

Checklist

Reviewed the contributing document
Rebased on top of master (no merge commits)
Ran clang-format v8.0.0 (via make format or the Visual Studio extension)
Compiles
Ran all tests
If change impacts performance, include supporting evidence per the performance document

dmkozh · 2024-08-22T22:51:54Z

Cargo.toml

@@ -3,7 +3,7 @@ members = ["src/rust", "lib/tracy-client-sys"]

 [patch.crates-io]
 tracy-client-sys = { path = "lib/tracy-client-sys" }
-stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "ae805d0f8c28ca86327a834eea9ce7d29b0a63bb" }
+stellar-xdr = { git = "https://github.com/stellar/rs-stellar-xdr", rev = "9a31b81412712ab2dfbe32cbeaba6bdb754c264b" }


We need to bump this to the most recent revision and wrap all the new XDR uses into the next version macro (or wait until the protocol release)

dmkozh · 2024-08-22T22:53:02Z

src/main/main.cpp

@@ -382,7 +382,7 @@ main(int argc, char* const* argv)
    // built with vnext, causing a curr diff against next. This works now
    // because the xdr is indentical, but the moment that changes this checkk
    // will fail and will need to be fixed.
-    checkXDRFileIdentity();
+    // checkXDRFileIdentity();


Don't forget to revert this

dmkozh · 2024-08-22T23:09:19Z

src/bucket/BucketList.cpp

+template <> BucketListDepth BucketListBase<LiveBucket>::kNumLevels = 11;
+
+// TODO: I made this number up, do some analysis and pick a better value or
+// make this a configurable network config.


Just curious, can we actually change this value at runtime in-between ledgers?

No. We will probably end up making this a config setting at which point I'll change it to a dynamic runtime value, but for now it's easier just to inherit the older BucketList's global const value for this refactor.

dmkozh · 2024-08-22T23:16:25Z

src/bucket/BucketList.cpp

+
+    for (uint32_t i = static_cast<uint32>(mLevels.size()) - 1; i != 0; --i)
+    {
+        /*


Don't forget to remove this (or uncomment if that's universally useful for debuggin)

dmkozh · 2024-08-22T23:21:16Z

src/bucket/BucketManagerImpl.cpp

+        if (protocolVersionStartsFrom(currentHeader.ledgerVersion,
+                                      ProtocolVersion::V_22))
+        {
+            // TODO: Hash Archive Bucket


What's the status of this TODO? I.e. is the PR blocked on this?

dmkozh · 2024-08-22T23:21:46Z

src/bucket/BucketManagerImpl.cpp

@@ -1148,26 +1304,30 @@ BucketManagerImpl::assumeState(HistoryArchiveState const& has,
    ZoneScoped;
    releaseAssertOrThrow(mApp.getConfig().MODE_ENABLES_BUCKETLIST);

-    for (uint32_t i = 0; i < BucketList::kNumLevels; ++i)
+    // TODO: Assume archival bucket state


What about this TODO?

The outstanding TODOs are outside the scope of this PR. I've added some additional context on their dependencies. They require changing the history archive structure, which is a pretty big endeavor.

Let's maybe open an issue or a few issues for these - in general, Core codebase avoids TODOs merged, and if we do have some, I would prefer to have them point at the issues.

We do, the master issue has most of these covered in sub-issues: #4392. I agree it's a little messy but there's a lot of changes that are all kinda have dependencies and I'm trying to merge them in small pieces

src/bucket/BucketSnapshot.cpp

dmkozh · 2024-08-22T23:28:26Z

src/bucket/BucketSnapshot.h

+                  std::vector<BulkLoadReturnT>& result,
+                  LedgerKeyMeter* lkMeter) const;
+
+    // friend struct BucketLevelSnapshot<BucketT>;


Please remove

dmkozh · 2024-08-22T23:48:21Z

src/bucket/LedgerCmp.h

+            if (aty != HOT_ARCHIVE_DELETED && aty != HOT_ARCHIVE_LIVE)
+            {
+                throw std::runtime_error(
+                    "Malformed bucket: expected DELETED/RESTORED key.");


We don't have 'RESTORED' key category, do we?

dmkozh · 2024-08-22T23:56:00Z

src/util/ProtocolVersion.h

@@ -33,7 +33,8 @@ enum class ProtocolVersion : uint32_t
    V_18,
    V_19,
    V_20,
-    V_21
+    V_21,
+    V_22


Don't forget to update this to include v23 and use it where appropriate.
I also would suggest using a named constant instead of the direct version checks in order to make changes easier.

marta-lokhova · 2024-08-29T22:00:33Z

Note: changes in this PR impact BucketListDB quite a bit, I think we need to resolve #4433 to make sure we have good coverage. Wdyt?

dmkozh

LGTM on the surface level. The code looks fine for now. It would be nicer if this PR was smaller, but I'm not sure if there is a way to stage this change. Rebasing should at least help getting rid of meta diffs.

dmkozh · 2024-09-26T23:13:51Z

src/bucket/Bucket.h

+};
+
+template <class BucketT> class SearchableBucketListSnapshot;
+class LiveBucket : public Bucket,


Live/Hot buckets/bucket lists would benefit from at least top-level doc comments.

Added some comments and rebased.

SirTyson requested review from marta-lokhova and dmkozh July 30, 2024 18:38

SirTyson force-pushed the archival-bucketlist branch from 76b6594 to 05d6144 Compare August 13, 2024 04:46

SirTyson marked this pull request as ready for review August 13, 2024 04:46

SirTyson force-pushed the archival-bucketlist branch from 5cace22 to ca87d88 Compare August 13, 2024 18:35

dmkozh reviewed Aug 22, 2024

View reviewed changes

SirTyson force-pushed the archival-bucketlist branch 6 times, most recently from 6ca5887 to 344647f Compare September 19, 2024 17:45

dmkozh reviewed Sep 27, 2024

View reviewed changes

SirTyson added 12 commits October 1, 2024 10:36

Refactored BucketList to LiveBucketList

5dabdeb

Basic Hot Archive Structure

0fbd7cf

Templated Bucket helper classes

b3d1472

Template BucketList refactor

70ace7e

Added Hot Archive merge logic

84e39a9

BucketList tests for archive

e21e9b9

Added BucketListDB support for Hot Archive

fa9333b

Minor fixes

07e819d

Added Hot Archive BucketListDB tests and fixes

fb18c92

Fix BucketList test for versions < p22

f2aeef9

Fixed BucketMeta version bug

0cd7a3e

Minor fixes

827357c

SirTyson force-pushed the archival-bucketlist branch 2 times, most recently from 8cd6a2b to 2ee0adc Compare October 1, 2024 18:16

Cleaned up rebase

4e5753c

SirTyson force-pushed the archival-bucketlist branch from 2ee0adc to 4e5753c Compare October 2, 2024 00:36

SirTyson mentioned this pull request Oct 10, 2024

Remove SQL support for Ledger State #4504

Draft

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Archival bucketlist #4403

Archival bucketlist #4403

SirTyson commented Jul 30, 2024 •

edited

Loading

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

SirTyson Sep 17, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

SirTyson Sep 17, 2024

dmkozh Sep 26, 2024

SirTyson Sep 26, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

dmkozh Aug 22, 2024

marta-lokhova commented Aug 29, 2024

dmkozh left a comment

dmkozh Sep 26, 2024

SirTyson Oct 2, 2024

@@ @@ -33,7 +33,8 @@ enum class ProtocolVersion : uint32_t @@
                   V_18,
                   V_19,
                   V_20,
-                  V_21
+                  V_21,
+                  V_22

Archival bucketlist #4403

Are you sure you want to change the base?

Archival bucketlist #4403

Conversation

SirTyson commented Jul 30, 2024 • edited Loading

Description

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marta-lokhova commented Aug 29, 2024

dmkozh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirTyson commented Jul 30, 2024 •

edited

Loading