Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RPC Branch for Persistent Entry Eviction #4498

Draft
wants to merge 32 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
7afd1d9
Refactored BucketList to LiveBucketList
SirTyson Jul 21, 2024
d68e2e6
Basic Hot Archive Structure
SirTyson Jul 22, 2024
eec2317
Templated Bucket helper classes
SirTyson Jul 22, 2024
df03525
Template BucketList refactor
SirTyson Jul 25, 2024
616acbf
Added Hot Archive merge logic
SirTyson Jul 25, 2024
1822848
BucketList tests for archive
SirTyson Jul 26, 2024
7d942a8
Added BucketListDB support for Hot Archive
SirTyson Jul 29, 2024
c2b8264
Minor fixes
SirTyson Jul 31, 2024
85ba865
Added Hot Archive BucketListDB tests and fixes
SirTyson Aug 1, 2024
be8798c
Fix BucketList test for versions < p22
SirTyson Aug 13, 2024
8bf5225
Fixed BucketMeta version bug
SirTyson Aug 15, 2024
f10080e
Minor fixes
SirTyson Sep 11, 2024
051d454
Cleaned up rebase
SirTyson Sep 16, 2024
24640dd
Make background eviction mandatory
SirTyson Oct 7, 2024
7a9054a
Remove BucketList flag and cleanup non-ltx code
SirTyson Oct 8, 2024
f7afd74
In-memory mode commits/queries SQL offers table
SirTyson Oct 9, 2024
437cf20
Removed DEPRECATED_SQL_LEDGER_STATE and associated test modes
SirTyson Oct 9, 2024
2bf6c44
Database invariants now work with BucketListDB
SirTyson Oct 9, 2024
febe281
Made in-memory mode testing only
SirTyson Oct 10, 2024
2c65abb
Support upgrade-db for BucketListDB
SirTyson Oct 10, 2024
94b908b
Removed LedgerEntry SQL code
SirTyson Oct 10, 2024
9fab7e6
Add p23 test meta
SirTyson Sep 17, 2024
17bc05e
Added persistent entry eviction
SirTyson Sep 24, 2024
182e3be
Added Hot Archive support for tx apply
SirTyson Sep 24, 2024
c192334
Added basic creation proof framework
SirTyson Sep 26, 2024
2ed77e0
Added basic restoration proof framework
SirTyson Sep 27, 2024
be6b602
Fix some test pre p23
SirTyson Oct 2, 2024
1fc045a
Added deleted entries and live entries to Hot Archive during ledger c…
SirTyson Oct 3, 2024
f31c654
Add getledgerentry endpoint for state archival
SirTyson Oct 3, 2024
1dfeaf1
Added proof generation endpoints
SirTyson Oct 3, 2024
f09f6ea
added restore tx meta
SirTyson Oct 10, 2024
54a8be1
Better test coverage
SirTyson Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/integration.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ stellar-core generates several types of data that can be used by applications, d

Full [Ledger](ledger.md) snapshots are available in both:
* [history archives](history.md) (checkpoints, every 64 ledgers, updated every 5 minutes)
* in the case of captive-core (enabled via the `--in-memory` command line option) the ledger is maintained within the stellar-core process and ledger-state need to be tracked as it changes via "meta" updates.
* in the case of captive-core the ledger is maintained within the stellar-core process and ledger-state need to be tracked as it changes via "meta" updates.

## Ledger State transition information (transactions, etc)

Expand Down
17 changes: 3 additions & 14 deletions docs/quick-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,27 +147,16 @@ transactions or ledger states) must be downloaded and verified sequentially. It
worthwhile to save and reuse such a trusted reference file multiple times before regenerating it.

##### Experimental fast "meta data generation"
`catchup` has a command line flag `--in-memory` that when combined with the
`METADATA_OUTPUT_STREAM` allows a stellar-core instance to stream meta data instead
of using a database as intermediate store.
`catchup` when combined with the
`METADATA_OUTPUT_STREAM` allows a stellar-core instance to stream meta data.

This has been tested as being orders of magnitude faster for replaying large sections
of history.

If you don't specify any value for stream the command will just replay transactions
in memory and throw away all meta. This can be useful for performance testing the transaction processing subsystem.

The `--in-memory` flag is also supported by the `run` command, which can be used to
run a lightweight, stateless validator or watcher node, and this can be combined with
`METADATA_OUTPUT_STREAM` to stream network activity to another process.

By default, such a stateless node in `run` mode will catch up to the network starting from the
network's most recent checkpoint, but this behaviour can be further modified using two flags
(that must be used together) called `--start-at-ledger <N>` and `--start-at-hash <HEXHASH>`. These
cause the node to start with a fast in-memory catchup to ledger `N` with hash `HEXHASH`, and then
replay ledgers forward to the current state of the network.

A stateless and meta-streaming node can additionally be configured with
A meta-streaming node can additionally be configured with
`EXPERIMENTAL_PRECAUTION_DELAY_META=true` (if unspecified, the default is
`false`). If `EXPERIMENTAL_PRECAUTION_DELAY_META` is `true`, then the node will
delay emitting meta for a ledger `<N>` until the _next_ ledger, `<N+1>`, closes.
Expand Down
8 changes: 1 addition & 7 deletions docs/software/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,13 +159,7 @@ apply.
checkpoint from a history archive.
* **run**: Runs stellar-core service.<br>
Option **--wait-for-consensus** lets validators wait to hear from the network
before participating in consensus.<br>
(deprecated) Option **--in-memory** stores the current ledger in memory rather than a
database.<br>
(deprecated) Option **--start-at-ledger <N>** starts **--in-memory** mode with a catchup to
ledger **N** then replays to the current state of the network.<br>
(deprecated) Option **--start-at-hash <HASH>** provides a (mandatory) hash for the ledger
**N** specified by the **--start-at-ledger** option.
before participating in consensus.
* **sec-to-pub**: Reads a secret key on standard input and outputs the
corresponding public key. Both keys are in Stellar's standard
base-32 ASCII format.
Expand Down
20 changes: 1 addition & 19 deletions docs/stellar-core_example.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -229,14 +229,6 @@ FLOOD_DEMAND_BACKOFF_DELAY_MS = 500
# against each other.
MAX_DEX_TX_OPERATIONS_IN_TX_SET = 0

# DEPRECATED_SQL_LEDGER_STATE (bool) default false
# When set to true, SQL is used to store all ledger state instead of
# BucketListDB. This is not recommended and may cause performance degregradation.
# This is deprecated and will be removed in the future. Note that offers table
# is still maintained in SQL when this is set to false, but all other ledger
# state tables are dropped.
DEPRECATED_SQL_LEDGER_STATE = false

# BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT (Integer) default 14
# Determines page size used by BucketListDB for range indexes, where
# pageSize == 2^BUCKETLIST_DB_INDEX_PAGE_SIZE_EXPONENT. If set to
Expand All @@ -258,11 +250,6 @@ BUCKETLIST_DB_INDEX_CUTOFF = 20
# this value is ingnored and indexes are never persisted.
BUCKETLIST_DB_PERSIST_INDEX = true

# BACKGROUND_EVICTION_SCAN (bool) default true
# Determines whether eviction scans occur in the background thread. Requires
# that DEPRECATED_SQL_LEDGER_STATE is set to false.
BACKGROUND_EVICTION_SCAN = true

# EXPERIMENTAL_BACKGROUND_OVERLAY_PROCESSING (bool) default false
# Determines whether some of overlay processing occurs in the background
# thread.
Expand Down Expand Up @@ -601,17 +588,12 @@ MAX_SLOTS_TO_REMEMBER=12
# only a passive "watcher" node.
METADATA_OUTPUT_STREAM=""

# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true causes a stateless node
# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true causes a node
# which is streaming meta to delay streaming the meta for a given ledger until
# it closes the next ledger. This ensures that if a local bug had corrupted the
# given ledger, then the meta for the corrupted ledger will never be emitted, as
# the node will not be able to reach consensus with the network on the next
# ledger.
#
# Setting EXPERIMENTAL_PRECAUTION_DELAY_META to true in combination with a
# non-empty METADATA_OUTPUT_STREAM (which can be configured on the command line
# as well as in the config file) requires an in-memory database (specified by
# using --in-memory on the command line).
EXPERIMENTAL_PRECAUTION_DELAY_META=false

# Number of ledgers worth of transaction metadata to preserve on disk for
Expand Down
1 change: 0 additions & 1 deletion docs/stellar-core_example_validators.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
NETWORK_PASSPHRASE="Example configuration"

DATABASE="sqlite3://example.db"
DEPRECATED_SQL_LEDGER_STATE = false

NODE_SEED="SA7FGJMMUIHNE3ZPI2UO5I632A7O5FBAZTXFAIEVFA4DSSGLHXACLAIT a3"
NODE_HOME_DOMAIN="domainA"
Expand Down
1 change: 0 additions & 1 deletion docs/stellar-core_standalone.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ NODE_IS_VALIDATOR=true

#DATABASE="postgresql://dbname=stellar user=postgres password=password host=localhost"
DATABASE="sqlite3://stellar.db"
DEPRECATED_SQL_LEDGER_STATE = false

COMMANDS=["ll?level=debug"]

Expand Down
1 change: 0 additions & 1 deletion docs/stellar-core_testnet.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
NETWORK_PASSPHRASE="Test SDF Network ; September 2015"

DATABASE="sqlite3://stellar.db"
DEPRECATED_SQL_LEDGER_STATE = false

# Stellar Testnet validators
[[HOME_DOMAINS]]
Expand Down
1 change: 0 additions & 1 deletion docs/stellar-core_testnet_legacy.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ KNOWN_PEERS=[
"core-testnet3.stellar.org"]

DATABASE="sqlite3://stellar.db"
DEPRECATED_SQL_LEDGER_STATE = false
UNSAFE_QUORUM=true
FAILURE_SAFETY=1

Expand Down
1 change: 0 additions & 1 deletion docs/stellar-core_testnet_validator.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ PUBLIC_HTTP_PORT=false
NETWORK_PASSPHRASE="Test SDF Network ; September 2015"

DATABASE="sqlite3://stellar.db"
DEPRECATED_SQL_LEDGER_STATE = false

# Configuring the node as a validator
# note that this is an unsafe configuration in this particular setup:
Expand Down
Loading
Loading