incentives: cache top online accounts and use when building AbsentParticipationAccounts #6085

cce · 2024-07-26T17:24:51Z

Summary

In #5757 a mechanism was introduced to suspend "absentee" accounts that don't participate (by making a proposal, or heartbeat as in #5799), by adding a block header AbsentParticipationAccounts, similar to ExpiredParticipationAccounts.

Currently, the list is generated by considering any account touched by a transaction in the current block, since this data is readily available at endOfBlock(). This PR adds a periodically-updated cache of top online accounts to the ledger, to find additional online accounts not mentioned in the current block.

All of these tracked addresses will now be checked for absentee or expired status each round. To get a recent list of top online accounts, this PR uses recent work done by the votersTracker and state proof worker. (Every 256 rounds, the state proof system performs a TopOnlineAccounts query.) This adds access to the votersTracker to fetch the most recent list of top online addresses, and for each address looks up the latest round's data from the online account cache.

LastProposed and LastHeartbeat are added to the online accounts table's DB representation in this PR. This also fixes an issue introduced in #5965 where uses of ledgercore.OnlineAccountData (which didn't have LastHeartbeat/LastProposed fields) were replaced by basics.OnlineAccountData (which did) and ended up with those fields not being set in a couple of conversions from AccountData.

Test Plan

update TestAbsenteeChecks
update TestExpiredAccountGeneration
maybe update TestExpiredAccountGenerationWithDiskFailure?
update TestAbsentTracking
added new TestLatestCompletedVotersUpTo
update TestAbsenteeChallenges
update TestEvalFunctionForExpiredAccounts
disable Payouts for state proof E2E test TestTotalWeightChanges (TODO return later after auto-heartbeating added, to ensure the accounts aren't suspended)
~~update test/e2e-go/features/incentives/suspension_test.go~~ (TODO return later after heartbeats)

…ding AbsentParticipationAccounts

ledger/ledgercore/onlineacct.go

algorandskiy

LGTM.

Just a thought - why not to init top online on startup and then maintain the list in acctonline while processing incoming blocks?

cce · 2024-07-27T00:43:55Z

Just a thought - why not to init top online on startup and then maintain the list in acctonline while processing incoming blocks?

My first approach was to make it a field in the onlineAccounts tracker like the ao.voters sub-tracker, but as I kept cutting it down to MVP and removed its access to the onlineAccounts.deltas, and took it out of being called in onlineAccounts.newBlockImpl(), I moved it out since it had no dependencies left. It felt like I kept adding duplicate state that was already being maintained in onlineAccounts, and duplicate logic for looking it up. So I wanted to try just the absolute minimal approach to start, by relying totally on the onlineAccounts tracker's caching system (which we already know works correctly) rather than add a new cached list of online accounts and delta-processing code..

cmd/tealdbg/localLedger.go

ledger/eval/eval.go

ledger/eval/prefetcher/prefetcher_alignment_test.go

ledger/toponline.go

ledger/eval/eval.go

codecov · 2024-08-09T17:23:41Z

Codecov Report

Attention: Patch coverage is 77.57009% with 24 lines in your changes missing coverage. Please review.

Project coverage is 56.24%. Comparing base (619d257) to head (c558d59).
Report is 16 commits behind head on master.

Files with missing lines	Patch %	Lines
ledger/eval/eval.go	87.27%	4 Missing and 3 partials ⚠️
ledger/ledger.go	75.00%	2 Missing and 2 partials ⚠️
ledger/ledgercore/votersForRound.go	0.00%	4 Missing ⚠️
ledger/tracker.go	0.00%	3 Missing ⚠️
cmd/tealdbg/localLedger.go	0.00%	2 Missing ⚠️
daemon/algod/api/server/v2/dryrun.go	0.00%	2 Missing ⚠️
ledger/ledgercore/accountdata.go	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6085      +/-   ##
==========================================
+ Coverage   56.22%   56.24%   +0.02%     
==========================================
  Files         494      494              
  Lines       69954    70040      +86     
==========================================
+ Hits        39330    39394      +64     
- Misses      27947    27962      +15     
- Partials     2677     2684       +7

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…break TestAbsenteeChecks

…andidates

ledger/eval/eval.go

ledger/ledger.go

Co-authored-by: John Jannotti <[email protected]>

ledger/ledger.go

ledger/eval/eval.go

…andidates

…ineAccountsList

gmalouf · 2024-10-08T19:07:33Z

ledger/acctonline_expired_test.go

@@ -458,7 +458,7 @@ func TestOnlineAcctModelSimple(t *testing.T) {
 	})
 	// test same scenario on double ledger
 	t.Run("DoubleLedger", func(t *testing.T) {
-		m := newDoubleLedgerAcctModel(t, protocol.ConsensusFuture, true)
+		m := newDoubleLedgerAcctModel(t, protocol.ConsensusV39, true) // TODO simulate heartbeats


Why not keep this on future?

It fails because heartbeats aren't implemented, but proposers aren't being set, so the big accounts are challenged and kicked offline, and all the stake numbers don't match the test expectations. I could have tried to fix this by ensuring all the test accounts show up as proposers as often as necessary to avoid suspension, but I thought maybe it would be better to see after heartbeats were implemented whether that would make the tests pass without as much modification.

gmalouf · 2024-10-08T19:11:48Z

ledger/eval/cow.go

@@ -47,6 +47,7 @@ type roundCowParent interface {
 	// lookup retrieves agreement data about an address, querying the ledger if necessary.
 	lookupAgreement(basics.Address) (basics.OnlineAccountData, error)
 	onlineStake() (basics.MicroAlgos, error)
+	knockOfflineCandidates() (map[basics.Address]basics.OnlineAccountData, error)


Maybe a NIT: should we actually call this top online accounts or similar naming? It's very clear from comments that's what we are requesting, more a debate over if the name should be based on what it's sourced from vs the use-case we have for this atm.

It is a potentially stale list of top online accounts, if new accounts appeared online in the last 256 rounds (since the last state proof) they wouldn't appear. So the word "candidates" was intended to make it seem a little less definitive that this was the complete list of top online accounts for the round... but happy to pick any other name, I wasn't particularly happy with this name.

This is already being used in a method JJ called "generateKnockOfflineAccountsList" in #5757 which is where the "knockOffline" part came from.

ledger/eval/eval.go

gmalouf · 2024-10-08T19:18:53Z

test/e2e-go/features/stateproofs/stateproofs_test.go

@@ -810,6 +810,9 @@ func TestTotalWeightChanges(t *testing.T) {
 	a := require.New(fixtures.SynchronizedTest(t))

 	consensusParams := getDefaultStateProofConsensusParams()
+	consensusParams.Payouts = config.ProposerPayoutRules{} // TODO re-enable payouts when nodes aren't suspended


How are we tracking these?

I guess I should make an issue to address the "update this test once heartbeats are implemented" TODOs in this PR

jannotti

I think we will not have the top online cache before the first state proof, right? Maybe it would make sense to seed it during genesis (since the onlince accounts are listed out for us in the genesis file, I think). That could avoid special cases in the tests.

ledger/eval/eval.go

jannotti · 2024-10-09T18:00:41Z

ledger/eval/eval.go

-func (eval *BlockEvaluator) endOfBlock() error {
+// When generating a block, participating addresses are passed to prevent a
+// proposer from suspending itself.
+func (eval *BlockEvaluator) endOfBlock(participating ...basics.Address) error {


Why ...basics.Address instead of []basics.Address? I assume callers always have a slice, as opposed to call sites with, say, 5 explicit arguments.

it's true, this is just me optimizing for a smaller diff, to not change other endOfBlock callers, but the idea is to pass a slice — can change

jannotti · 2024-10-09T18:04:01Z

ledger/eval/eval.go

+		IncentiveEligible     bool // currently unused below, but may be needed in the future
+	}
+	candidates := make(map[basics.Address]candidateData)
+	partAddrs := util.MakeSet(participating...)


Do we do anything else with this slice? Maybe we should push the Set type up through the callers, so that it is built as a Set when it is first created to pass to endOfBlock?

It's used in GenerateBlock while making a map of end-of-block account state for participating addresses, to include in the UnfinishedBlock ... if we pushed it up to GenerateBlock then it could protect against looking up the same participating address twice, if duplicate addresses were passed to GenerateBlock.

ledger/eval/eval.go

gmalouf · 2024-10-15T18:27:42Z

ledger/eval/eval.go

+	if maxSuspensions > 0 {
+		knockOfflineCandidates, err := eval.state.knockOfflineCandidates()
+		if err != nil {
+			// Log an error and keep going; generating lists of absent and expired


So this implies some nodes can "choose" not to search for absent/expired accounts.

gmalouf · 2024-10-15T18:29:21Z

ledger/eval/eval.go

+
+	// Now, check these candidate accounts to see if they are expired or absent.
+	for accountAddr, acctData := range candidates {
+		if acctData.MicroAlgosWithRewards.IsZero() {


100% of time, zero balance implies being closed?

gmalouf · 2024-10-15T18:30:42Z

ledger/eval/eval.go

+//
+// This function is passed a list of participating addresses so a node will not
+// propose a block that suspends or expires itself.
+func (eval *BlockEvaluator) generateKnockOfflineAccountsList(participating []basics.Address) {


participating is really "participating accounts excluding any I host"

gmalouf

I'm good in general, a few small comments.

gmalouf · 2024-10-15T18:33:29Z

ledger/eval/eval_test.go

 		blkEval = l.nextBlock(t)
+		//require.Empty(t, vb.Block().ExpiredParticipationAccounts)


why is this added commented out?

gmalouf · 2024-10-15T18:35:23Z

ledger/eval/eval_test.go

 	challenge := byte(0)
-	for i := uint64(0); i < uint64(1210); i++ { // A bit past one grace period (200) past challenge at 1000.
-		vb := l.endBlock(t, blkEval)
+	for i := uint64(0); i < uint64(1200); i++ { // Just before first suspension at 1171


Would this not go past first suspension - why 1200?

gmalouf · 2024-10-15T18:36:45Z

ledger/eval/eval_test.go

-	}
-
-	st := txn.Sign(keys[0])
-	err = eval.Transaction(st, transactions.ApplyData{})


Why remove all of these eval.Transaction calls?

gmalouf · 2024-10-15T18:41:51Z

ledger/ledger.go

+	}
+
+	// fetch fresh data up to this round from online account cache. These accounts should all
+	// be in cache, as long as proto.StateProofTopVoters < onlineAccountsCacheMaxSize.


This feels like a condition to call out in the consensus file.

cce requested review from jannotti and algorandskiy July 26, 2024 17:25

algorandskiy added the New Feature label Jul 26, 2024

periodically track top N online accounts in Ledger, and use when buil…

21db44d

…ding AbsentParticipationAccounts

cce force-pushed the track-incentive-candidates branch from 975ddb4 to 21db44d Compare July 26, 2024 17:35

cce commented Jul 26, 2024

View reviewed changes

ledger/ledgercore/onlineacct.go Outdated Show resolved Hide resolved

run make msgp

e968515

algorandskiy reviewed Jul 26, 2024

View reviewed changes

jannotti reviewed Jul 31, 2024

View reviewed changes

cmd/tealdbg/localLedger.go Outdated Show resolved Hide resolved

ledger/eval/eval.go Outdated Show resolved Hide resolved

ledger/eval/prefetcher/prefetcher_alignment_test.go Outdated Show resolved Hide resolved

ledger/toponline.go Outdated Show resolved Hide resolved

cce added 2 commits August 9, 2024 10:15

update votersTracker

8587b28

CR fixes

38d4b8d

cce commented Aug 9, 2024

View reviewed changes

ledger/eval/eval.go Show resolved Hide resolved

cce added 2 commits August 9, 2024 10:58

rename and simplify

9740ddc

add LastProposed/LastHeartbeat to BaseOnlineAccountData handling

b8b9673

cce mentioned this pull request Aug 9, 2024

refactor: remove ledgercore.OnlineAccountData #5965

Merged

cce added 2 commits August 9, 2024 11:43

fill out LastHeartbeat/LastProposed from algorand#5965

0b7fbad

fix remaining tests

b2f1130

cce added 2 commits August 9, 2024 14:40

fix missing AccountData => OnlineAccountData transformation function …

5242990

…break TestAbsenteeChecks

Merge remote-tracking branch 'upstream/master' into track-incentive-c…

0baf81f

…andidates

jannotti reviewed Aug 20, 2024

View reviewed changes

ledger/eval/eval.go Show resolved Hide resolved

ledger/eval/eval.go Show resolved Hide resolved

ledger/ledger.go Outdated Show resolved Hide resolved

Update ledger/ledger.go

be464cf

Co-authored-by: John Jannotti <[email protected]>

cce requested a review from algorandskiy August 21, 2024 15:22

cce commented Aug 21, 2024

View reviewed changes

ledger/ledger.go Outdated Show resolved Hide resolved

Update ledger/ledger.go

97d0bcf

algorandskiy reviewed Aug 21, 2024

View reviewed changes

ledger/eval/eval.go Show resolved Hide resolved

cce added 2 commits August 28, 2024 09:16

Merge remote-tracking branch 'upstream/master' into track-incentive-c…

55d5068

…andidates

update TestAbsenteeChecks

01b150a

cce force-pushed the track-incentive-candidates branch from f5b42d4 to 01b150a Compare September 18, 2024 15:26

cce added 3 commits September 18, 2024 13:37

Merge remote-tracking branch 'upstream/master' into track-incentive-c…

0f954d1

…andidates

also consider candidates for ExpiredParticipationAccounts

c24e809

update TestExpiredAccountGeneration

c252d95

gmalouf assigned cce Sep 18, 2024

cce added 6 commits September 19, 2024 13:37

Fix TestAbsentTracking

bb82a97

add TestLatestCompletedVotersUpTo

04c0a84

don't propose to suspend yourself, and update TestAbsenteeChallenges

c41a49a

Update TestExpiredAccountGeneration

e476730

update TestAbsenteeChecks for proposer change

9115aae

update TestTotalWeightChanges and update comment on generateKnockOffl…

23b9996

…ineAccountsList

cce marked this pull request as ready for review October 4, 2024 17:31

cce added 2 commits October 4, 2024 14:07

update TestEvalFunctionForExpiredAccounts

9bc39aa

update TestOnlineActModel*

c558d59

cce force-pushed the track-incentive-candidates branch from 1c4c898 to c558d59 Compare October 4, 2024 18:40

gmalouf requested review from jannotti and algorandskiy October 4, 2024 19:18

gmalouf reviewed Oct 8, 2024

View reviewed changes

jannotti reviewed Oct 9, 2024

View reviewed changes

gmalouf reviewed Oct 15, 2024

View reviewed changes

gmalouf approved these changes Oct 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incentives: cache top online accounts and use when building AbsentParticipationAccounts #6085

incentives: cache top online accounts and use when building AbsentParticipationAccounts #6085

cce commented Jul 26, 2024 •

edited

Loading

algorandskiy left a comment •

edited

Loading

cce commented Jul 27, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

gmalouf Oct 8, 2024

cce Oct 10, 2024 •

edited

Loading

gmalouf Oct 8, 2024

cce Oct 10, 2024 •

edited

Loading

gmalouf Oct 8, 2024

cce Oct 10, 2024

jannotti left a comment

jannotti Oct 9, 2024

cce Oct 9, 2024

jannotti Oct 9, 2024

cce Oct 9, 2024

gmalouf Oct 15, 2024

gmalouf Oct 15, 2024

gmalouf Oct 15, 2024

gmalouf left a comment

gmalouf Oct 15, 2024

gmalouf Oct 15, 2024

gmalouf Oct 15, 2024

gmalouf Oct 15, 2024

		blkEval = l.nextBlock(t)
		//require.Empty(t, vb.Block().ExpiredParticipationAccounts)

incentives: cache top online accounts and use when building AbsentParticipationAccounts #6085

Are you sure you want to change the base?

incentives: cache top online accounts and use when building AbsentParticipationAccounts #6085

Conversation

cce commented Jul 26, 2024 • edited Loading

Summary

Test Plan

algorandskiy left a comment • edited Loading

Choose a reason for hiding this comment

cce commented Jul 27, 2024 • edited Loading

codecov bot commented Aug 9, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

cce Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cce Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jannotti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmalouf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cce commented Jul 26, 2024 •

edited

Loading

algorandskiy left a comment •

edited

Loading

cce commented Jul 27, 2024 •

edited

Loading

codecov bot commented Aug 9, 2024 •

edited

Loading

cce Oct 10, 2024 •

edited

Loading

cce Oct 10, 2024 •

edited

Loading