simulate: resource population #6015

joe-p · 2024-06-05T22:59:02Z

Summary

When a user calls simulate with UnnamedResources enabled, simulate should suggest to the user how they can populate the resource arrays in their transactions to properly send the transaction group to the network.

Test Plan

Test ResourcePopulator works with simple local (not group sharing) resources
Test ResourcePopulator with group sharing
Test ResourcePopulator resource limit detection with group sharing (ie. it is able to find the correct transaction to put a resource in)
Test Simulate with ResourcePopulator functionality
Test /simulate endpoint with ResourcePopulator functionality
Write smaller tests for better ledger/simulation/resources.go coverage

codecov · 2024-06-06T11:17:49Z

Codecov Report

Attention: Patch coverage is 76.13636% with 63 lines in your changes missing coverage. Please review.

Project coverage is 55.97%. Comparing base (97ab559) to head (ec7a36a).
Report is 10 commits behind head on master.

❗ Current head ec7a36a differs from pull request most recent head 351f915

Please upload reports for the commit 351f915 to get more accurate results.

Files	Patch %	Lines
ledger/simulation/resources.go	76.58%	42 Missing and 17 partials ⚠️
ledger/simulation/simulator.go	66.66%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6015      +/-   ##
==========================================
+ Coverage   55.88%   55.97%   +0.09%     
==========================================
  Files         482      482              
  Lines       68571    68835     +264     
==========================================
+ Hits        38320    38531     +211     
- Misses      27646    27680      +34     
- Partials     2605     2624      +19

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joe-p · 2024-07-24T20:20:05Z

In 99abd2f I have added resource population to the simulate API but for some reason this struct is not being properly encoded in the response

type PopulatedResourceArrays struct {
	Accounts []basics.Address    `codec:"accounts"`
	Assets   []basics.AssetIndex `codec:"assets"`
	Apps     []basics.AppIndex   `codec:"apps"`
	Boxes    []logic.BoxRef      `codec:"boxes"`
}

Boxes is encoded properly, but Accounts, Assets, and Apps are coming up nil. This behavior can be seen when running TestSimulateWithUnnamedResources with the changes in this commit. This seems to be an issue with the handler or encoder because TestPopulateResources shows simulate itself returns references (in this case addresses) as expected.

@jasonpaulos Any idea where the data is getting lost? In general, what is the best way to debug these algod API tests?

jannotti · 2024-08-19T20:02:25Z

daemon/algod/api/server/v2/utils.go

@@ -576,6 +576,7 @@ func convertTxnGroupResult(txnGroupResult simulation.TxnGroupResult) PreEncodedS
 		AppBudgetAdded:           omitEmpty(txnGroupResult.AppBudgetAdded),
 		AppBudgetConsumed:        omitEmpty(txnGroupResult.AppBudgetConsumed),
 		UnnamedResourcesAccessed: convertUnnamedResourcesAccessed(txnGroupResult.UnnamedResourcesAccessed),
+		PopulatedResourceArrays:  txnGroupResult.PopulatedResourceArrays,


Probably needs omitEmpty?

daemon/algod/api/algod.oas2.json

daemon/algod/api/server/v2/handlers.go

daemon/algod/api/server/v2/utils.go

ledger/simulation/resources.go

jannotti · 2024-08-20T15:55:02Z

ledger/simulation/resources.go

+}
+
+func (r *txnResources) addApp(aid basics.AppIndex) {
+	r.apps[aid] = struct{}{}


I think you want to track that this adds availability of the app account too.

Suggested change

r.apps[aid] = struct{}{}

r.apps[aid] = struct{}{}

r.accounts[aid.Address()] = struct{}{}

jannotti · 2024-08-20T15:56:47Z

ledger/simulation/resources.go

+		}
+
+		for _, app := range txn.ForeignApps {
+			p.txnResources[groupIndex].staticApps[app] = struct{}{}


App account becomes accessible too.

Suggested change

p.txnResources[groupIndex].staticApps[app] = struct{}{}

p.txnResources[groupIndex].staticApps[app] = struct{}{}

p.txnResources[groupIndex].staticAccounts[app.Address()] = struct{}{}

jannotti · 2024-08-20T16:00:52Z

ledger/simulation/resources.go

+	return hasField || hasStatic || hasRef
+}
+
+func (r *txnResources) addAccount(addr basics.Address) {


I don't love the style of needing to call hasRoom() and then invoking add* unconditionally. Could the add* methods return error instead?
This would require you have two forms of some add* methods, or an extra argument that means "add this if you can do so by adding only one reference".

Let me know what you think of 2ea34f2. Rather than having an additional argument the call sites short circuit if it returns nil

ledger/simulation/resources.go

jasonpaulos

Finally submitting my review that's been in progress for a while. Note there's some overlap with what @jannotti has already posted

daemon/algod/api/algod.oas2.json

jasonpaulos · 2024-07-30T14:46:36Z

daemon/algod/api/algod.oas2.json

+        },
+        "populated-resource-arrays": {
+          "description": "Present if populate-resource-arrays is true in the request. In that case, it will be a map of transaction index in the group to populated resource arrays. There may be moure resource arrays given than transaction in the group, which means more app call transactions would be needed for extra resources.",
+          "type": "object"


Unless there is a very compelling reason, I would recommend against using a map with non-string keys in the response object. It makes JSON encoding much more difficult.

I'd instead suggest an array, or (my preference) adding this as a property to individual SimulateTransactionResult objects, with another array here for any extra txns.

ledger/simulation/resources.go

jasonpaulos · 2024-07-30T16:42:47Z

ledger/simulation/resources.go

+	staticAssets   map[basics.AssetIndex]struct{}
+	staticApps     map[basics.AppIndex]struct{}
+	staticAccounts map[basics.Address]struct{}
+	staticBoxes    []logic.BoxRef


FYI you can use logic.BoxRef as a map key

jasonpaulos · 2024-07-30T17:39:49Z

ledger/simulation/resources.go

+		accounts:           make(map[basics.Address]struct{}),
+		boxes:              []logic.BoxRef{},
+		maxTotalRefs:       consensusParams.MaxAppTotalTxnReferences,
+		maxAccounts:        consensusParams.MaxAppTxnAccounts,


Are you choosing to ignore the individual limits on assets, apps, and boxes because they happen to be the same as the total ref limit? I'd rather not have this code rely on that always being true

jasonpaulos · 2024-08-27T18:40:00Z

ledger/simulation/resources.go

+}
+
+// PopulatedResourceArrays is a struct that contains all the populated arrays for a txn
+type PopulatedResourceArrays struct {


To be consistent with the other structs in this package, I would prefer this struct not have codec tags and be directly exposed through the REST API. Instead a new model should be added to the oas2 spec and then daemon/algod/api/server/v2/utils.go can convert between this simulation.PopulatedResourceArrays and the generated model struct.

daemon/algod/api/server/v2/handlers.go

jasonpaulos · 2024-08-27T19:53:04Z

test/e2e-go/restAPI/simulate/simulateRestAPI_test.go

+			},
+		},
+	}
+	a.Equal(expectedResult, resp)


Re: the test failure that you're seeing. This is what the server sends back (decoded from the msgpack response):

"unnamed-resources-accessed": { "app-locals": [ { "account": "7WCFW7UO7NA3NTQWDPKMFT4NNTOU4A7M5IF5VGPB3OCH6TMLD4X6QX453M", "app": 1005 } ], "asset-holdings": [ { "account": "7WCFW7UO7NA3NTQWDPKMFT4NNTOU4A7M5IF5VGPB3OCH6TMLD4X6QX453M", "asset": 1002 } ], "boxes": [ { "app": 1006, "name:b64": "QQ==" } ], "extra-box-refs": 1 }

I don't think there's anything going wrong with the encoding or e2e aspect, but rather a logic error inside the simulate package when multiple different resource types are unspecified. I noticed that the test you're comparing against, TestPopulateResources, is much simpler and only tests accounts. I suggest replicating this in a unit test and debugging further there. Generally speaking the most complex test cases and corner cases should be unit tests in simulation_eval_test.go, then all the e2e test needs to do is make sure all inputs get picked up properly and all outputs are present as expected in the response (to catch encoding errors or fields being dropped).

Co-authored-by: Jason Paulos <[email protected]>

Co-authored-by: Jason Paulos <[email protected]> Co-authored-by: John Jannotti <[email protected]>

joe-p · 2024-10-09T13:36:22Z

In 292a9b9 I updated the API model to avoid using map[int] but for some reason it's not encoding the response properly: msgpack decode error [pos 50]: no matching struct field found when decoding stream map with key PopulatedResourceArrays.

If I print the raw response I see this:

��last-round��txn-groups��PopulatedResourceArrays��app-budget-added��app-budget-consumed��failed-at��failure-message��transaction SSG3ROSUBRSMXPTZYOORYAXCDYURLGM5OJJF5J3LMJ74LFEWJJAA: logic eval error: invalid Account reference CV6S42NRDBJZKQDDQPUVXSD4KUJ5FSYHXENR74ZPTTIEQASD2WBRBFODRY. Details: app=1006, pc=57, opcodes=store 2; load 0; balance�txn-results��app-budget-consumed��txn-result��pool-error��txn��sig�@۔sq��3b�l3��^�J��b�=�29�IұH=� �P��I��r��-e��A��}��q��?<��txn��apid��fee��fv��gh� ��Gp��y8�d��"��\>-c�P�2�P��݆~ݢlv��snd� ��ʏe�Lz�Y��W�[�+!��O��P�Фtype�appl�version

So for some reason PopulatedResourceArrays is present where I would expect it to be omitted (and if it was included I would expect it to be populated-resource-arrays) given the fact that it's defined as

// PreEncodedSimulateTxnResult mirrors model.SimulateTransactionResult
type PreEncodedSimulateTxnResult struct {
	Txn                      PreEncodedTxInfo                        `codec:"txn-result"`
	AppBudgetConsumed        *uint64                                 `codec:"app-budget-consumed,omitempty"`
	LogicSigBudgetConsumed   *uint64                                 `codec:"logic-sig-budget-consumed,omitempty"`
	TransactionTrace         *model.SimulationTransactionExecTrace   `codec:"exec-trace,omitempty"`
	UnnamedResourcesAccessed *model.SimulateUnnamedResourcesAccessed `codec:"unnamed-resources-accessed,omitempty"`
	FixedSigner              *string                                 `codec:"fixed-signer,omitempty"`
	PopulatedResourceArrays  *model.ResourceArrays                   `codec:"populated-resource-arrays,omitempty"`
}

Any ideas on what might be happening here?

jannotti · 2024-10-09T14:29:01Z

It almost seems like the codec line was completely ignored for encoding, since it has the default name and omitempty was ineffective. Yet, decoding was surprised to see the capitalized form. I don't know the context of your testing - is there any chance you encoded that bytestream before the codec line was added, then decoded it after?

kylebeee · 2024-10-09T16:22:56Z

At a glance it seems to me like you might have a bug somewhere where you're assigning *simulation.PopulatedResourceArrays to PreEncodedSimulateTxnResult.PopulatedResourceArrays instead of *model.ResourceArray but its not super clear where that would be happening & i dont know the go-algorand code base well enough to say definitively.

Where are you printing the raw response?

joe-p changed the title ~~Feat/populate_resources~~ resource population Jun 5, 2024

joe-p mentioned this pull request Jun 5, 2024

ResourcePopulator joe-p/go-algorand#2

Closed

ResourcePopulator

5ba0a9a

joe-p force-pushed the feat/populate_resources branch from 466fd50 to 5ba0a9a Compare June 5, 2024 23:06

joe-p changed the title ~~resource population~~ simulate: resource population Jun 5, 2024

TestPopulatorWithGlobalResources (arrays only)

028b8c4

joe-p added 14 commits June 6, 2024 07:30

test addBox

941ba6e

use ElementsMatch

a69091c

addHolding

d9e77ef

test variable renaming

4667647

appLocals

cbd1d8a

restore default limits

d79780f

fix rekey field

e6d59ba

populate with static properties and remove zeroAddr

54cd38c

empty boxes

673b03d

ensure duplicates are removed

ceddaa2

overflow txn resources

62213a0

use ConsensusParams

f616449

fix empty box count

6a4f1ee

golangci-lint

0524c3c

algorandskiy added Enhancement external contribution labels Jun 7, 2024

algorandskiy requested review from jasonpaulos and jannotti June 7, 2024 17:13

joe-p added 5 commits June 14, 2024 16:31

Merge branch 'master' into feat/populate_resources

73e5b4c

PopulateResourceArrays in simulate (untested)

6d8c161

populate from ResourceTracker

022c565

initial TestPopulateResources

93dd4e9

group sharing and no group sharing TestPopulateResources

3f78752

joe-p added 5 commits June 19, 2024 07:58

test with non appl in group

359b5b0

modify visibility on resource pop stuff

2229b0a

remove duplicate partitiontest.PartitionTest(t)

ec7a36a

don't make RekeyTo address available

351f915

add PopulateResourceArrays to simulate API (WIP)

99abd2f

jannotti reviewed Aug 20, 2024

View reviewed changes

jasonpaulos reviewed Aug 27, 2024

View reviewed changes

jasonpaulos mentioned this pull request Aug 28, 2024

Simulate: Pack transaction reference arrays for unnamed resources #5684

Closed

gmalouf assigned joe-p Sep 19, 2024

joe-p and others added 9 commits October 2, 2024 17:30

Apply suggestions from code review

96f1af1

Co-authored-by: Jason Paulos <[email protected]>

Apply suggestions from code review

02534ae

Co-authored-by: Jason Paulos <[email protected]> Co-authored-by: John Jannotti <[email protected]>

PopulateResourceArrays -> PopulateResources

0815e29

static -> prefilled

b4f7f54

replace ifs with switch

e665817

hasAccount short circuit logic

c9e6e5d

check for room and return error in add... methods

2ea34f2

mixed resources test

93e7b81

use arrays in txn and group result rather than map[int] for API (WIP)

292a9b9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simulate: resource population #6015

simulate: resource population #6015

joe-p commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

joe-p commented Jul 24, 2024

jannotti Aug 19, 2024

jannotti Aug 20, 2024

jannotti Aug 20, 2024

jannotti Aug 20, 2024

joe-p Oct 2, 2024

jasonpaulos left a comment

jasonpaulos Jul 30, 2024

jasonpaulos Jul 30, 2024

jasonpaulos Jul 30, 2024

jasonpaulos Aug 27, 2024

jasonpaulos Aug 27, 2024

joe-p commented Oct 9, 2024 •

edited

Loading

jannotti commented Oct 9, 2024

kylebeee commented Oct 9, 2024

	r.apps[aid] = struct{}{}
	r.apps[aid] = struct{}{}
	r.accounts[aid.Address()] = struct{}{}

	p.txnResources[groupIndex].staticApps[app] = struct{}{}
	p.txnResources[groupIndex].staticApps[app] = struct{}{}
	p.txnResources[groupIndex].staticAccounts[app.Address()] = struct{}{}

simulate: resource population #6015

Are you sure you want to change the base?

simulate: resource population #6015

Conversation

joe-p commented Jun 5, 2024 • edited Loading

Summary

Test Plan

codecov bot commented Jun 6, 2024 • edited Loading

Codecov Report

joe-p commented Jul 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jasonpaulos left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joe-p commented Oct 9, 2024 • edited Loading

jannotti commented Oct 9, 2024

kylebeee commented Oct 9, 2024

joe-p commented Jun 5, 2024 •

edited

Loading

codecov bot commented Jun 6, 2024 •

edited

Loading

joe-p commented Oct 9, 2024 •

edited

Loading