Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make merkle tree leaf addition operations atomic #71

Merged
merged 7 commits into from
Sep 23, 2024
Merged

Conversation

jimthematrix
Copy link
Contributor

@jimthematrix jimthematrix commented Sep 17, 2024

fixes #21

Copy link
Contributor

@EnriqueL8 EnriqueL8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A view on the batch TX logic

// the node's reference key (not the index) is used as the key to
// store the node in the DB
n := core.SMTNode{
RefKey: ref.Hex(),
}
err := s.p.DB().Table(s.nodesTableName).First(&n).Error
err := batchOrDb.Table(nodesTableName).First(&n).Error
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm struggling why a read operation would need a batch transaction

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here we want the query to hit the records that have been added in the batch, but have not been committed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah gotcha

Copy link
Contributor

@EnriqueL8 EnriqueL8 Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but if you are looking relying on getNode before inserting then you need to make sure to use the DB directly instead of batch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel batchOrDb should be dbTx here. this layer shouldn't care about that difference.

go-sdk/internal/sparse-merkle-tree/smt/merkletree.go Outdated Show resolved Hide resolved
Comment on lines 281 to 284
if _, err := batch.GetNode(k); err == nil {
return nil, ErrNodeIndexAlreadyExists
}
err := mt.db.InsertNode(n)
err := batch.InsertNode(n)
Copy link
Contributor

@EnriqueL8 EnriqueL8 Sep 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is where I don't understand the batch function where you get a node but the TX is not committed yet and then you are adding an insert to the same transaction

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for example here, you would use db instead of batch for GetNode?

Copy link
Contributor

@EnriqueL8 EnriqueL8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarifications on the batching in GROM , makes sense to me

Copy link
Contributor

@EnriqueL8 EnriqueL8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a conversation with @jimthematrix , we can move to ON CONFLICT insert and if there is on conflict do nothing and rework the way the read works

@jimthematrix
Copy link
Contributor Author

@EnriqueL8 thanks for catching that. just added a commit to make the inserts `ON CONFLICT DO NOTHING" with a justification in the inline comments

Copy link
Contributor

@Chengxuan Chengxuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jimthematrix core logic looks good to me

I think we should address the following before the merge:

  1. test temp db file cleanup check
  2. log rollback failures

I also made a suggestion for readability, which is to consider renaming batch to highlight it's a DB tx.

}

func (s *SqliteTestSuite) TearDownTest() {
os.Remove(s.dbfile.Name())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

error check is regressed, this will cause removal failure due to the wrong file name not being captured.

Comment on lines 83 to 107
batch, err := mt.db.BeginBatch()
if err != nil {
return err
}
newRootKey, err := mt.addLeaf(batch, node, mt.rootKey, 0, path)
if err != nil {
log.L().Errorf("Error adding leaf node %s: %v, rolling back", node.Ref().Hex(), err)
_ = batch.Rollback()
return err
}
mt.rootKey = newRootKey

// update the root node index in the storage
err = mt.db.UpsertRootNodeIndex(mt.rootKey)
log.L().Infof("Upserting root node index to %s", mt.rootKey.Hex())
err = batch.UpsertRootNodeIndex(mt.rootKey)
if err != nil {
log.L().Errorf("Error upserting root node %s: %v, rolling back", mt.rootKey.Hex(), err)
_ = batch.Rollback()
return err
}
log.L().Infof("Committing batch operations for adding leaf node %s", node.Ref().Hex())
err = batch.Commit()
if err != nil {
log.L().Errorf("Error committing batch operations for adding leaf node %s: %v", node.Ref().Hex(), err)
_ = batch.Rollback()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the core logic implements Make merkle tree leaf addition operations atomic. Some readability suggestions. Here are the steps as per my understanding:

  1. A single DB tx is created to ensure atomicity of multiple DB txs
  2. Add a new leaf node as part of the started DB TX
  3. Upsert MT root in the same DB TX
  4. Commit only if both queries succeeded, otherwise rollback.

So I think it would improve the readability if batch is renamed to reflect it's a DB tx.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we should log Rollback errors as warnings so leakage of DB txs can be monitored.

go-sdk/integration-test/smt_test.go Outdated Show resolved Hide resolved
}

func (s *SqliteTestSuite) TearDownTest() {
os.Remove(s.dbfile.Name())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
os.Remove(s.dbfile.Name())
if err := os.Remove(s.dbfile.Name()); err != nil {
panic(err);
}

}

func (s *MerkleTreeTestSuite) TearDownTest() {
os.Remove(s.dbfile.Name())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
os.Remove(s.dbfile.Name())
if err := os.Remove(s.dbfile.Name()); err != nil {
panic(err);
}

jimthematrix and others added 3 commits September 22, 2024 21:09
Co-authored-by: Chengxuan Xing <[email protected]>
Signed-off-by: jimthematrix <[email protected]>
Signed-off-by: Jim Zhang <[email protected]>
@jimthematrix
Copy link
Contributor Author

@Chengxuan the remaining comments should be addressed. I think panic'ing on file remove failure in the test temp folder during test cleanup feels too heavy handed. but a test failure seems reasonable

@Chengxuan Chengxuan dismissed EnriqueL8’s stale review September 23, 2024 04:32

All comments addressed

@Chengxuan Chengxuan merged commit e110455 into main Sep 23, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sparse merkle tree implementation - hardening
3 participants