[SMST] feat: Use compact SMST proofs

[red-0ne]

Summary

This PR changes the proofs being sent in MsgSubmitPRoof and stored by the SubmitProof keeper from SparseMerkleClosestProof to SparseCompactMerkleClosestProof.

Issue

Proofs one of the most block-space consuming primitives. The SMST support compact proofs that could help reduce their sizes.

• [Load Testing] Permissionless Demand Implementation Umbrella Ticket  #758

Type of change

Select one or more from the following:

• New feature, functionality or library
• Consensus breaking; add the consensus-breaking label if so. See [Infra] Automatically add the consensus-breaking label #791 for details
• Bug fix
• Code health or cleanup
• Documentation
• Other (specify)

Testing

• Unit Tests: make go_develop_and_test
• LocalNet E2E Tests: make test_e2e
• DevNet E2E Tests: Add the devnet-test-e2e label to the PR.

Sanity Checklist

• I have tested my changes using the available tooling
• I have commented my code
• I have performed a self-review of my own code; both comments & source code

[Olshansk]

Amazed that everyone just worked!

Any chance you can add a log or a unit test to show the difference in size?

I'm REALLy curious and can be done in a followup.

[red-0ne]

Any chance you can add a log or a unit test to show the difference in size?

@Olshansk, I'm really grateful that you called this out.

I added this test /pkg/relayer/session/sessiontree_test.go

Which shows that there is barely any improvement in the proof size 😢

I added a (gzip) compressed proofs to the test to have something to compare with.

Running for different leaf sizes, 1000 run each and accumulating size, this a sample result

numLeaf=10: compactionRatio: 0.990917, compressionRatio: 0.804937
numLeaf=100: compactionRatio: 0.989091, compressionRatio: 0.803551
numLeaf=1000: compactionRatio: 0.992390, compressionRatio: 0.805643
numLeaf=10000: compactionRatio: 0.989311, compressionRatio: 0.803930
numLeaf=100000: compactionRatio: 0.992038, compressionRatio: 0.805471
numLeaf=1000000: compactionRatio: 0.990228, compressionRatio: 0.804995

[Olshansk]

@red-0ne Thanks for putting in the extra test. This is fine and what I expected.

Want to use this as a learning lesson. Nothing we don't already know but something I want to reiterate:

1. Compact proof sounds sexy, eh? This is a classic example of "sounds sexy to an engineer" but not actually useful.
   • Probabilistic proof is what will actually ensure scalability
2. Understanding what's happening under the hood
   • We inherited "Compact Proofs" from the library we forked
   • I didn't implement it, neither did Harry, neither did you
   • None of us looked in the source code
   • I'm 100% sure we can make it more efficient, but again, it "sounds sexy", right?
3. The importance of visibility / observability
   • This doesn't always need to be a dashboard
   • It doesn't always need to be telemetry
   • It could be a log or a unit test
   • This is a great example of using t.Log
   • This raises the importance of a dashboard for probabilistic proofs

[github-actions]

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks.

You may need to run make trigger_ci to submit an empty commit that'll trigger the tests.

GCP workloads (requires changing the namespace to 823)
Grafana network dashboard for devnet-issue-823