chore(sequencer)!: exclusively use Borsh encoding for stored data (ENG-768)

[Fraser999]

Summary

This PR primarily changes the encoding format of all data being written to storage to Borsh.

Background

We currently have a variety of different encoding formats, and this can be confusing, sub-optimal (in terms of storage space consumed and serialization/deserialization performance) and potentially problematic as e.g. JSON-encoding leaves a lot of flexibility in how the actual serialized data can look.

This PR is part one of three which aim to improve the performance and quality of the storage component. As such, the APIs of the various StateReadExt and StateWriteExt extension traits were updated slightly in preparation for the upcoming changes. In broad terms, for getters this meant having ref parameters rather than value ones (even for copyable types like [u8; 32] this is significantly more performant), and for "putters", parameters which are used for DB keys are refs, while the DB value parameters become values, since in the next PR these values will be added to a cache.

Changes

• Added a new storage module. This will ultimately contain our own equivalents of the cnidarium types, but for now consists only of a collection of submodules for types currently being written to storage. There is a top-level enum StoredValue which becomes the only type being written to storage by our own code.
• To accommodate for converting between the storage types and the corresponding domain types defined in astria-core and astria-merkle, some of these have been provided with constructors named unchecked_from_parts. This allows the type to be constructed from a trusted source like our own DB, skipping any validation steps which might otherwise be done.
• Updated all StateReadExt and StateWriteExt traits to use the new StoredValue, which internally uses Borsh encoding.
• Updated the APIs of all these extension traits as described above. This change resulted in a slew of similar updates to avoid needless copying [u8; 32] and [u8; 20] arrays, and that has unfortunately made the PR larger than expected.
• A few core types which currently derive or implement serde traits have had that removed, since it only existed to support JSON-encoding the type for storing. In one case it was for JSON-encoding to a debug log line, but I replaced that with a Display impl.

Testing

• Existing unit tests of the StateReadExt and StateWriteExt traits already had almost full coverage. Where coverage was missing, I added round trip tests.

Breaking Changelist

• The on-disk format of stored data has changed.

Related Issues

Closes #1434.

[SuperFluffy]

High level comment on the approach taken in this design:

At a high level, sequencer is defined by a set of components, each of which provides ways to change and read their respective state. So far this also meant that each component had full control over how the objects written to its state are converted to/from the domain types.

The storage and stored_value modules invert this logic by defining and passing in the on-disk types into the components.

I am ambivalent about this change since on the one hand it potentially allows avoiding code duplication. But on the other hand it breaks with the separation between the components we had so far.

[Fraser999]

So far this also meant that each component had full control over how the objects written to its state are converted to/from the domain types.

I see that as one of the main problems with the existing approach. It's because we've allowed this flexibility that different components use different serialization for the same domain types. In a lot of cases, there is only the domain type - the component would serialize directly from the domain type into bytes. I'm very keen to remove that flexibility so that all components have to use Borsh and we have a complete collection of storage types which map to/from domain types, both serving different purposes (domain types supporting business logic only, storage types supporting being held in a DB only).

The storage and stored_value modules invert this logic by defining and passing in the on-disk types into the components.

This is only temporary until #1436 is completed. When that work is completed, the components will be dealing with the domain types only.

But on the other hand it breaks with the separation between the components we had so far.

Well, I don't think there's complete separation currently. For example, accounts::StateReadExt is used in a few other components. But this isn't really changed with the new approach IMO. Ultimately the components will continue to access/modify state relating to other components via their state extension traits.

[SuperFluffy]

Reviewed the changes in astria-core for now (most of which I think we should not have in this PR). Going to review the rest of sequencer now.

[SuperFluffy]

I have went through about half the changes, but I think my comments apply universally.

1. We should provide domain types for all objects that have their own on-disk borsh types (for example, fee has storage::Fee, it feels natural to make it a domain::Fee, too). The on-disk types right now are stricter (which is good) than the domain types.
2. I am ambivalent about many on-disk types being pub(crate). I prefer if the on-disk types were module private and never crossed the boundary.
3. A lot of error instances lack context but have a comment above them noting that enough error context is provided. Because skipping context by just escaping with an ? operator is all too common among junior engineers, I'd prefer to require them everywhere rather than to come up with a heuristic as to when to use them and when to skip.

[SuperFluffy]

Comments:

1. the errors for failing to serialize from/to the on-disk type should use Debug (very rare, very bad if it happens: it's fine to be extra noisy).
2. on-disk types with a Display impl should should have a unit test so that their display impl matches that of the domain type (to avoid confusion) non-exhaustive list of types where this applies: RollupId, BlockHash.
3. we should provide an OndiskType trait that encapsulates the From/TryFrom impls we have everywhere right now.
4. we really really should split the tests into separate modules. This is not the first PR where I wish I could just ignore the tests when going through the code. :-/

[SuperFluffy]

Builder to the rescue. :)

[Fraser999]

A good constructor would be even better! One which took an iterator over RollupTransactions so we don't need to care how it's held internally :)

[SuperFluffy]

Thank you for the heroic task of going through all of sequencer and fixing its state write/read. I fat fingered my previous review but the comments were intended for this approval.

I don't see any blockers. From the previous list my main ask is to get remove all custom display impls in favor of just the derived Debug - we discussed this in the relevant comment chain, but as you noted the Display impls are intended for providing errors on failed state-decoding.

If that happens we are in so deep waters, that I think spewing a Debug log all over the screen is perfectly acceptable.

(side note that didn't make it into the list of the other comment:

1. We should extend snapshot tests to all keys - but that's likely covered in your open followup PR.

)