Feat/rpc endpoints to fetch data from key

[hugocaillard]

Description

Add two new endpoints:

• /v2/clarity_marf_value/:clarity_marf_key
• /v2/clarity_metadata/:principal/:contract_name/:clarity_metadata_key

I'm currently working on a Clarinet feature that allows to simulate running (or, said differently, to fork the mainnet state in the simnet data store). This requires the ability to fetch values from marf and metadata keys.

These new endpoints are similar to already existing endpoint (such as getdatavar, getmapentry, getcontractsrc, etc).

Applicable issues

N/A

Additional info (benefits, drawbacks, caveats)

Read more about this feature in the Clarinet issue: hirosystems/clarinet#1503

I confirmed that the 2 new endpoints allow to achieve the desired goal by running a local devnet with these changes and forking the Clarinet simnet state from it.

Checklist

• Test coverage for new or modified code paths
• Changelog is updated
• Required documentation changes (e.g., docs/rpc/openapi.yaml and rpc-endpoints.md for v2 endpoints, event-dispatcher.md for new events)

[jcnelson]

I strongly disagree with using the MARF key as input. To do so would require an explanation to an RPC API consumer of how to construct a Clarity MARF key, which essentially requires them to read the majority of SIP-005. In addition, this would effectively require the RPC endpoint to determine whether or not the key is well-formed, since passing an ill-formed key would require a different HTTP error code than HTTP 404 (which is not something I think you want to waste your time doing).

Instead, you should do the following:

• Accept the hash of the key as input. Then, you don't have to worry about verifying that it is well-formed, and you don't have to worry about explaining to the RPC API consumer how to construct it. Furthermore, it's much safer and easier to determine if a string is a well-formed hash instead of a well-formed Clarity MARF key.
• Update the Clarity DB to take the hash as an argument, instead of the key (in support of the above).

[jcnelson]

Also, I dislike the path you chose, since it doesn't leave room for semantic expansion. Instead, can you use the following: /v2/clarity/marf/{key-hash}, where key-hash is the hash of the SIP-005 key.

[hugocaillard]

In my use case (of Clarinet consuming this API), I already know the key (but not the hash). But I agree that it's not ideal to have the key as a use input.
Can you guide me to some code where I could see how to get the key <> key_hash conversion?

Update the Clarity DB to take the hash as an argument, instead of the key (in support of the above).

Not sure to follow here, where should I perform this change?

---

I think I just lack knowledge about this repo to fully address this comment, thanks for providing extra context 🙏

[jcnelson]

Sure -- the (string) key's bytes are hashed via SHA512/256 to produce the MARF key.

Not sure to follow here, where should I perform this change?

The trait ClarityBackingStore needs to be modified to add a method like get_data_with_proof(), but which takes an implementation of the MarfTrieId trait instead of a &str key. There currently isn't a way to query the MARF via the ClarityDB with a bare hash.

[jcnelson]

Clarity metadata keys have a well-defined structure, so the RPC endpoint should return HTTP 400 if the metadata key is ill-formed. You will need to expand this method (or try_parse_request -- up to you) to validate the structure of the Clarity metadata key, including determining whether or not the key's StoreType and var_name values are supported values. var_name will, unfortunately, take some effort because the Clarity DB codebase uses bare string literals in its calls to ClarityDatabase::make_metadata_key() instead of enums, so your PR should address this as well.

[hugocaillard]

Apparently the metadata var_name can be an arbitrary string, where here it's a variable_name, a map_name, a token_name).

Can we really use an enum here?

[jcnelson]

Yes; however, some reserved var_name strings can only be paired with certain StoreTypes. For example, the use of StoreType::Contract would require var_name to be contract-size, contract-src, contract-data-size, or contract, but nothing else.

One way to implement these constraints could be to implement an enum to capture all of the valid StoreType / var-name pairings, and permit only the StoreType variants which allow arbitrary var_name values to have arbitrary var_name values.