From 1ca5aa5daf38b1f5fe56fbe959d942f349193be6 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Mon, 18 Mar 2024 21:26:35 -0700 Subject: [PATCH 01/18] feat: revamp store & upload specs --- w3-store.md | 959 ++++++++++++++++++++++++++++++++-------------------- 1 file changed, 599 insertions(+), 360 deletions(-) diff --git a/w3-store.md b/w3-store.md index 1ed6822..e21b961 100644 --- a/w3-store.md +++ b/w3-store.md @@ -1,631 +1,870 @@ -# Storage and upload protocol +# W3 Storage Protocol ![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) ## Editors - [Yusef Napora](https://github.com/yusefnapora), [DAG House](https://dag.house) +- [Irakli Gozalishvili](https://github.com/gozala) ## Authors +- [Irakli Gozalishvili](https://github.com/gozala) - [Yusef Napora](https://github.com/yusefnapora), [DAG House](https://dag.house) ## Abstract -This spec describes the data storage protocol used by web3.storage's "w3up" platform. +In the W3 protocol user owned (name)space represent a data storage primitive that can be managed using W3 storage protocol defined here. Storage protocol allows space owners to manage state across compatible storage provider services using defined set of [UCAN] capabilities. Use of [UCAN] authorization system enables space access to be shared by delegating corresponding capabilities to desired audience. -The protocol is modeled as a set of capabilities which can be fulfilled by a [provider](./w3-provider.md) via UCAN invocations. In the web3.storage implementation, the providers are implemented using [ucanto](https://github.com/web3-storage/ucanto), a type-safe UCAN RPC framework. +## Language -- [Background](#background) - - [Spaces](#spaces) -- [Capabilities](#capabilities) - - [`store/` namespace](#store-namespace) - - [`store/*`](#store) - - [`store/add`](#storeadd) - - [`store/get`](#storeget) - - [`store/remove`](#storeremove) - - [`store/list`](#storelist) - - [`upload/` namespace](#upload-namespace) - - [`upload/*`](#upload) - - [`upload/add`](#uploadadd) - - [`upload/get`](#uploadget) - - [`upload/remove`](#uploadremove) - - [`upload/list`](#uploadlist) +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). -## Background +# Introduction -The w3up protocol accepts data in [CAR](https://ipld.io/specs/transport/car/) format, and clients are expected to encode data into CARs before invoking storage capabilities. +The base storage layer of the space is [Content Address]ed [Content ARchive][CAR] files. In practice this means that user wishing to store files or directories of files needs produce an [IPLD] [DAG] in [UnixFS] format and then encoded into one or multiple [Content ARchive]s that can be stored using W3 storage protocol. -The current w3up clients encode files and directories using [UnixFS](https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs) and the [DAG-PB codec](https://ipld.io/specs/codecs/dag-pb/spec/), however, the storage protocol makes no assumptions about the format of the DAG (or DAGs) contained within a CAR, and any valid IPLD codec may be used. +> Large DAGs get "sharded" across multiple [Content ARchive]s and stored individually to meet size limits of the storage provider. -Large DAGs may be "sharded" across multiple CAR files to fit within storage and transport size limits. In this case, the [`store/add`](#storeadd) invocation for each of the shards (apart from the first) will include a CID link to a previous shard, so that the shards can be grouped into a single logical unit. +Separately `upload/` protocol can be utilized allowing user to create standalone entities (files, directories) representing entry points to the DAGs contained by the [Content ARchive]s. E.g. when storing a file or a directory [UnixFS] root [CID] and archives are captured using `upload/add` capability allowing viewer to effectively fetch and re-assemble it. -Because a given CAR can contain many DAGs, and each DAG may itself contain sub-DAGs that can be viewed as standalone entities (e.g. files in a nested directory), the protocol provides separate capabilities for CAR storage and identifying the root of "interesting" DAGs within the CAR. The [`store/` namespace](#store-namespace) defines the capabilities related to CAR storage, while the [`upload/` namespace](#upload-namespace) defines the capabilities for linking the root CID of a DAG to the CAR (or CARs) that contain it. +## Concepts -### Spaces +### Roles -The `store` and `upload` capabilities operate on "spaces," which are unique namespaces identified by `did:key` URIs. The `with` resource field of all the capabilities described in this spec MUST be a valid `did:key` URI that identifies a space. +There are several distinct roles that [principal]s may assume in this specification: -The private key for a space is generated locally by each space's owner and is never shared with the service provider. +| Name | Description | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| Principal | The general class of entities that interact with a UCAN. Identified by a DID that can be used in the `iss` or `aud` field of a UCAN. | +| Agent | A [Principal] identified by [`did:key`] identifier, representing a user in an application. | +| Issuer | A [principal] delegating capabilities to another [principal]. It is the signer of the [UCAN]. Specified in the `iss` field of a UCAN. | +| Audience | Principal access is shared with. Specified in the `aud` field of a UCAN. | -The private key for a space is able to issue invocations for the `store` and `upload` capabilities, however we do not recommend issuing invocations directly using the space private key. Instead, the space key should be used to issue a long-lived delegation to an "agent," allowing the agent to invoke capabilities related to the space. If desired, the private key for the space can be saved in "cold storage" and used to issue further delegations. -## Capabilities +### Space -This section describes the capabilities that form the storage protocol, along with details relevant to invoking capabilities with a service provider. +A namespace, often referred as a "space", is an owned resource that can be shared. It corresponds to a unique asymmetric cryptographic keypair and is identified by a [`did:key`] URI. The `store/` and `upload/` capabilities can be used to manage content stored in the given space at given storage provider. -When invoking any capability listed in this spec, the agent MUST include proof that they have been delegated the capability for the space identified in the `with` URI. For example, if an agent is trying to invoke [`store/add`](#storeadd), they must include proof that they were delegated `store/add`, or a capability from which `store/add` can be derived, e.g. [`store/*`](#store). +## Content Archive -In the special case where the private key corresponding to the space itself is the issuer of the invocation (as opposed to invocations issued by an agent), the proof may be omitted, as the space key is considered the "owner" of the space. In all other cases, the invocation UCAN must include a delegation that includes the requested capability for the space that the invocation is acting upon. +[Content Archive][CAR] ofter referred as [CAR] is a primary primitive for storing shards of the content in the space. -## `store/` namespace +# Capabilities -The `store/` namespace contains capabilities relating to storage of CAR files. +## Store Capabilities -### `store/*` +Capabilities under `store/*` namespace can be used to manage [CAR]s that provider is storing and serving on behalf of the [space]. -> Delegate all capabilities in the `store/` namespace +### Store Capabilities IPLD Schema -The `store/*` capability is the "top" capability of the `store/*` namespace. `store/*` can be delegated to a user agent, but cannot be invoked directly. Instead, it allows the agent to derive any capability in the `store/` namespace, provided the resource URI matches the one in the `store/*` capability delegation. +```ipldsch +type StoreCapability union { + StoreAddCapability "store/add" + StoreGetCapability "store/get" + StoreListCapability "store/list" + StoreRemoveCapability "store/remove" +} representation inline { + discriminantKey "can" +} +``` + +### Storage Provider + +The audience of the invocation (`aud` field) MUST be the [provider] DID of the storage provider implementing this protocol. + +### Storage Space + +The subject of the invocation (`with` field) MUST be the DID of the target MUST be target [space]. -In other words, if an agent has a delegation for `store/*` for a given space URI, they can invoke any capability in the `store/` namespace using that space as the resource. +### Content Archive Identifier -### `store/add` +The `nb.link` field of the invocation MUST be an [IPLD Link] to the desired [CAR]. Link MUST have Content Addressable aRchive (CAR) `0x0202` codec code. It is RECOMMENDED to support SHA2-256 multihash code `0x12`. Implementers are MAY choose to support other additional hashing algorithms. -> Request storage of a CAR file +### Store Add -The `store/add` capability allows an agent to store a CAR file into the space identified by the `did:key` URI in the `with` field. The agent must encode the CAR locally and provide the CAR's CID and size using the `nb.link` and `nb.size` fields, allowing a service to provision a write location for the agent to submit the CAR. +Authorized agent MAY invoke `store/add` capability on the [space] subject (`with` field) to request that provider (`aud` field) store and serve [CAR] on their behalf. -#### Derivations +Invoking `store/add` capability for a [CAR] that is already added to space SHOULD be a noop. -`store/add` can be derived from a [`store/*`](#store) or [`*`][ucan-spec-top] capability with a matching `with` field. +#### Store Add Capability -#### Caveats +##### Store Add IPLD Schema -When invoking a `store/add` capability, the `link` caveat MUST be set to the CID of the CAR being stored. +```ipldsch +type StoreAddCapability struct { + with SpaceDID + nb StoreAdd +} + +type StoreAdd struct { + link &ContentArchive + size Int + origin optional &ContentArchive +} -Each `store/add` invocation applies to a single CAR. When storing large DAGs that are "sharded" across multiple CARs, the `origin` caveat is used to link the shards together in a sequence, similar to a linked list. The first shard to be uploaded will have no `origin` field defined, while each subsequent shard will include an `origin` that links to the previous one. +type ContentArchive = bytes +type SpaceDID = DID +type DID = string +``` + +##### Store Add Content -When preparing sharded CARs, note that each shard must be a valid CAR file with a CAR header; simply chunking a large CAR file into pieces is not sufficient. Instead, split your DAG along block boundaries and create new CAR shards when adding additional blocks would put the total size of the current CAR over the size limit (see below). +Capability invocation MUST specify [Content archive Identifier]. -Support for sharded CARs implies that the service provider must also support CARs containing "partial DAGs," meaning DAGs that contain links to blocks that are not present in the CAR file. +##### Store Add Content Size -The `size` caveat sets a limit on the size (in bytes) of the stored CAR. Agents should check their delegation's `nb.size` field and ensure that they only send CARs with a size below the limit. Larger DAGs may be sharded across multiple CARs as described above. +Capability invocation MUST set `nb.size` field to the byte size of the [content archive][CAR]. -Regardless of whether `nb.size` is set in the delegation, the agent must include an `nb.size` field in their invocation, with a value that is equal to the size in bytes of the CAR to be stored. If a limit has been set in the delegation, the size must be less than or equal to the limit. -| `field` | `value` | `required?` | `context` | -| ----------- | --------------------------------- | ----------- | ---------------------------------------------------------- | -| `link` | CAR CID string, e.g. `bag123...` | โœ” | CID of CAR that the user wants to store. | -| `size` | size in bytes | โœ” | The size of the CAR to be uploaded in bytes. | -| `origin` | CAR CID string, e.g. `bagabc...` | | Optional link to related CARs. See below for more details. | +##### Store Add Origin -#### Invocation +> Status: Deprecated ๐Ÿ›‘ +> +> Field was intended for establish causal order, but proved impractical. -To invoke `store/add`, an agent constructs a UCAN with the shape described below. +Capability invocation MAY set _optional_ `nb.origin` field to a causally related [CAR] like a previous shard of the content DAG. -Example: +##### Store Add Example ```js { - can: "store/add", - with: "did:key:abc...", - nb: { - link: "bag...", - size: 1234 + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "store/add", + with: "did:key:zAl..1ce", + nb: { + link: { "/": "bag...7ldq" }, + size: 42_600_000 + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -The `nb.origin` field may be set to provide a link to a related CAR file. This is useful when storing large DAGs that are sharded across multiple CAR files. In this case, the agent can link each uploaded shard with a previous one. Providing the `origin` field informs the service that the CAR being stored is a shard of the larger DAG, as opposed to an intentionally partial DAG. +#### Store Add Receipt -Note that the `nb.link` and `nb.origin` caveats contain the CID of the CAR itself, not the CID of any content contained within the CAR. It should have the [multicodec code value `0x0202`](https://github.com/multiformats/multicodec/blob/master/table.csv#L135), which is reserved for data in CAR format. When encoded to a CID string using the default encoding, this results in a CID with a prefix of `bagb`. +Provider MUST issue signed receipt containing `StoreAddResult`. -#### Responses +##### Store Add Result -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +###### Store Add Result IPLD Schema -Executing a `store/add` invocation with a service provider should return a result object. +```ipldsch +type StoreAddResult union { + StoreAddSuccess "ok" + StoreAddFailure "error" +} representation keyed -If the invocation fails, the result will include an `error` field with a value of `true`, and a `message` field containing additional details. +type StoreAddSuccess union { + StoreAddDone "done" + StoreAddPending "upload" +} representation inline { + discriminantKey "status" +} -On success, the response will include a `status` field, which may have one of the following values: +type StoreAddDone struct { + with SpaceDID + link &ContentArchive +} + +type StoreAddPending struct { + with SpaceDID + link &ContentArchive + url URL + headers HTTPHeaders +} -- `done`: indicates that this CAR has already been stored, and no further action is needed -- `upload`: indicates that the client must upload the CAR to the provided URL. +type URL = string +type HTTPHeaders {String: String} + +type StoreAddFailure struct { + message String +} +``` -If `status == 'upload'`, the response will include additional fields containing information about the request, including the URL and headers needed to upload: +###### Store Add Done -| `field` | `type` | `description` | -| --------- | ------------------------ | --------------------------------------------------------------- | -| `url` | `string` | A URL that will accept a `PUT` request containing the CAR data. | -| `headers` | `Record` | HTTP headers that must be attached to the `PUT` request. | -| `with` | `string` | The space resource URI used in the invocation. | -| `link` | `string` | The CAR CID specified in the invocation's `link` field. | -| `allocated` | `number` | Total bytes allocated in the space to accommodate the stored item. May be zero if the item is *already* stored in the space. | +Capability provider MUST succeed request with a `"done"` status if provider is able to store requested [CAR] on user behalf right-away. In other words provider already has addressed [CAR] file. -The client should then make an HTTP `PUT` request to the `url` specified in the response, attaching all the included `headers`. The body of the request MUST be CAR data, whose size exactly equals the size specified in the `store/add` invocation's `size` caveat. Additionally, the CID of the uploaded CAR must match the invocation's `link` caveat. In other words, attempting to upload any data other than that authorized by the `store/add` invocation will fail. +###### Store Add Upload -#### Implementations +Capability provider MUST succeed request with a `"upload"` status if it is able to allocate memory for the requested [CAR]. It MUST set `url` field to an HTTP PUT endpoint where addressed [CAR] can be uploaded. -- @web3-storage/capabilities [`store/add` capability validator](https://github.com/web3-storage/w3up/blob/main/packages/capabilities/src/store.js#L30) -- @web3-storage/upload-api [`store/add` method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/store/add.js#L12) +Provider MUST set `headers` field to the set of HTTP headers that uploading agent MUST set on an HTTP PUT request. -### `store/get` +HTTP PUT endpoint set in `url` field MUST verify that uploaded bytes do correspond to the addressed [CAR] and specified `size`. -> Get metadata about a CAR shard from a space +###### Store Add Failure -The `store/get` capability can be invoked to get metadata about a CAR from a [space](#spaces). +Capability provider SHOULD fail invocation if the subject [space] has not been provisioned with the provider. -This may be used to check for inclusion, or to get the `size` and `origin` properties for a shard. +Capability provider SHOULD fail invocation if the subject [space] has not been provisioned with enough storage capacity to store requested archive. -#### Derivations +### Store Get -`store/get` can be derived from a [`store/*`](#store) or [`*`][ucan-spec-top] capability with a matching `with` field. +Authorized agent MAY invoke `store/get` capability on the [space] subject (`with` field) to query a state of the of the specified [CAR] (`nb.link` field) of the replica held by the provider (`aud` field). -#### Caveats +#### Store Get Capability -When invoking `store/get`, the `link` caveat must be set to the CID of the CAR file to get metadata for. +##### Store Get IPLD Schema + +```ipldsch +type StoreGetCapability struct { + with SpaceDID + nb StoreGet +} + +type StoreGet struct { + link &ContentArchive +} +``` -If a delegation contains a `link` caveat, an invocation derived from it must have the same CAR CID in its `link` field. A delegation without a `link` caveat may be invoked with any `link` value. +##### Store Get Content -| `field` | `value` | `required?` | `context` | -| --------- | --------------------------------- | ----------- | --------------------------------------------- | -| `link` | CAR CID string, e.g. `bag...` | | The CID of the CAR to get metadata for | +Capability invocation MUST specify [Content archive Identifier]. -#### Invocation +##### Store Get Example ```js { - can: "store/get", - with: "did:key:abc...", - nb: { - link: "bag...", + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "store/get", + with: "did:key:zAl..1ce", + nb: { + link: { "/": "bag...7ldq" }, + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses +#### Store Get Receipt -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +Capability provider MUST issue signed receipt containing `StoreGetResult`. -Executing a `store/get` invocation with a service provider should return a response object. +##### Store Get Result -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +###### Store Get Result IPLD Schema -On success, the response object will have the following shape: - -```ts -interface StoreListItem { - /** CID of the stored CAR. */ - link: string +```ipldsch +type StoreGetResult struct { + StoreGetSuccess "ok" + StoreGetFailure "error" +} representation keyed +``` - /** Size in bytes of the stored CAR */ - size: number +###### Store Get Success - /** Link to a related CAR, used for sharded uploads */ - origin?: string, +Capability provider MUST issue `StoreGetSuccess` result for every content archive that has been added to the space. - /** ISO-8601 timestamp when CAR was added to the space */ - insertedAt: string, +```ipldsch +type StoreGetSuccess { + link &ContentArchive + size Int + origin optional &ContentArchive } ``` -#### Implementations - -- @web3-storage/capabilities [store/get validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/store.js#L90) -- @web3-storage/upload-api [store/get method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/store/get.js) +###### Store Get Failure -### `store/remove` +Capability provider MUST issue `StoreGetFailure` result for every content archive that either has not been added or was since removed at the time of the invocation. -> Remove a stored CAR from a space - -The `store/remove` capability can be invoked to remove a CAR file from a [space](#spaces). +```ipldsch +type StoreGetFailure { + message string +} +``` -This may or may not cause the CAR to be removed completely from the underlying storage system; for example, if the CAR exists in other spaces, it will not be removed. +### Store Remove -`store/remove` will remove the CAR from the listing provided by [`store/list`](#storelist) for the space. Removal may also have billing implications, depending on the service provider (e.g. by affecting storage quotas). +Authorized agent MAY invoke `store/remove` capability to remove content archive from the subject space (`with` field). -#### Derivations +#### Store Remove Capability -`store/remove` can be derived from a [`store/*`](#store) or [`*`][ucan-spec-top] capability with a matching `with` field. +##### Store Remove IPLD Schema -#### Caveats +```ipldsch +type StoreRemoveCapability struct { + with SpaceDID + nb StoreRemove +} -When invoking `store/remove`, the `link` caveat must be set to the CID of the CAR file to remove. +type StoreRemove struct { + link &ContentArchive +} +``` -If a delegation contains a `link` caveat, an invocation derived from it must have the same CAR CID in its `link` field. A delegation without a `link` caveat may be invoked with any `link` value. +##### Store Remove Content -| `field` | `value` | `required?` | `context` | -| --------- | --------------------------------- | ----------- | --------------------------------------------- | -| `link` | CAR CID string, e.g. `bag...` | โœ” | The CID of the CAR file to remove. | +Capability invocation MUST specify desired [Content archive Identifier]. -#### Invocation +##### Store Remove Example ```js { - can: "store/remove", - with: "did:key:abc...", - nb: { - link: "bag...", + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "store/remove", + with: "did:key:zAl..1ce", + nb: { + link: "bag...7ldq", + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +#### Store Remove Receipt -Executing a `store/remove` invocation with a service provider should return a response object. +Capability provider MUST issue signed receipt containing `StoreRemoveResult`. -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +##### Store Remove Result -On success, the response object will be empty. +###### Store Remove Result IPLD Schema -#### Implementations - -- @web3-storage/capabilities [store/remove validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/store.js#L106) -- @web3-storage/upload-api [store/remove method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/store/remove.js#L11) +```ipldsch +type StoreRemoveResult struct { + StoreRemoveSuccess "ok" + StoreRemoveFailure "error" +} representation keyed +``` -### `store/list` +###### Store Remove Success -> Obtain a list of stored CARs +Capability provider MUST issue `StoreRemoveSuccess` after it has removed the archive from the space. -The `store/list` capability can be invoked to request a list of CARs in a given memory space. +Capability provider MUST set `size` field to the number of bytes that were freed from space. -The `with` field of the invocation must be set to the DID of the memory space to be listed. +Capability provider MUST issue `StoreRemoveSuccess` even when specified archive is not in space. In that case it MUST set `size` to `0`. -#### Derivations +```ipldsch +type StoreRemoveSuccess { + size Int +} +``` -`store/list` can be derived from a [`store/*`](#store) or [`*`][ucan-spec-top] capability with a matching `with` field. +### Store List -#### Caveats +Authorized agent MAY invoke `store/list` capability on the [space] subject (`with` field) to list [CAR]s added to it at the time of invocation. -When invoking `store/list` the `size` caveat may be set to the desired number of results to return per invocation. If there are more total results than will fit into the given `size`, the response will include an opaque `cursor` field that can be used to continue the listing in a subsequent invocation by setting the `cursor` caveat to the value in the response. The `pre` caveat may be set to return the page of results preceding the cursor. +#### Store List Capability -| `field` | `value` | `required?` | `context` | -| -------- | --------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | -| `size` | `number` | | The desired number of results to return. | -| `cursor` | `string` | | An opaque string included in a prior `store/list` response that allows the service to provide the next "page" of results. | -| `pre` | `boolean` | | If true, return the page of results preceding the cursor. | +##### Store List IPLD Schema -#### Invocation +```ipldsch +type StoreListCapability struct { + with SpaceDID + nb StoreList +} -```js -{ - can: "store/list", - with: "did:key:abc..", - nb: { - size: 40, - cursor: 'cursor-value-from-previous-invocation', - } +type StoreList struct { + cursor optional &string + size optional int + pre optional bool } ``` -#### Responses +##### Store List Cursor -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +The optional `nb.cursor` MAY be specified in order to paginate over the list of the added [CAR]s. -Executing a `store/list` invocation with a service provider should return a response object. +##### Store List Size -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +The optional `nb.size` MAY be specified to signal desired page size, that is number of items in the result. -On success, the response object will have the following shape: +##### Store List Pre -```ts -interface StoreListResponse { - /** Cursor that can be used in a subsequent store/list invocation to fetch the next page of results */ - cursor?: string +The optional `nb.pre` field MAY be set to `true` to request a page of results preceding cursor. If `nb.pre` is omitted or set to `false` provider MUST respond with a page following the specified `nb.cursor`. - /** Number of results in this page of listings. */ - size: number +##### Store List Example - /** Items in this page of listings. */ - results: StoreListItem[] +```js +{ + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "store/list", + with: "did:key:zAl..1ce", + nb: { + size: 40, + cursor: 'cursor-value-from-previous-invocation', + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } + } } ``` -The `results` field contains an array of `StoreListItem` objects: +#### Store List Receipt -```ts -interface StoreListItem { - /** CID of the stored CAR. */ - link: string +Capability provider MUST issue signed receipt containing `StoreListResult`. - /** Size in bytes of the stored CAR */ - size: number +##### Store List Result - /** Link to a related CAR, used for sharded uploads */ - origin?: string, +###### Store List Result IPLD Schema - /** ISO-8601 timestamp when CAR was added to the space */ - insertedAt: string, -} +```ipldsch +type StoreListResult union { + StoreListSuccess "ok" + StoreListFailure "error" +} representation keyed ``` -#### Implementations +###### Store List Success -- @web3-storage/capabilities [store/list validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/store.js#L126) -- @web3-storage/upload-api [store/list method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/store/list.js#L10) +Capability provider MUST issue `StoreListSuccess` result containing page of entries added to the space. -## `upload/` namespace - -The `upload/` namespace contains capabilities relating to "uploads", which represent user data that is contained in one or more CAR files that have previously been stored using [`store/add`](#storeadd). +```ipldsch +type StoreListSuccess struct { + cursor optional string + before optional string + after optional string + size int + results [StoreListItem] +} -An upload is essentially an index that maps "data CIDs" to CAR CIDs. Data CIDs are the root CID of user-uploaded data items, for example, files that have been encoded into UnixFS. A given data item may be stored in a single CAR, or it may be split into multiple CAR "shards," as described in the [`store/add` section](#storeadd). +type StoreListItem struct { + link &ContentArchive + size int + origin optional &ContentArchive +} +``` -Similarly, a CAR can potentially contain many data items. This is true even if the CAR has only a single root CID. For example, when storing a CAR containing a nested directory structure, you could create one "upload" for the root of the directory structure, and a separate upload for a file nested inside. +###### Store List Failure -### `upload/*` +```ipldsch +type StoreListFailure struct { + message string +} +``` -> Delegate all abilities in the `upload/*` namespace. +## Upload Capabilities -The `upload/*` capability can be delegated to a user agent, but cannot be invoked directly. Instead, it allows the audience to derive any capability in the `upload/` namespace, provided the resource URI matches the one in the `upload/*` capability delegation. +Capabilities under `upload/*` namespace can be used to manage list of top level content entries. While not required, it is generally assumed that user content like file will be turned into [UnixFS] DAG packed and stored in space as one or more [content archive][CAR]s. The root of the DAG is then added to the upload list. -The `upload/*` capability (and all capabilities in the `upload/` namespace) can be derived from a [`*`][ucan-spec-top] capability with a matching resource URI. +### Upload Capabilities IPLD Schema -### `upload/add` +```ipldsch +type UploadCapability union { + UploadAddCapability "upload/add" + UploadGetCapability "upload/get" + UploadListCapability "upload/list" + UploadRemoveCapability "upload/remove" +} representation inline { + discriminantKey "can" +} +``` -> Add an upload to a space. +### Upload Add -`upload/add` can be invoked to register a given DAG as being contained in a given set of CARs. The resulting "upload" will be associated with the memory space identified by the DID in the `with` field. +Authorized agent MAY invoke `upload/add` capability on the [space] subject (`with` field) to request that provider (`aud` field) include content identified by `nb.root` in the list of content entries for the space. -An `upload/add` invocation requires the root CID of the DAG to register as an "upload," along with the CID of one or more CARs that contain the complete DAG. It is expected that these CARs have previously been stored in the space using [`store/add`](#storeadd); a service provider MAY return an error response if this is not the case. +It is expected that CARs containing content are stored in the space using [`store/add`] capability. Provider MAY enforce this invariant by failing invocation or choose to succeed invocation but fail to serve the content when requested. -#### Derivations +> โš ๏ธ Behavior of calling `upload/add` with a same `root` and different `shards` is not specified by the protocol. w3up reference implementation allows such invocations and updates `shards` to union of all shards across invocations. -`upload/add` can be derived from an [`upload/*`](#upload) or [`*`][ucan-spec-top] capability with a matching `with` resource URI. +#### Upload Add Capability -#### Caveats +##### Upload Add IPLD Schema -When invoking `upload/add`, the `root` caveat must be set to the root CID of the data item. +```ipldsch +type UploadAddCapability struct { + with SpaceDID + nb Upload +} -The `shards` array must contain at least one CID of a CAR file, which is expected to have been previously stored. +type Upload struct { + root &any + shards [&ContentArchive] +} +``` -Taken together, the CARs in the `shards` array should contain all the blocks in the DAG identified by the `root` CID. +###### Upload Root -| `field` | `value` | `required?` | `context` | -| ----------- | -------------------------------------------------------- | ----------- | ---------------------------------------------------------------- | -| `root` | data CID string, e.g. `bafy...` | โœ” | The CID of the data item that was uploaded. | -| `shards` | array of CID strings, e.g. `[ "bag123...", "bag234..."]` | โœ” | The CIDs of CAR files containing the full DAG for the data item. | +The `nb.root` field MUST be set to the [IPLD Link] of the desired content entry. -#### Invocation +###### Upload Shards -To invoke `upload/add`, an agent constructs a UCAN with the shape described below. +The `nb.shards` field MUST be set to the list of [IPLD Link]s for the [Content Archive][CAR]s containing the upload entry. -Example: +###### Upload Add Example ```js { - can: "upload/add", - with: "did:key:abc...", - nb: { - root: "bafyabc...", - shards: ["bagb1...", "bagb2..."] + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "upload/add", + with: "did:key:zAl..1ce", + nb: { + root: { "/": "bafy...k3ve" }, + shards: [{ "/": "bag...7ldq" }] + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses +#### Upload Add Receipt -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +Capability provider MUST issue signed receipt containing `UploadAddResult`. -Executing an `upload/add` invocation with a service provider should return a result object. +##### Upload Add Result -If the invocation fails, the result will include an `error` field with a value of `true`, and a `message` field containing additional details. +###### Upload Add Result IPLD Schema -On success, the response object will have the following shape: +```ipldsch +type UploadAddResult union { + UploadAddSuccess "ok" + UploadAddFailure "error" +} representation keyed -```ts -interface UploadAddResponse { - root: string - shards?: string[] +type UploadAddSuccess struct { + root &any + shards [&ContentArchive] } -``` - -#### Implementations - -- @web3-storage/capabilities [upload/add validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/upload.js#L54) -- @web3-storage/upload-api [upload/add method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/upload/add.js#L12) -### `upload/get` - -> Get metadata about a upload from a space - -The `upload/get` capability can be invoked to get metadata about an upload from a [space](#spaces). +type UploadAddFailure struct { + message string +} +``` -This may be used to check for inclusion, or to get the set of CAR shards associated with a root CID. +### Upload Get -The `with` resource URI must be set to the DID of the space to get the upload metadata from. +Authorized agent MAY invoke `upload/get` capability on the [space] subject (`with` field) to query a state of the of the specified upload entry. -#### Derivations +#### Upload Get Capability -`upload/get` can be derived from an [`upload/*`](#upload) or [`*`][ucan-spec-top] capability with a matching `with` resource URI. +##### Upload Get IPLD Schema -#### Caveats +```ipldsch +type UploadGetCapability struct { + with SpaceDID + nb UploadGet +} -The `root` caveat must contain the root CID of the item to get metadata for. +type UploadGet struct { + root &any +} +``` -| `field` | `value` | `required?` | `context` | -| --------- | --------------------------------- | ----------- | ------------------------------------------------------------- | -| `root` | data CID string, e.g. `bafy...` | | The root CID of the upload to get metadata for | +##### Upload Get Root -#### Invocation +The `nb.root` field MUST be set to the [IPLD Link] of the desired content entry. -To invoke `upload/get`, an agent prepares a UCAN with the following shape: +##### Upload Get Example ```js { - can: "upload/get", - with: "did:key:abc...", - nb: { - root: "bafyabc..." + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "upload/get", + with: "did:key:zAl..1ce", + nb: { + root: { "/": "bafy...ro0t" }, + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses - -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +#### Upload Get Receipt -Executing an `upload/get` invocation with a service provider should return a response object. +Capability provider MUST issue signed receipt containing `UploadGetResult`. -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +##### Upload Get Result -On success, the response object will have the following shape: +###### Upload Get Result IPLD Schema -```ts -interface UploadListItem { - /** Root CID of the uploaded data item. */ - root: string +```ipldsch +type UploadGetResult struct { + UploadGetSuccess "ok" + UploadGetFailure "error" +} representation keyed +``` - /** CIDs of CARs that contain the complete DAG for the uploaded data item. */ - shards: string[] +###### Upload Get Success - /** ISO-8601 timestamp when the upload was added to the space. */ - insertedAt: string, +Capability provider MUST issue `UploadGetSuccess` result for the upload entry that has been added to the space. - /** ISO-8601 timestamp when the upload entry was last modified. */ - updatedAt: string, +```ipldsch +type UploadGetSuccess { + link &any + shards [&ContentArchive] } ``` -#### Implementations +###### Upload Get Failure -- @web3-storage/capabilities [upload/list validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/upload.js#L142) -- @web3-storage/upload-api [upload/list method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/upload/list.js#L10) +Capability provider MUST issue `UploadGetFailure` result if content entry has not been added or was since removed from space at the time of the invocation. -### `upload/remove` - -> Remove an upload from a space. +```ipldsch +type UploadGetFailure { + message string +} +``` -`upload/remove` can be invoked to remove the link between an uploaded data CID and the CARs containing the data. +### Upload Remove -Note that this will not remove the stored CARs; clients will need to use [`store/remove`](#storeremove) to remove the CARs once all uploads referencing those CARs have been removed. +Authorized agent MAY invoke `upload/remove` capability to remove upload entry from the list in the specified space (`with` field). -The `with` resource URI must be set to the DID of the space to remove the upload from. +> โš ๏ธ Please note that removing upload entry SHOULD NOT remove [content archive][CAR]s containing contain. -#### Derivations +#### Upload Remove Capability -`upload/remove` can be derived from an [`upload/*`](#upload) or [`*`][ucan-spec-top] capability with a matching `with` resource URI. +##### Upload Remove IPLD Schema -#### Caveats +```ipldsch +type UploadRemoveCapability struct { + with SpaceDID + nb UploadRemove +} -The `root` caveat must contain the root CID of the data item to remove. +type UploadRemove struct { + root &any +} +``` -| `field` | `value` | `required?` | `context` | -| --------- | --------------------------------- | ----------- | ------------------------------------------------------------- | -| `root` | data CID string, e.g. `bafy...` | โœ” | The CID of the data item to remove. | +##### Upload Remove Root -#### Invocation +The `nb.root` field MUST be set to the [IPLD Link] of the desired content entry. -To invoke `upload/remove`, an agent prepares a UCAN with the following shape: +##### Upload Remove Example ```js { - can: "upload/remove", - with: "did:key:abc...", - nb: { - root: "bafyabc..." + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "upload/remove", + with: "did:key:zAl..1ce", + nb: { + root: { "/": "bafy...ro0t" }, + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses +#### Upload Remove Receipt + +Capability provider MUST issue signed receipt containing `UploadRemoveResult`. + +##### Upload Remove Result -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +###### Upload Remove Result IPLD Schema -Executing an `upload/remove` invocation with a service provider should return a response object. +```ipldsch +type UploadRemoveResult struct { + UploadRemoveSuccess "ok" + UploadRemoveFailure "error" +} representation keyed -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +type UploadRemoveSuccess { + link &any + shards [&ContentArchive] +} -On success, the response object will be empty. +type UploadGetFailure { + message string +} +``` -#### Implementations +### Upload List -- @web3-storage/capabilities [upload/remove validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/upload.js#L117) -- @web3-storage/upload-api [upload/remove method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/upload/remove.js#L11) +Authorized agent MAY invoke `upload/list` capability on the [space] subject (`with` field) to list upload entries at the time of the invocation. -### `upload/list` -> Obtain a list of uploaded data items. +#### Upload List Capability -The `upload/list` capability can be invoked to request a list of metadata about uploads. +##### Upload List IPLD Schema -#### Derivations +```ipldsch +type UploadListCapability struct { + with SpaceDID + nb UploadList +} -`upload/list` can be derived from an [`upload/*`](#upload) or [`*`][ucan-spec-top] capability with a matching `with` resource URI. +type UploadList struct { + cursor optional &string + size optional int + pre optional bool +} +``` -#### Caveats +##### Upload List Cursor -When invoking `upload/list` the `size` caveat may be set to the desired number of results to return per invocation. If there are more total results than will fit into the given `size`, the response will include an opaque `cursor` field that can be used to continue the listing in a subsequent invocation by setting the `cursor` caveat to the value in the response. The `pre` caveat may be set to return the page of results preceding the cursor. +The optional `nb.cursor` MAY be specified in order to paginate over the list of upload entries. -| `field` | `value` | `required?` | `context` | -| -------- | --------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | -| `size` | `number` | | The desired number of results to return. | -| `cursor` | `string` | | An opaque string included in a prior `store/list` response that allows the service to provide the next "page" of results. | -| `pre` | `boolean` | | If true, return the page of results preceding the cursor. | +##### Upload List Size -#### Invocation +The optional `nb.size` MAY be specified to signal desired page size, that is number of items in the result. + +##### Upload List Pre + +The optional `nb.pre` field MAY be set to `true` to request a page of results preceding cursor. If `nb.pre` is omitted or set to `false` provider MUST respond with a page following the specified `nb.cursor`. + +##### Upload List Example ```js { - can: "upload/list", - with: "did:key:abc..", - nb: { - size: 40, - cursor: 'cursor-value-from-previous-invocation', + "v": "0.9.1", + "iss": "did:key:z6Mkk...xALi", + "aud": "did:web:web3.storage", + "att": [ + { + can: "upload/list", + with: "did:key:zAl..1ce", + nb: { + size: 40, + cursor: 'cursor-value-from-previous-invocation', + } + } + ], + "prf": [{ "/": "bafy...prf1" }], + "exp": 1685602800, + "s": { + "/": { + "bytes": "7aEDQJbJqmyMxTxcK05XQKWfvxG+Tv+LWCJeE18RSMnciCZ/RQ21U75LA0uFSvIjdqnF5RaauZTE8mh2ZYMBBejdJQ4" + } } } ``` -#### Responses +#### Upload List Receipt -*Note*: This section is non-normative and subject to change, pending the [formalization of receipts and invocations][invocation-spec-pr]. +Capability provider MUST issue signed receipt containing `UploadListResult`. -Executing an `upload/list` invocation with a service provider should return a response object. +##### Upload List Result -If a failure occurs, the response will have an `error` field with a value of `true`, and a `message` string field with details about the error. +###### Upload List Result IPLD Schema -On success, the response object will have the following shape: +```ipldsch +type UploadListResult union { + UploadListSuccess "ok" + UploadListFailure "error" +} representation keyed +``` -```ts -interface UploadListResponse { - /** Cursor that can be used in a subsequent upload/list invocation to fetch the next page of results */ - cursor?: string +###### Upload List Success - /** Number of results in this page of listings. */ - size: number +Capability provider MUST issue `UploadListSuccess` result containing page of upload entries for to the space. - /** Items in this page of listings. */ - results: UploadListItem[] +```ipldsch +type UploadListSuccess struct { + cursor optional string + before optional string + after optional string + size int + results [UploadListItem] } -``` - -The `results` field contains an array of `UploadListItem` objects: -```ts -interface UploadListItem { - /** Root CID of the uploaded data item. */ - root: string - - /** CIDs of CARs that contain the complete DAG for the uploaded data item. */ - shards: string[] +type UploadListItem struct { + root &any + shards optional [&ContentArchive] +} +``` - /** ISO-8601 timestamp when the upload was added to the space. */ - insertedAt: string, +###### Upload List Failure - /** ISO-8601 timestamp when the upload entry was last modified. */ - updatedAt: string, +```ipldsch +type UploadListFailure struct { + message string } ``` -#### Implementations -- @web3-storage/capabilities [upload/list validator](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/capabilities/src/upload.js#L142) -- @web3-storage/upload-api [upload/list method](https://github.com/web3-storage/w3up/blob/5d52e447c14e7f7fd334e7ff575e032b7b0d89d7/packages/upload-api/src/upload/list.js#L10) - - [ucan-spec-top]: https://github.com/ucan-wg/spec#52-top [invocation-spec-pr]: https://github.com/web3-storage/specs/pull/34 + +[CAR]:https://ipld.io/specs/transport/car/ +[Content Address]:https://web3.storage/docs/concepts/content-addressing/ +[UnixFS]:https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs +[IPLD]:https://ipld.io/docs/ +[DAG-PB]:https://ipld.io/specs/codecs/dag-pb/spec/ +[DAG]:https://en.wikipedia.org/wiki/Directed_acyclic_graph +[space]:#space +[IPLD Link]:https://ipld.io/docs/schemas/features/links/ +[UCAN]:https://github.com/ucan-wg/spec/blob/692e8aab59b763a783fe1484131c3f40d997b69a/README.md +[principal]:https://github.com/ucan-wg/spec/blob/692e8aab59b763a783fe1484131c3f40d997b69a/README.md#321-principals +[provider]:#provider +[`did:mailto`]:./did-mailto.md +[`did:key`]:https://w3c-ccg.github.io/did-method-key/ +[customer]:#customer +[account]:./w3-account.md#account +[space]:#space +[subscription]:#subscription +[provision]:#provision +[DID]:https://www.w3.org/TR/did-core/ +[Content Archive Identifier]:#content-archive-identifier +[`store/add`]:Store-Add From 65701855cb6b57caebec4f69f681dd06a81de680 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Mon, 18 Mar 2024 23:10:58 -0700 Subject: [PATCH 02/18] draft: blob spec --- w3-blob.md | 199 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) create mode 100644 w3-blob.md diff --git a/w3-blob.md b/w3-blob.md new file mode 100644 index 0000000..3453ade --- /dev/null +++ b/w3-blob.md @@ -0,0 +1,199 @@ +# W3 Blob Protocol + +![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) + +- [Irakli Gozalishvili](https://github.com/gozala) + +## Authors + +- [Irakli Gozalishvili](https://github.com/gozala) + +## Abstract + +W3 blob protocol allows authorized agents to store arbitrary content blobs with a storage provider. + +## Language + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC2119](https://datatracker.ietf.org/doc/html/rfc2119). + +# Introduction + +W3 blob protocol provides core building block for storing content and sharing access to it through UCAN authorization system. It is successor to the [store protocol] which no longer requires use of [Content Archive][CAR]s even if in practice clients can continue to use it for storing shards of large DAGs. + +## Concepts + +### Roles + +There are several distinct roles that [principal]s may assume in this specification: + +| Name | Description | +| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| Principal | The general class of entities that interact with a UCAN. Identified by a DID that can be used in the `iss` or `aud` field of a UCAN. | +| Agent | A [Principal] identified by [`did:key`] identifier, representing a user in an application. | +| Issuer | A [principal] delegating capabilities to another [principal]. It is the signer of the [UCAN]. Specified in the `iss` field of a UCAN. | +| Audience | Principal access is shared with. Specified in the `aud` field of a UCAN. | + + +### Space + +A namespace, often referred as a "space", is an owned resource that can be shared. It corresponds to a unique asymmetric cryptographic keypair and is identified by a [`did:key`] URI. + +### Blob + +Blob is a fixed size byte array addressed by the [multihash]. Usually blobs are used to represent set of IPLD blocks at different byte ranges. + +# Capabilities + +## Add Blob + +Authorized agent MAY invoke `/space/content/add/blob` capability on the [space] subject to store specific byte array. + +> Note that storing a blob does not imply advertising it on the network or making it publicly available. + +### Add Blob Capability + +#### Add Blob Capability Schema + +```ipldsch +type struct AddBlob { + cmd "/space/content/add/blob" + sub SpaceDID + args Blob +} + +type struct Blob { + content Multihash + size Int +} +``` + +#### Blob Content + +Blob `content` field MUST be a [multihash] digest of the blob payload bytes, uniquely identifying blob. + +#### Blob Size + +Blob `size` field MUST be set to the size of the blob in bytes. + +### Add Blob Receipt + +#### Add Blob Result + +```ipldsch +type BlobAddResult union { + BlobAddSuccess "ok" + BlobAddFailure "error" +} representation keyed + +type BlobAddSuccess union { + BlobAddAllocation "allocated" + &UCANLocationClaim "committed" +} representation keyed + +type BlobAddAllocation struct { + content Multihash + size Int + + url URL + headers HTTPHeaders +} + +type UCANLocationClaim struct { + iss DID + aud DID + sub SpaceDID + cmd "assert/location" + args LocationClaim + nonce bytes + exp Int +} + +type LocationClaim struct { + content Multihash + url URL + range ByteRange +} + +type ByteRange struct { + offset Int + length Int +} representation tuple +``` + +#### Add Blob Success + +##### Add Blob Allocated + +Capability provider MUST issue receipt with `BlobAddAllocation` success result if underlying space has enough capacity for the requested blob, but has not such blob locally. + +The `url` field MUST be set to the HTTP PUT endpoint where `content` matching specified [multihash] and `size` could be uploaded. + +The `headers` field MUST be set to the HTTP headers to be send along with `content` to the HTTP PUT endpoint. + +###### Add Blob Committed + +Capability provider MUST issue receipt with `LocationClaim` capability delegated to the [space] when requested blob is added to the space, which can happen if provider already has a matching blob locally. + +By delegating `LocationClaim` to the space provider makes a commitment to serve corresponding blob at byte range in from given URL to all authorized agents. + +> Not that since `aud` of the claim is the `space` DID, space can authorize other principals by re-delegating this claim. + +## Get Blob + +Authorized agent MAY invoke `/space/content/get/blob` capability on the [space] subject to query a state of the corresponding blob. + +### Get Blob Capability + +#### Get Blob Capability Schema + +```ipldsch +type struct GetBlob { + cmd "/space/content/get/blob" + sub SpaceDID + args BlobQuery +} + +type struct BlobQuery { + content Multihash +} +``` + +### Get Blob Receipt + +#### Get Blob Result + +```ipldsch +type BlobAddResult union { + BlobGetSuccess "ok" + BlobGetFailure "error" +} representation keyed + +type BlobGetSuccess union { + BlobAddAllocation "allocated" + &UCANLocationClaim "committed" +} representation keyed + +type BlobGetFailure struct { + message string +} +``` + +#### Get Blob Success + +Capability provider MUST return the state of the requested blob. If memory for the blob was allocated but content has not been yet uploaded result MUST be of `BlobAddAllocation` case. If blob has been allocated and uploaded result MUST of `UCANLocationClaim` case. + +> Please note that implementation MAY check local state on invocation and update from `BlobAddAllocation` to `UCANLocationClaim`, however this SHOULD be done in referentially transparent way. In other words no side effects should be exhibited, instead function should lazily compute and cache. + +## Publish Blob + +Blob can be published by authorizing read interface (e.g. IPFS gateway) by re-delegating `LocationClaim` for a corresponding blob to it. + +> Note that same applies to publishing blob on IPNI, new capability is not necessary, user simply needs to re-delegate `LocationClaim` to the DID representing IPNI publisher. IPNI publisher in turn may publish delegation to DID with publicly known private key allowing anyone to perform the reads. + + +[store protocol]:./w3-store.md +[CAR]:https://ipld.io/specs/transport/car/ +[multihash]:https://github.com/multiformats/multihash +[space]:#space + + From 9544c3e608420aaca81cdbce99018d2aeacc8a7b Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 09:22:58 -0700 Subject: [PATCH 03/18] fix: ignore some words --- .github/workflows/words-to-ignore.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index 75d963b..c72e56c 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -160,3 +160,6 @@ dag-json-encoded base64url-multibase-encoded Uint8Array ed25519 +multihash +IPNI +ARchive From 7f9c5487da6945bdcbbd76654201d16df2b05684 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 09:23:41 -0700 Subject: [PATCH 04/18] fix: grammar --- w3-blob.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/w3-blob.md b/w3-blob.md index 3453ade..c147f6d 100644 --- a/w3-blob.md +++ b/w3-blob.md @@ -182,18 +182,18 @@ type BlobGetFailure struct { Capability provider MUST return the state of the requested blob. If memory for the blob was allocated but content has not been yet uploaded result MUST be of `BlobAddAllocation` case. If blob has been allocated and uploaded result MUST of `UCANLocationClaim` case. -> Please note that implementation MAY check local state on invocation and update from `BlobAddAllocation` to `UCANLocationClaim`, however this SHOULD be done in referentially transparent way. In other words no side effects should be exhibited, instead function should lazily compute and cache. +> Please note that implementation MAY check local state on invocation and update from `BlobAddAllocation` to `UCANLocationClaim`, however this SHOULD be done in way that preserves [referential transparency]. In other words no side effects should be observable, instead function should lazily compute and cache. ## Publish Blob Blob can be published by authorizing read interface (e.g. IPFS gateway) by re-delegating `LocationClaim` for a corresponding blob to it. -> Note that same applies to publishing blob on IPNI, new capability is not necessary, user simply needs to re-delegate `LocationClaim` to the DID representing IPNI publisher. IPNI publisher in turn may publish delegation to DID with publicly known private key allowing anyone to perform the reads. +> Note that same applies to publishing blob on [IPNI], new capability is not necessary, user simply needs to re-delegate `LocationClaim` to the DID representing [IPNI] publisher. [IPNI] publisher in turn may publish delegation to DID with publicly known private key allowing anyone to perform the reads. [store protocol]:./w3-store.md [CAR]:https://ipld.io/specs/transport/car/ [multihash]:https://github.com/multiformats/multihash [space]:#space - - +[referential transparency]:https://en.wikipedia.org/wiki/Referential_transparency +[IPNI]:https://github.com/ipni/specs/blob/main/ipni.md From 0ab6911920c2c9edeff30bf77d3eff71bbbe54b9 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 09:22:58 -0700 Subject: [PATCH 05/18] fix: ignore some words --- .github/workflows/words-to-ignore.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index 75d963b..c72e56c 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -160,3 +160,6 @@ dag-json-encoded base64url-multibase-encoded Uint8Array ed25519 +multihash +IPNI +ARchive From 90e9607765c849b9e741ecae80d1e6d903a564c1 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 09:30:32 -0700 Subject: [PATCH 06/18] chore: ignore more words --- .github/workflows/words-to-ignore.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index c72e56c..2ca3c19 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -162,4 +162,7 @@ Uint8Array ed25519 multihash IPNI -ARchive +aRchive +SHA2-256 +noop +Pre From 4f94fa925aec91d6877794c9cfbf6f7c479c39b7 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 09:30:32 -0700 Subject: [PATCH 07/18] chore: ignore more words --- .github/workflows/words-to-ignore.txt | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index c72e56c..2ca3c19 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -162,4 +162,7 @@ Uint8Array ed25519 multihash IPNI -ARchive +aRchive +SHA2-256 +noop +Pre From 6ede80c9dbbc99d367a1a8ecc5f6eb32587f3b4b Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 11:14:22 -0700 Subject: [PATCH 08/18] Apply suggestions from code review Co-authored-by: Vasco Santos --- w3-store.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/w3-store.md b/w3-store.md index e21b961..20fb1ec 100644 --- a/w3-store.md +++ b/w3-store.md @@ -1,6 +1,6 @@ # W3 Storage Protocol -![status:wip](https://img.shields.io/badge/status-wip-orange.svg?style=flat-square) +![reliable](https://img.shields.io/badge/status-reliable-green.svg?style=flat-square) ## Editors @@ -14,7 +14,7 @@ ## Abstract -In the W3 protocol user owned (name)space represent a data storage primitive that can be managed using W3 storage protocol defined here. Storage protocol allows space owners to manage state across compatible storage provider services using defined set of [UCAN] capabilities. Use of [UCAN] authorization system enables space access to be shared by delegating corresponding capabilities to desired audience. +In the W3 protocol user owned (name)space represents a data storage primitive that can be managed using W3 storage protocol defined here. Storage protocol allows space owners to manage state across compatible storage provider services using defined set of [UCAN] capabilities. Use of [UCAN] authorization system enables space access to be shared by delegating corresponding capabilities to desired audience. ## Language @@ -24,7 +24,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S The base storage layer of the space is [Content Address]ed [Content ARchive][CAR] files. In practice this means that user wishing to store files or directories of files needs produce an [IPLD] [DAG] in [UnixFS] format and then encoded into one or multiple [Content ARchive]s that can be stored using W3 storage protocol. -> Large DAGs get "sharded" across multiple [Content ARchive]s and stored individually to meet size limits of the storage provider. +> Large DAGs get "sharded" across multiple [Content ARchive]s and stored individually to meet potential size limits of the storage provider. Separately `upload/` protocol can be utilized allowing user to create standalone entities (files, directories) representing entry points to the DAGs contained by the [Content ARchive]s. E.g. when storing a file or a directory [UnixFS] root [CID] and archives are captured using `upload/add` capability allowing viewer to effectively fetch and re-assemble it. @@ -283,6 +283,7 @@ Capability provider MUST issue `StoreGetSuccess` result for every content archiv type StoreGetSuccess { link &ContentArchive size Int + # deprecated origin optional &ContentArchive } ``` @@ -364,7 +365,7 @@ type StoreRemoveResult struct { ###### Store Remove Success -Capability provider MUST issue `StoreRemoveSuccess` after it has removed the archive from the space. +Capability provider MUST issue `StoreRemoveSuccess` after it unlinked the archive from the space. Capability provider MUST set `size` field to the number of bytes that were freed from space. @@ -668,7 +669,7 @@ type UploadGetFailure { Authorized agent MAY invoke `upload/remove` capability to remove upload entry from the list in the specified space (`with` field). -> โš ๏ธ Please note that removing upload entry SHOULD NOT remove [content archive][CAR]s containing contain. +> โš ๏ธ Please note that removing upload entry MUST NOT remove [content archive][CAR]s containing contain. #### Upload Remove Capability From 192c183daca8936335591449646b7f8c3af774be Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 11:18:35 -0700 Subject: [PATCH 09/18] fix: spelling errors --- .github/workflows/words-to-ignore.txt | 1 + w3-store.md | 10 +++++----- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index 2ca3c19..e6ab17e 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -166,3 +166,4 @@ aRchive SHA2-256 noop Pre +unlinked diff --git a/w3-store.md b/w3-store.md index e21b961..04943b3 100644 --- a/w3-store.md +++ b/w3-store.md @@ -22,11 +22,11 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "S # Introduction -The base storage layer of the space is [Content Address]ed [Content ARchive][CAR] files. In practice this means that user wishing to store files or directories of files needs produce an [IPLD] [DAG] in [UnixFS] format and then encoded into one or multiple [Content ARchive]s that can be stored using W3 storage protocol. +The base storage layer of the space is [Content Address]ed [Content Archive][CAR] files. In practice this means that user wishing to store files or directories of files needs produce an [IPLD] [DAG] in [UnixFS] format and then encoded into one or multiple [Content Archive]s that can be stored using W3 storage protocol. -> Large DAGs get "sharded" across multiple [Content ARchive]s and stored individually to meet size limits of the storage provider. +> Large DAGs get "sharded" across multiple [Content Archive]s and stored individually to meet size limits of the storage provider. -Separately `upload/` protocol can be utilized allowing user to create standalone entities (files, directories) representing entry points to the DAGs contained by the [Content ARchive]s. E.g. when storing a file or a directory [UnixFS] root [CID] and archives are captured using `upload/add` capability allowing viewer to effectively fetch and re-assemble it. +Separately `upload/` protocol can be utilized allowing user to create standalone entities (files, directories) representing entry points to the DAGs contained by the [Content Archive]s. E.g. when storing a file or a directory [UnixFS] root [CID] and archives are captured using `upload/add` capability allowing viewer to effectively fetch and re-assemble it. ## Concepts @@ -48,7 +48,7 @@ A namespace, often referred as a "space", is an owned resource that can be share ## Content Archive -[Content Archive][CAR] ofter referred as [CAR] is a primary primitive for storing shards of the content in the space. +[Content Archive][CAR] often referred as [CAR] is a primary primitive for storing shards of the content in the space. # Capabilities @@ -79,7 +79,7 @@ The subject of the invocation (`with` field) MUST be the DID of the target MUST ### Content Archive Identifier -The `nb.link` field of the invocation MUST be an [IPLD Link] to the desired [CAR]. Link MUST have Content Addressable aRchive (CAR) `0x0202` codec code. It is RECOMMENDED to support SHA2-256 multihash code `0x12`. Implementers are MAY choose to support other additional hashing algorithms. +The `nb.link` field of the invocation MUST be an [IPLD Link] to the desired [CAR]. Link MUST have Content Addressable Archive (CAR) `0x0202` codec code. It is RECOMMENDED to support SHA2-256 multihash code `0x12`. Implementers are MAY choose to support other additional hashing algorithms. ### Store Add From 6784ecd247c5da7069ffc5b3239d454ae93911ac Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 11:21:35 -0700 Subject: [PATCH 10/18] fix: broken link --- w3-store.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/w3-store.md b/w3-store.md index 7c772be..0e3ffe0 100644 --- a/w3-store.md +++ b/w3-store.md @@ -868,4 +868,4 @@ type UploadListFailure struct { [provision]:#provision [DID]:https://www.w3.org/TR/did-core/ [Content Archive Identifier]:#content-archive-identifier -[`store/add`]:Store-Add +[`store/add`]:#Store-Add From e95aab224ed9f2f4740aa08fbac23821a0f64a72 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 11:27:44 -0700 Subject: [PATCH 11/18] fix: md lint issues --- w3-store.md | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-) diff --git a/w3-store.md b/w3-store.md index 0e3ffe0..08626c4 100644 --- a/w3-store.md +++ b/w3-store.md @@ -41,7 +41,6 @@ There are several distinct roles that [principal]s may assume in this specificat | Issuer | A [principal] delegating capabilities to another [principal]. It is the signer of the [UCAN]. Specified in the `iss` field of a UCAN. | | Audience | Principal access is shared with. Specified in the `aud` field of a UCAN. | - ### Space A namespace, often referred as a "space", is an owned resource that can be shared. It corresponds to a unique asymmetric cryptographic keypair and is identified by a [`did:key`] URI. The `store/` and `upload/` capabilities can be used to manage content stored in the given space at given storage provider. @@ -116,7 +115,6 @@ Capability invocation MUST specify [Content archive Identifier]. Capability invocation MUST set `nb.size` field to the byte size of the [content archive][CAR]. - ##### Store Add Origin > Status: Deprecated ๐Ÿ›‘ @@ -744,7 +742,6 @@ type UploadGetFailure { Authorized agent MAY invoke `upload/list` capability on the [space] subject (`with` field) to list upload entries at the time of the invocation. - #### Upload List Capability ##### Upload List IPLD Schema @@ -843,29 +840,15 @@ type UploadListFailure struct { } ``` - - -[ucan-spec-top]: https://github.com/ucan-wg/spec#52-top -[invocation-spec-pr]: https://github.com/web3-storage/specs/pull/34 - [CAR]:https://ipld.io/specs/transport/car/ [Content Address]:https://web3.storage/docs/concepts/content-addressing/ [UnixFS]:https://docs.ipfs.tech/concepts/file-systems/#unix-file-system-unixfs [IPLD]:https://ipld.io/docs/ -[DAG-PB]:https://ipld.io/specs/codecs/dag-pb/spec/ [DAG]:https://en.wikipedia.org/wiki/Directed_acyclic_graph [space]:#space [IPLD Link]:https://ipld.io/docs/schemas/features/links/ [UCAN]:https://github.com/ucan-wg/spec/blob/692e8aab59b763a783fe1484131c3f40d997b69a/README.md [principal]:https://github.com/ucan-wg/spec/blob/692e8aab59b763a783fe1484131c3f40d997b69a/README.md#321-principals -[provider]:#provider -[`did:mailto`]:./did-mailto.md -[`did:key`]:https://w3c-ccg.github.io/did-method-key/ -[customer]:#customer -[account]:./w3-account.md#account -[space]:#space -[subscription]:#subscription -[provision]:#provision -[DID]:https://www.w3.org/TR/did-core/ [Content Archive Identifier]:#content-archive-identifier -[`store/add`]:#Store-Add +[`store/add`]:#store-add +[provider]:./w3-provider.md#provider From 57511772a36a68145c9dd968ea11f09882d51bec Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 11:49:25 -0700 Subject: [PATCH 12/18] chore: remove double blank --- w3-store.md | 1 - 1 file changed, 1 deletion(-) diff --git a/w3-store.md b/w3-store.md index 08626c4..3c4caba 100644 --- a/w3-store.md +++ b/w3-store.md @@ -345,7 +345,6 @@ Capability invocation MUST specify desired [Content archive Identifier]. } ``` - #### Store Remove Receipt Capability provider MUST issue signed receipt containing `StoreRemoveResult`. From 727d905ceefdd6eb46c19227dbcb3e7a35cdb09b Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 12:00:45 -0700 Subject: [PATCH 13/18] fix: capture allocation fields --- w3-store.md | 20 ++++++++++++++------ 1 file changed, 14 insertions(+), 6 deletions(-) diff --git a/w3-store.md b/w3-store.md index 3c4caba..ece87be 100644 --- a/w3-store.md +++ b/w3-store.md @@ -172,15 +172,18 @@ type StoreAddSuccess union { } type StoreAddDone struct { - with SpaceDID - link &ContentArchive + with SpaceDID + link &ContentArchive + allocated Int } type StoreAddPending struct { - with SpaceDID - link &ContentArchive - url URL - headers HTTPHeaders + with SpaceDID + link &ContentArchive + url URL + headers HTTPHeaders + + allocated Int } type URL = string @@ -195,6 +198,8 @@ type StoreAddFailure struct { Capability provider MUST succeed request with a `"done"` status if provider is able to store requested [CAR] on user behalf right-away. In other words provider already has addressed [CAR] file. +Capability provider MUST set `allocated` field to number of bytes it had no allocate for the [CAR], which MUST be either equal to `nb.size` of the invocation or `0` if no additional memory had to be allocated. + ###### Store Add Upload Capability provider MUST succeed request with a `"upload"` status if it is able to allocate memory for the requested [CAR]. It MUST set `url` field to an HTTP PUT endpoint where addressed [CAR] can be uploaded. @@ -203,6 +208,9 @@ Provider MUST set `headers` field to the set of HTTP headers that uploading agen HTTP PUT endpoint set in `url` field MUST verify that uploaded bytes do correspond to the addressed [CAR] and specified `size`. +Capability provider MUST set `allocated` field to number of bytes it had no allocate for the [CAR], which MUST be either equal to `nb.size` of the invocation or `0` if no additional memory had to be allocated. + + ###### Store Add Failure Capability provider SHOULD fail invocation if the subject [space] has not been provisioned with the provider. From 0140d3e0f9d8f93e2085672618e7145dca8dea92 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 12:02:54 -0700 Subject: [PATCH 14/18] fix: lint --- w3-store.md | 1 - 1 file changed, 1 deletion(-) diff --git a/w3-store.md b/w3-store.md index ece87be..cc7fed7 100644 --- a/w3-store.md +++ b/w3-store.md @@ -210,7 +210,6 @@ HTTP PUT endpoint set in `url` field MUST verify that uploaded bytes do correspo Capability provider MUST set `allocated` field to number of bytes it had no allocate for the [CAR], which MUST be either equal to `nb.size` of the invocation or `0` if no additional memory had to be allocated. - ###### Store Add Failure Capability provider SHOULD fail invocation if the subject [space] has not been provisioned with the provider. From b292df388a2a8f4d5f13dfdb95842c443faaf8c7 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Tue, 19 Mar 2024 12:27:47 -0700 Subject: [PATCH 15/18] fix: broken link --- w3-blob.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/w3-blob.md b/w3-blob.md index c147f6d..d0bb074 100644 --- a/w3-blob.md +++ b/w3-blob.md @@ -33,7 +33,6 @@ There are several distinct roles that [principal]s may assume in this specificat | Issuer | A [principal] delegating capabilities to another [principal]. It is the signer of the [UCAN]. Specified in the `iss` field of a UCAN. | | Audience | Principal access is shared with. Specified in the `aud` field of a UCAN. | - ### Space A namespace, often referred as a "space", is an owned resource that can be shared. It corresponds to a unique asymmetric cryptographic keypair and is identified by a [`did:key`] URI. @@ -190,10 +189,9 @@ Blob can be published by authorizing read interface (e.g. IPFS gateway) by re-de > Note that same applies to publishing blob on [IPNI], new capability is not necessary, user simply needs to re-delegate `LocationClaim` to the DID representing [IPNI] publisher. [IPNI] publisher in turn may publish delegation to DID with publicly known private key allowing anyone to perform the reads. - [store protocol]:./w3-store.md [CAR]:https://ipld.io/specs/transport/car/ [multihash]:https://github.com/multiformats/multihash [space]:#space [referential transparency]:https://en.wikipedia.org/wiki/Referential_transparency -[IPNI]:https://github.com/ipni/specs/blob/main/ipni.md +[IPNI]:https://github.com/ipni/specs/blob/main/IPNI.md From 6a60a0af7cc4a0b818b6feafc237cbe9fc007f62 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Fri, 22 Mar 2024 01:49:43 -0700 Subject: [PATCH 16/18] chore: second draft --- w3-blob.md | 380 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 301 insertions(+), 79 deletions(-) diff --git a/w3-blob.md b/w3-blob.md index d0bb074..763dcc0 100644 --- a/w3-blob.md +++ b/w3-blob.md @@ -49,143 +49,358 @@ Authorized agent MAY invoke `/space/content/add/blob` capability on the [space] > Note that storing a blob does not imply advertising it on the network or making it publicly available. +### Add Blob Diagram + +Following diagram illustrates execution workflow. + +```mermaid +flowchart TB +Add("โ–ถ๏ธ /space/content/add/blob ๐Ÿ‘ฉโ€๐Ÿ’ป ๐Ÿค–") + +Added("๐Ÿงพ { ok: { claim ๐Ÿšฆ } }") + +Allocate("โญ๏ธ /service/blob/allocate ๐Ÿค–") + +Accept("โญ๏ธ /service/blob/accept ๐Ÿค–") + +Accepted("๐Ÿงพ { ok: { claim ๐ŸŽซ } }") + + +Claim("๐ŸŽซ /assert/location ๐Ÿค–๐Ÿ‘ฉโ€๐Ÿ’ป") + +Add --o Added +Added -.-> Allocate +Added -.-> Accept +Allocate --> Accept +Accept --o Accepted +Accepted -- claim --> Claim +Added -- claim --> Accept +``` + +**Iconography** + +- Icon on the left describes type of the node +- Icon on the right describes issuer / audience pair if only one audience is the issuer +- โ–ถ๏ธ Task +- โญ๏ธ Next Task (a.k.a Effect) +- ๐Ÿงพ Receipt +- ๐ŸŽซ Delegation / Commitment +- ๐Ÿšฆ Await +- ๐Ÿ‘ฉโ€๐Ÿ’ป Alice +- ๐Ÿค– Service + +### Add Blob Invocation Example + +Shown Invocation example illustrates Alice requesting to add 2MiB blob to her space. + +```js +{ + "cmd": "/space/content/add/blob", + "sub": "did:key:zAlice", + "iss": "did:key:zAlice", + "aud": "did:web:web3.storage", + "args": { + "blob": { + // multihash of the blob as byte array + "content": { "/": { "bytes": "mEi...sfKg" } }, + // size of the blob in bytes + "size": 2_097_152, + } + } +} +``` + +### Add Blob Receipt Example + +Shows an example receipt for the above `/space/content/add/blob` capability invocation. + +> โ„น๏ธ We use `// "/": "bafy..` comments to denote CID of the parent object. + +```js +{ // "/": "bafy..add", + "iss": "did:web:web3.storage", + "aud": "did:key:zAlice", + "cmd": "/ucan/assert/result" + "sub": "did:web:web3.storage", + "args": { + // refers to the invocation from the example + "ran": { "/": "bafy..add" }, + "out": { + "ok": { + // result of the add is the content (location) claim + // that is produced as result of "bafy..accept" + "claim": { "await/ok": { "/": "bafy...accept" } } + } + }, + // Previously `next` was known as `fx` instead, which is + // set of tasks to be scheduled. + "next": [ + // 1. System attempts to allocate memory in user space for the blob. + { // "/": "bafy...alloc", + "cmd": "/service/blob/allocate", + "sub": "did:web:web3.storage", + "args": { + // space where memory is allocated + "space": "did:key:zAlice", + "blob": { + // multihash of the blob as byte array + "content": { "/": { "bytes": "mEi...sfKg" } }, + // size of the blob in bytes + "size": 2_097_152, + } + } + }, + // 2. System will attempt to accept received content + // if matches blob multihash and size. + { // "/": "bafy...accept", + "cmd": "/service/blob/accept", + "sub": "did:web:web3.storage", + "args": { + "space": "did:key:zAlice", + "blob": { + // multihash of the blob as byte array + "content": { "/": { "bytes": "mEi...sfKg" } }, + "size": 2_097_152, + }, + // This task is blocked on allocation + _: { "await/ok": { "/": "bafy...alloc" } } + } + } + ] + } +} +``` + ### Add Blob Capability #### Add Blob Capability Schema -```ipldsch -type struct AddBlob { - cmd "/space/content/add/blob" - sub SpaceDID - args Blob +```ts +type AddBlob = { + cmd: "/space/content/add/blob" + sub: SpaceDID + args: { + blob: Blob + } } -type struct Blob { - content Multihash - size Int +type Blob = { + content: Multihash + size: int } + +type Multihash = bytes +type SpaceDID = string ``` #### Blob Content -Blob `content` field MUST be a [multihash] digest of the blob payload bytes, uniquely identifying blob. +The `args.blob.content` field MUST be a [multihash] digest of the blob payload bytes. Implementation SHOULD support SHA2-256 algorithm. Implementation MAY in addition support other hashing algorithms. #### Blob Size -Blob `size` field MUST be set to the size of the blob in bytes. +Blob `args.blob.size` field MUST be set to the number of bytes in the blob content. ### Add Blob Receipt +#### Add Blob Receipt Schema + +```ts +// Only operation specific fields are covered the +// rest are implied +type AddBlobReceipt = { + out: Result + next: [ + AllocateBlob, + AcceptBlob, + ] +} + +type AddBlobOk = { + claim: { + "await/ok": Link + } +} + +type AddBlobError = { + message: string +} +``` + #### Add Blob Result -```ipldsch -type BlobAddResult union { - BlobAddSuccess "ok" - BlobAddFailure "error" -} representation keyed +Invocation MUST fail if any of the following is true + +1. Provided **sub**ject space is not provisioned with a provider. +1. Provided `blob.size` is outside of supported range. +1. Provided `blob.content` is not a valid [multihash]. +1. Provided `blob.content` [multihash] hashing algorithm is not supported. + +Invocation MUST succeed if non of the above is true. Success value MUST be an object with a `claim` field set to [await/ok] of the task that produces [location claim]. + +Task linked from the `claim` of the success value MUST be present in the receipt effects _(`next` field)_. -type BlobAddSuccess union { - BlobAddAllocation "allocated" - &UCANLocationClaim "committed" -} representation keyed +#### Add Blob Effects -type BlobAddAllocation struct { - content Multihash - size Int +Successful invocation MUST start a workflow consisting of following tasks, that MUST be set in receipt effects (`next` field) in the following order. - url URL - headers HTTPHeaders +1. [Allocate Blob] +1. [Accept Blob] + +## Allocate Blob + +Authorized agent MAY invoke `/service/blob/allocate` capability on the [provider] subject to create a memory address where `blob` content can be written via HTTP `PUT` request. + +### Allocate Blob Capability + +#### Allocate Blob Capability Schema + +```ts +type BlobAllocate = { + cmd: "/service/blob/allocate" + sub: ProviderDID + args: { + space: SpaceDID + blob: Blob + cause: Link + } } +``` + +#### Allocation Space + +The `args.space` field MUST be set to the [DID] of the user space where allocation takes place. + +#### Allocation Blob + +The `args.blob` field MUST be set to the `Blob` the space is allocated for. + +#### Allocation Cause + +The `args.cause` field MUST be set to the [Link] for an [Add Blob] task, that caused an allocation. + +### Allocate Blob Receipt + +Allocations MUST fail if `space` does not have enough capacity for the `blob` and succeed otherwise. -type UCANLocationClaim struct { - iss DID - aud DID - sub SpaceDID - cmd "assert/location" - args LocationClaim - nonce bytes - exp Int +#### Allocate Blob Receipt Schema + +```ts +type BlobAllocateReceipt = { + ran: Link + out: Result + next: [] } -type LocationClaim struct { - content Multihash - url URL - range ByteRange +type BlobAllocateOk = { + size: int + address?: BlobAddress } -type ByteRange struct { - offset Int - length Int -} representation tuple +type BlobAddress = { + url: string + headers: {[key:string]: string} +} ``` -#### Add Blob Success +### Allocation Size + +The `out.ok.size` MUST be set to the number of bytes that were allocated for the `Blob`. It MUST be equal to either: + +1. The `args.blob.size` of the invocation. +2. `0` if space already has memory allocated for the `args.blob`. + +### Allocation Address + +The optional `out.ok.address` SHOULD be omitted when content for the allocated is already available on site. Otherwise it MUST be set to the `BlobAddress` that can receive a blob content. + +The `url` of the `BlobAddress` MUST be an HTTP(S) location that can receive blob content via HTTP `PUT` request, as long as HTTP headers from `headers` dictionary are set on the request. -##### Add Blob Allocated +It is RECOMMENDED that issued `BlobAddress` only accept `PUT` payload that matches requested `blob`, both content [multihash] and size. -Capability provider MUST issue receipt with `BlobAddAllocation` success result if underlying space has enough capacity for the requested blob, but has not such blob locally. +> โ„น๏ธ If enforcing above recommendation is not possible implementation MUST enforce in [Accept Blob] invocation. -The `url` field MUST be set to the HTTP PUT endpoint where `content` matching specified [multihash] and `size` could be uploaded. +### Allocation Effects -The `headers` field MUST be set to the HTTP headers to be send along with `content` to the HTTP PUT endpoint. +Allocation MUST have no effects. -###### Add Blob Committed +## Accept Blob -Capability provider MUST issue receipt with `LocationClaim` capability delegated to the [space] when requested blob is added to the space, which can happen if provider already has a matching blob locally. +Authorized agent MAY invoke `/service/blob/accept` capability on the [provider] subject. Invocation MUST either succeed when content is delivered on allocated `address` or fail if no content is allocation expires without content being delivered. -By delegating `LocationClaim` to the space provider makes a commitment to serve corresponding blob at byte range in from given URL to all authorized agents. +โ„น๏ธ Implementation that is unable to enforce to reject HTTP PUT request that do not match blob [multihash] or `size` SHOULD enforce that invariant in this invocation by failing task if no valid content has been delivered. -> Not that since `aud` of the claim is the `space` DID, space can authorize other principals by re-delegating this claim. +### Accept Blob Capability -## Get Blob +#### Accept Blob Capability Schema -Authorized agent MAY invoke `/space/content/get/blob` capability on the [space] subject to query a state of the corresponding blob. +```ts +type BlobAccept = { + cmd: "/service/blob/accept" + sub: ProviderDID + args: { + blob: Blob + exp: int + } +} +``` -### Get Blob Capability +### Accept Blob Receipt -#### Get Blob Capability Schema +#### Accept Blob Receipt Schema + +```ts +type BlobAcceptReceipt = { + ran: Link + out: Result + next: [] +} -```ipldsch -type struct GetBlob { - cmd "/space/content/get/blob" - sub SpaceDID - args BlobQuery +type BlobAcceptOk = { + claim: Link } -type struct BlobQuery { - content Multihash +type BlobAcceptError = { + message: string } ``` -### Get Blob Receipt +#### Blob Accept Effects -#### Get Blob Result +Receipt MUST not have any effects. -```ipldsch -type BlobAddResult union { - BlobGetSuccess "ok" - BlobGetFailure "error" -} representation keyed +## Location Claim -type BlobGetSuccess union { - BlobAddAllocation "allocated" - &UCANLocationClaim "committed" -} representation keyed +Location claim represents commitment from the issuer to the audience that +content matching the `content` [multihash] can be read via HTTP [range request] -type BlobGetFailure struct { - message string +### Location Claim Capability + +#### Location Claim Capability Schema + +```ts +type LocationClaim = { + cmd: "assert/location" + sub: ProviderDID + args: { + content: Multihash + url: string + range: [start:int, end:int, ...int[]] + } } ``` -#### Get Blob Success +# Coordination + +## Accept Content -Capability provider MUST return the state of the requested blob. If memory for the blob was allocated but content has not been yet uploaded result MUST be of `BlobAddAllocation` case. If blob has been allocated and uploaded result MUST of `UCANLocationClaim` case. +[Accept Blob] invocation will block until content is delivered, however some implementations may not be able to observe when content was received. Those implementations can await for subsequent [Add Blob] invocations and re-check whether content has been received. -> Please note that implementation MAY check local state on invocation and update from `BlobAddAllocation` to `UCANLocationClaim`, however this SHOULD be done in way that preserves [referential transparency]. In other words no side effects should be observable, instead function should lazily compute and cache. +Note that implementation MUST be idempotent and same receipts MUST be returned to the caller, yet pending tasks could be updated. -## Publish Blob +## Publishing Blob -Blob can be published by authorizing read interface (e.g. IPFS gateway) by re-delegating `LocationClaim` for a corresponding blob to it. +Blob can be published by authorizing read interface (e.g. IPFS gateway) by delegating it [Location Claim] that has been obtained from the provider. > Note that same applies to publishing blob on [IPNI], new capability is not necessary, user simply needs to re-delegate `LocationClaim` to the DID representing [IPNI] publisher. [IPNI] publisher in turn may publish delegation to DID with publicly known private key allowing anyone to perform the reads. @@ -193,5 +408,12 @@ Blob can be published by authorizing read interface (e.g. IPFS gateway) by re-de [CAR]:https://ipld.io/specs/transport/car/ [multihash]:https://github.com/multiformats/multihash [space]:#space -[referential transparency]:https://en.wikipedia.org/wiki/Referential_transparency [IPNI]:https://github.com/ipni/specs/blob/main/IPNI.md +[await/ok]:https://github.com/ucan-wg/invocation?tab=readme-ov-file#await +[location claim]:#location-claim +[Add Blob]:#add-blob +[Allocate Blob]:#allocate-blob +[Accept Blob]:#receive-blob +[DID]:https://www.w3.org/TR/did-core/ +[Link]:https://ipld.io/docs/schemas/features/links/ +[range request]:https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests From 76ef963b63c4b2ea960a715d871e921b5cbaf469 Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Fri, 22 Mar 2024 09:29:47 -0700 Subject: [PATCH 17/18] Apply suggestions from code review --- .github/workflows/words-to-ignore.txt | 2 ++ w3-blob.md | 5 +++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/.github/workflows/words-to-ignore.txt b/.github/workflows/words-to-ignore.txt index e6ab17e..8ce22f1 100644 --- a/.github/workflows/words-to-ignore.txt +++ b/.github/workflows/words-to-ignore.txt @@ -167,3 +167,5 @@ SHA2-256 noop Pre unlinked +2MiB +ok diff --git a/w3-blob.md b/w3-blob.md index 763dcc0..5d3fa9e 100644 --- a/w3-blob.md +++ b/w3-blob.md @@ -77,7 +77,7 @@ Accepted -- claim --> Claim Added -- claim --> Accept ``` -**Iconography** +#### Iconography - Icon on the left describes type of the node - Icon on the right describes issuer / audience pair if only one audience is the issuer @@ -162,6 +162,7 @@ Shows an example receipt for the above `/space/content/add/blob` capability invo "content": { "/": { "bytes": "mEi...sfKg" } }, "size": 2_097_152, }, + exp: 1711122994101 // This task is blocked on allocation _: { "await/ok": { "/": "bafy...alloc" } } } @@ -413,7 +414,7 @@ Blob can be published by authorizing read interface (e.g. IPFS gateway) by deleg [location claim]:#location-claim [Add Blob]:#add-blob [Allocate Blob]:#allocate-blob -[Accept Blob]:#receive-blob +[Accept Blob]:#accept-blob [DID]:https://www.w3.org/TR/did-core/ [Link]:https://ipld.io/docs/schemas/features/links/ [range request]:https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests From bcccbc0c3605e0f7577b343071c2ec71dc0a8dfb Mon Sep 17 00:00:00 2001 From: Irakli Gozalishvili Date: Fri, 22 Mar 2024 09:30:30 -0700 Subject: [PATCH 18/18] Update w3-blob.md --- w3-blob.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/w3-blob.md b/w3-blob.md index 5d3fa9e..354edaa 100644 --- a/w3-blob.md +++ b/w3-blob.md @@ -162,7 +162,7 @@ Shows an example receipt for the above `/space/content/add/blob` capability invo "content": { "/": { "bytes": "mEi...sfKg" } }, "size": 2_097_152, }, - exp: 1711122994101 + exp: 1711122994101, // This task is blocked on allocation _: { "await/ok": { "/": "bafy...alloc" } } }