Processor design RFCs

docs/rfc/manifest-file.md

@@ -0,0 +1,102 @@
+# Manifest file for Hydra processor
+
+## Summary
+
+The Manifest file is a high-level `.yml` file which describes _how_ and _what_ the Hydra processor has to run. The manifest file + the mappings should the sufficient for running the processor from a clean state. It should replace the root `.env` presently used as a config file.
+
+## Goals and motivation
+
+1) We need a more structured and clear way to configure the processor
+
+2) As the complexity of the processor grows, a descriptive config with nested properties is required
+
+3) Smoother transition for The Graph users
+
+## Urgency
+
+As we add more features to the processor, we need a unified high-level definition of how the processor should apply the handlers.
+
+## Detailed Design
+
+### 1.1 Top-Level API
+
+Similar to TheGraph design
+
+| Field  | Type | Description   |
+| --- | --- | --- |
+| **specVersion** | *String*   | A Semver version indicating which version of this API is being used.|
+| **schema**   | [*Schema*](#12-schema) | The GraphQL schema of this subgraph.|
+| **description**   | *String* | An optional description of the subgraph's purpose. |
+| **repository**   | *String* | An optional link to where the subgraph lives. |
+| **dataSources**| [*Data Source Spec*](#13-data-source)| Each data source spec defines the data that will be ingested as well as the transformation logic to derive the state of the subgraph's entities based on the source data.|
+
+### 1.2 Schema
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **file**| [*Path*](#16-path) | The path of the GraphQL IDL file (no IPFS support for now) |
+
+### 1.3 Data Source
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **kind** | *String* | The type of data source. Possible values: *substrate/runtime* (vs *ethereum/contract* for TheGraph).|
+| **name** | *String* | The name of the source data. Will be used to generate APIs in the mapping and also for self-documentation purposes. |
+| **network** | *String* | For blockchains, this describes which network the subgraph targets. For example, "kusama" or "joystream/babylon". |
+| **source** | [*HydraIndexerSource*](#131-hydraindexersource) | The source data for ingestion. |
+| **mapping** | [*Mapping*](#132-mapping) | The transformation logic applied to the data prior to being indexed. |
+
+#### 1.3.1 Hydra Indexer Source
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **endpoint** | *String* | The endpoint for the Hydra Indexer gateway providing the data. |
+| **startBlock** | optional *BigInt* | The block to start processing this data source from. |
+| **endBlock** | optional *BigInt* | The block at which the processor stops and exits. |
+
+#### 1.3.2 Mapping
+
+##### 1.3.2.1 Substrate Mapping
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **kind** | *String* | Must be "substrate/events" for Substrate Events Mapping. |
+| **apiVersion** | *String* | Semver string of the version of the Mappings API that will be used by the mapping script. |
+| **language** | *String* | The language of the runtime for the Mapping API. For now only *typescript*. |
+| **entities** | *[String]* | A list of entities that will be ingested as part of this mapping. Must correspond to names of entities in the GraphQL IDL. |
+| **types** | *String* path to a file with definitions | A typescript file exporting the types and interfaces satisfying the event and extrinsic signatures in the handler definitions. Typically, the file reexports the standard polkadot types together with custom files |
+| **eventHandlers** | optional *EventHandler* | Handlers for specific events, which will be defined in the mapping script. |
+| **extrinsicHandlers** | optional *ExtrinsicHandler* | A list of functions that will trigger a handler on `system.ExtrinsicSuccess` event with the extrinsic data. |
+| **blockHandlers** | optional *BlockHandler* | Defines block filters and handlers to process matching blocks. |
+| **file** | [*Path*] | The path of the mapping script exporting the mapping functions. |
+
+#### 1.3.2.2 EventHandler
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **event** | *String* | An identifier for an event that will be handled in the mapping script. It must be in the form `<module>.<method>(type1,type2,...,)` as defined in the metadata file. For example, `balances.DustLost(AccountId,Balance)`. The declared types should be available via globally defined `typeRegistry`. |
+| **extrinsic** | optional *String* | The extrinsic that caused the event. If present, only events emitted by the specified extrinsics will be handled by the handler. Must have a fully qualified name in the form
+`<section>.<method>(type1,type2,...,)`|
+| **handler** | *String* | The name of an exported function in the mapping script that should handle the specified event. |
+| **virtual** | optional *Boolean* | If the event to be handled is _virtual_, that is, emitted by an extrinsic handler than by the chain |
+
+#### 1.3.2.3 ExtrinsicHandler
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **extrinsic** | *String* | An identifier for a function that will be handled in the mapping script in the form `<section>.<method>(name1: type1, name2: type2,...,)`. Example: `utility.batch(calls: Vec<Call>)`|
+| **filter** | optional *String* | An open_CRUD string specifying additional filtering to be applied, i.e. `calls[0].args.max_additional_gt > 10000`. The detailed syntax to be documented elsewhere and is subject to change.  |
+| **handler** | *String* | The name of an exported function in the mapping script that should handle the specified event. |
+| **emits** | list of *String* | A list of virtual events the handler _may_ emit |
+| **exports** | list of *String* | Extra types exported by the handler |
+
+#### 1.3.2.4 BlockHandler
+
+| Field | Type | Description |
+| --- | --- | --- |
+| **handler** | *String* | The name of an exported function in the mapping script that should handle the specified event. |
+| **filter** | optional *String* | The name of the filter that will be applied to decide on which blocks will trigger the mapping. If none is supplied, the handler will be called on every block. The detailed syntax to be documented elsewhere and is subject to change. |
+
+## Compatibility
+
+The changes are not compatible with Hydra v0.

docs/rfc/type-safe-mappings.md

@@ -0,0 +1,88 @@
+# Strongly typed mappings
+
+## Summary
+
+The mappings define how the processor should handle the events. This proposal describes how the event can be auto-generated to make it strongly typed.
+
+## Goals and motivation
+
+1) Manual type conversion from `SubstrateEvent` currently passed to the mappings is error-prone
+
+2) Runtime changes and deserialization issues are easy to spot
+
+## Urgency
+
+Subjectively, strong typings is a must if a large collection of mappings is going to be maintained for a long time.
+
+## Detailed Design
+
+The codegen will perform the following steps:
+
+1) Define all (event, extrinsic) pairs that are going to be handled according to the manifest file. Extrinsic may be optional
+2) Lookup the definitions and argument types in the metadata file, both for the event and the extrinsic. If the extrinsic was not specified, leave it as optional arbitrary `AnyJSON`.
+3) Generate classes extending `SubstrateEvent` with `get` methods performing the deserialization and type-casting. The classes are created using [TypeRegistry](https://github.com/polkadot-js/api/blob/master/packages/types/src/types/registry.ts) interface from `@polkadot/api`. By default, it contains all the base substrate type definitions but my be extended with custom types. This assumes that the required types can be created by name via `typeRegistry.createType(type, value)`. `typeRegistry` will be declared in `@dzlzv/hydra-common` and must be defined and exported in the processor runner.
+4) The types and interfaces must be available for import from `./types.ts`. The standard polkadot types should be re-exported together with additional custom and manually defined types and interfaces.
+5) The property names will be derived using the `name` property (if present), otherwise by lowercasing the type. If there are several properties of the same type, prefix with a number.
+
+Example of the strongly typed event and the autogenerated properties:
+
+```typescript
+// typeRegistry is declared there but must be defined elsewhere
+import { typeRegistry, SubstrateEvent } from '@dzlzv/hydra-common'
+// types.ts must reexport these types from @polkadot/api/intefaces
+import { AccountId, Balance, Compact } from '../types'
+
+export class BalancesBalanceSetEvent extends SubstrateEvent {
+
+    get params(): BalanceSet__Params {
+        return new BalanceSet__Params(this)
+    }  
+
+    get callArgs(): BalanceSet__CallArgs {
+        return new BalanceSet__CallArgs(this)
+    }
+}
+
+export class BalanceSet__Params {
+  _event: BalancesBalanceSetEvent;
+
+  constructor(event: BalancesBalanceSetEvent) {
+    this._event = event;
+  }
+
+  get accountId(): AccountId {
+    return typeRegistry.createType('AccoundId', this._event.params[0])
+  }
+
+  get balance0(): Balance {
+    return typeRegistry.createType('Balance', this._event.params[1])
+  }
+
+  get balance1(): Balance {
+    return typeRegistry.createType('Balance', this._event.params[2])
+  }
+}
+
+export class BalanceSet__CallArgs {
+  _event: BalancesBalanceSetEvent;
+
+  constructor(event: BalancesBalanceSetEvent) {
+    this._event = event;
+    if (this._event.extrinsic === undefined) {
+       throw new Error(`Extrinsic balances_set is expected`)
+    }
+  }
+
+  get who(): AccountId {
+    return typeRegistry.createType('LookupSource', this._event.extrinsic?.args[0])
+  }
+
+  get new_free(): Compact<Balance> {
+    return typeRegistry.createType('Compact<Balance>', this._event.extrinsic?.args[1])
+  }
+
+  get new_reserved(): Compact<Balance> {
+    return typeRegistry.createType('Compact<Balance>', this._event.extrinsic?.args[2])
+  }
+}
+```

docs/rfc/virtual-events.md

@@ -0,0 +1,105 @@
+# Virtual events
+
+## Summary
+
+Virtual events are similar to chain events but emitted by the processor itself. This allows to significantly extend the scope and potential use-cases for the mappings.
+
+## Goals and motivation
+
+There are multiple cases where relying only on the data provided by substrate events is not feasible.
+
+1) The extrinsic does not throw a relevant event to capture the data, e.g. during a `sudoAs` or `batchCall`.
+Example: The project RMRK relies on `system.remark` calls. There is no specific event that'd efficiently capture the required data.
+
+2) The event data requires additional transformation before it can be processed. For example, EVM runtime module would only emit `Log` events with low-level data. An intermediary extrinsic handler would listen to such low-level extrinsic calls, decode the data end emit virtual events with strongly typed data. The goal here is to make seamless migration of the graph mappings used for Ethereum to smart contracts deployed on Moonbeam or EVM palette.
+
+3) Allow reusable extrinsic handlers and events imported from external public libraries.
+
+4) At the first step, only extrinsicHandlers are allowed to emit virtual events. A logical extension would be to allow event handlers to emit
+virtual events as well, thus making it possible to have complex pipelines of arbitrary complexity.
+
+## Urgency
+
+Unless no other alternatives are suggested, this is needed to go forward with mappings for EVM smart contracts.
+
+## Detailed design
+
+Virtual events are placed in the processing queue straight after the `ExtrinsicSuccess` event and before the next chain event.
+
+For now, virtual events can be emitted only by extrinsic handlers. The virtual events emitted by the extrinsic handlers should be explicitly defined in the manifest file:
+
+```yml
+...
+eventHandlers:
+  - event: remarkCreated(RemarkData)
+    handler: handleRemarkCreated
+    virtual: true
+extrinsicHandlers:
+  - extrinsic: system.remark(remark: Bytes)
+    handler: handleRemarks
+    emits:
+        - remarkCreated(RemarkData)
+    exports:
+        - RemarkData
+```
+
+The handler will like this
+
+```typescript
+
+// mappings/handleRemarkCreated.ts
+import { RemarkData } from './handleRemarkCreated'
+
+export handleRemarkCreated(remarkData: RemarkData) {
+   // some business logic with remarkData
+}
+
+// mappings/handleRemarks.ts
+
+import { emit, EventData } from '@dzlzv/hydra-processor'
+import { RemarkExtrinsic } from './generated/RemarkExtrinsic'
+
+// some custom interface to be emitted along with the data
+export interface RemarkData extends EventData {
+   name: String;
+   url: String;
+}
+
+export handleBatchCalls(extrinsic: RemarkExtrinsic) {
+   const data: Bytes = extrinsic.args._remark
+   // parse and transform the raw data
+   const remarkData = parse(...)
+   emit('remarkCreated', remarkData)
+}
+
+/// Auto-generated in ./generared/RemarkExtrinsic
+
+// typeRegistry is declared there but must be defined elsewhere
+import { typeRegistry, SubtrateExtrinsic } from '@dzlzv/hydra-common'
+// types.ts must reexport these types from @polkadot/api/intefaces
+import { Bytes } from '../types'
+
+export class RemarkExtrinsic extends SubstrateExtrinsic {
+    get args(): RemarkExtrinsic__Args {
+        return new RemarkExtrinsic__Args(this)
+    }  
+}
+
+export class RemarkExtrinsic__Args {
+  _extrinsic: SubstrateExtrinsic;
+
+  constructor(extrinsic: SubstrateExtrinsic) {
+    this._extrinsic = extrinsic;
+  }
+
+  get _remark(): Bytes {
+    return typeRegistry.createType('Bytes', this._extrinsic.args[0].value)
+  }
+
+}
+
+```
+
+## Compatibility
+
+Not compatible with the previous Hydra versions.