Move replicator docs to replicator repository

.github/dependabot.yml

@@ -0,0 +1,14 @@
+version: 2
+updates:
+- package-ecosystem: npm
+  directory: "/docs"
+  schedule:
+    interval: daily
+    time: '20:00'
+  open-pull-requests-limit: 10
+- package-ecosystem: bundler
+  directory: "/docs"
+  schedule:
+    interval: daily
+    time: '20:00'
+  open-pull-requests-limit: 10

.github/workflows/docs-build.yml

@@ -0,0 +1,42 @@
+name: Docs / Build Hugo Site
+on:
+  push:
+    branches:
+      - 'master'
+  pull_request:
+    paths-ignore:
+      - "**.md"
+      - ".github/**"
+      - ".gitignore"
+      - ".gitattributes"
+    branches:
+      - 'master'
+jobs:
+  hugo:
+    name: Build Hugo Site
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash
+        working-directory: ./docs
+    steps:
+      - 
+        name: Checkout
+        uses: actions/checkout@v3
+      - 
+        name: Setup Hugo
+        uses: peaceiris/actions-hugo@v2
+        with:
+          hugo-version: 'latest'
+          extended: true
+      -
+        name: Install Node
+        uses: actions/setup-node@v4
+        with:
+            node-version: 19.9.0
+      -
+        name: Install Dependencies
+        run: npm install
+      - 
+        name: Build Hugo Site
+        run: hugo --minify

.github/workflows/build-docker.yml → .github/workflows/replicator-build.yml

@@ -1,4 +1,4 @@
-name: Build Docker Image
+name: Replicator / Build Docker Image
 on:
   push:
     paths:

.github/workflows/publish-docker.yml → .github/workflows/replicator-publish-docker.yml

@@ -1,4 +1,4 @@
-name: Publish Docker Image
+name: Replicator / Publish Docker Image
 on:
   push:
     paths:

.github/workflows/helm.yml → .github/workflows/replicator-publish-helm.yml

@@ -1,4 +1,4 @@
-name: Publish Helm Chart
+name: Replicator / Publish Helm Chart
 on:
   push:
     paths:

.gitignore

@@ -331,4 +331,11 @@ exclusive.lck
 .vscode
 version.txt
 /public/
-**/checkpoint
+**/checkpoint
+
+# Hugo Docs
+docs/public/
+docs/resources/
+docs/node_modules/
+docs/tech-doc-hugo
+docs/.DS_Store

README.md

@@ -16,13 +16,13 @@ EventStoreDB readers implement one additional filter, which executes during the
 
 ## Build
 
-On a standard x64 architecture:
-
 ```sh
 docker build .
 ```
 
-On an arm64 / Mac apple silicium:
+The default target architecture is amd64 (x86_64).
+
+You can build targeting arm64 (e.g to execute on Apple Silicon) like so:
 
 ```sh
 docker build --build-arg RUNTIME=linux-arm64 .

docs/assets/scss/_variables_project.scss

@@ -0,0 +1,6 @@
+/*
+
+Add styles or override variables from the theme here.
+
+*/
+

docs/content/_index.html

@@ -0,0 +1,38 @@
++++
+title = "Replicator"
+linkTitle = "Event Store Replicator"
++++
+
+{{< blocks/cover title="Event Store Replicator" image_anchor="top" height="max" color="primary" >}}
+<div class="mx-auto">
+	<a class="btn btn-lg btn-primary mr-3 mb-4" href="{{< relref "/docs" >}}">
+		Learn More <i class="fas fa-arrow-alt-circle-right ml-2"></i>
+	</a>
+	<a class="btn btn-lg btn-secondary mr-3 mb-4" href="{{< ref "helm" >}}">
+		Quick Start <i class="fas fa-rocket ml-2"></i>
+	</a>
+	<p class="lead mt-5">Replicate your events with ease</p>
+</div>
+{{< /blocks/cover >}}
+
+{{% blocks/lead color="primary" %}}
+Migrate from EventStoreDB on-prem to Event Store Cloud by replicating events.
+Keep your data safe by replicating events to a backup cluster.
+One-time or continuous replication, filtering events, schema and data transformations.
+{{% /blocks/lead %}}
+
+{{< blocks/section color="dark" type="row" >}}
+{{% blocks/feature icon="fas fa-plug" title="Multi-protocol" %}}
+Read and write to EventStoreDB v5 and v20+, using TCP or gRPC protocol.
+Both secure and unsecure connections are supported.
+{{% /blocks/feature %}}
+
+{{% blocks/feature icon="fas fa-filter" title="Filters" url="Filters" %}}
+Filter out obsolete and deleted events with scavenge, stream name, and event type filter. You can even use transformations for advanced filtering!
+{{% /blocks/feature %}}
+
+{{% blocks/feature icon="fas fa-random" title="Transformations" url="Transforms" %}}
+Upgrade your event schema, remove or add fields, change values, enrich events with information from external sources.
+{{% /blocks/feature %}}
+
+{{< /blocks/section >}}

docs/content/docs/Concepts/_index.md

@@ -0,0 +1,48 @@
+---
+title: "Concepts"
+linkTitle: "Concepts"
+date: 2021-03-05
+weight: 2
+description: >
+  Find out some basic concepts of Replicator.
+---
+
+Replicator is designed to copy events from one place to another, which sounds like a relatively simple task. Still, there are some concepts you need to understand before using the tool.
+
+## Reader
+
+Reader is an adapter for the infrastructure, where you want to copy events _from_. The _reader_ reads from a _source_.
+
+Currently, we support readers for EventStoreDB, using TCP and gRPC protocols. Each reader type requires its own configuration, which is usually just a connection string, specific to each reader type.
+
+The reader always reads events in sequence, but all the readers support batched reads.
+
+There is only one reader per running Replicator instance.
+
+## Sink and writers
+
+Reader is an adapter for the infrastructure, where you want to copy events _to_. The _sink_ has one or more _writers_. By using multiple writers, one sink can improve performance by parallelising writes.
+
+When using one writer for a sink, the order of events in the target remains exactly the same as it was in the source.
+
+When using more than one writer, the global order of events in the source cannot be guaranteed. However, multiple writers also enable partitioning. The default partition key is the stream name, which guarantees the order of events in each stream.
+
+You can only have one sink per running Replicator instance, but it might have multiple writers.
+
+## Checkpoint
+
+A running Replicator instance progresses linearly over a given stream of events, so it knows at any time, which events were already processed. As the process might be shut down for different reasons, it needs to maintain the last processed event position, so in case of restart, Replicator will start from there, and not from the very beginning. This way, you don't get duplicated events in the sink, and you can be sure that the replication process will eventually be completed.
+
+The location of the last processed event in the source is known as _checkpoint_. Replicator supports storing the checkpoint in [different stores]({{< ref "checkpoints" >}}). If you want to run the replication again, from the same source, using the same Replicator instance, you need to delete the checkpoint file.
+
+## Filters
+
+As you might want to ignore some events during replication, Replicator supports different [filters]({{< ref "filters" >}}). Filters allow you to cover cases like preventing some obsolete events from being replicated, or splitting one source to two targets. In the latter case, you can run two Replicator instances with different filters, so events will be distributed to different sinks.
+
+## Transforms
+
+After being in production for a while, most systems accumulate legacy data. You might want to remove some of it using filters, but you might also want to keep the data in a different format. Typical scenarios include evolution of event schema, missing fields, incorrect data format, oversharing (sensitive unprotected information), etc.
+
+These cases can be handler by using [transforms]({{< ref "transforms" >}}), which allow you to change any part of the event that comes from the source, before writing it to the sink.
+
+

docs/content/docs/Configuration/_index.md

@@ -0,0 +1,62 @@
+---
+title: "Configuration"
+linkTitle: "Configuration"
+date: 2021-03-05
+weight: 5
+description: >
+  Replicator configuration file explained.
+---
+
+Replicator uses a configuration file in YAML format. The file must be called `appsettings.yaml` and located in the `config` subdirectory, relative to the tool working directory.
+
+The settings file has the `replicator` root level, all settings are children to that root. It allows using the same format for the values override file when using Helm.
+
+Available configuration options are:
+
+| Option | Description |
+| :----- | :---------- |
+| `replicator.reader.connectionString` | Connection string for the source cluster or instance |
+| `replicator.reader.protocol` | Reader protocol (`tcp` or `grpc`) |
+| `replicator.reader.pageSize` | Reader page size (only applicable for TCP protocol |
+| `replicator.sink.connectionString` | Connection string for the target cluster or instance |
+| `replicator.sink.protocol` | Writer protocol (`tcp` or `grpc`) |
+| `replicator.sink.partitionCount` | Number of [partitioned]({{% ref "writers" %}}) concurrent writers |
+| `replicator.sink.partitioner` | Custom JavaScript [partitioner]({{% ref "writers" %}}) |
+| `replicator.sink.bufferSize` | Size of the sink buffer, `1000` events by default |
+| `replicator.scavenge` | Enable real-time [scavenge]({{% ref "scavenge" %}}) |
+| `replicator.runContinuously` | Set to `false` if you want Replicator to stop when it reaches the end of `$all` stream. Default is `true`, so the replication continues until you stop it explicitly. |
+| `replicator.filters` | Add one or more of provided [filters]({{% ref "filters" %}}) |
+| `replicator.transform` | Configure the [event transformation]({{% ref "Transforms" %}}) |
+| `replicator.transform.bufferSize` | Size of the prepare buffer (filtering and transformations), `1000` events by default |
+
+## Enable verbose logging
+
+You can enable debug-level logging by setting the `REPLICATOR_DEBUG` environment variable to any value.
+
+## Example configuration
+
+The following example configuration will instruct Replicator to read all the events from a local cluster with three nodes (`es1.acme.org`, `es2.acme.org` and `es3.acme.org`) using TCP protocol, and copy them over to the Event Store Cloud cluster with cluster ID `c2etr1lo9aeu6ojco781` using gRPC protocol. Replicator will also call an HTTP transformation function at `https://my.acme.org/transform`.
+
+The global order of events will be the same, as `partitionCount` is set to one.
+
+Scavenge filter is disabled, so Replicator will also copy deleted events, which haven't been scavenged by the server yet.
+
+```yaml
+replicator:
+  reader:
+    protocol: tcp
+    connectionString: "GossipSeeds=es1.acme.org:2113,es2.acme.org:2113,es3.acme.org:2113; HeartBeatTimeout=500; DefaultUserCredentials=admin:changeit; UseSslConnection=false;"
+    pageSize: 2048
+  sink:
+    protocol: grpc
+    connectionString: "esdb://admin:[email protected]:2113"
+    partitionCount: 1
+  transform:
+    type: http
+    config: https://my.acme.org/transform
+  scavenge: false
+  filters: []
+  checkpoint:
+    path: "./checkpoint"
+```
+

docs/content/docs/Deployment/Docker/_index.md

@@ -0,0 +1,33 @@
+---
+title: "Run Replicator in Docker"
+linkTitle: "Docker"
+date: 2021-03-05
+description: >
+  Running Replicator with Docker Compose
+---
+
+You can run Replicator using [Docker Compose](https://docs.docker.com/compose/), on any machine, which has Docker installed.
+
+We prepared a complete set of files for this scenario. You find those files in the [Replicator repository](https://github.com/EventStore/replicator/tree/master/compose).
+
+The Compose file includes the following components:
+- Replicator itself
+- Prometheus, pre-configured to scrape Replicator metrics endpoint
+- Grafana, pre-configured to use Prometheus, with the Replicator dashboard included
+
+## Configuration
+
+Before spinning up this setup, you need to change the `replicator.yml` file. Find out about Replicator settings on the [Configuration]({{< ref "Configuration" >}}) page. We included a [sample](https://github.com/EventStore/replicator/blob/02736f6e3dd18e41d5536f26ca4f9497733d5f3f/compose/replicator.yml) configuration file to the repository.
+
+{{< alert >}}The sample configuration file includes a JavaScript transform configuration as an example. It is not suitable for production purposes, so make sure you remove it from your configuration.{{< /alert >}}
+
+The sample configuration enables verbose logging using the `REPLICATOR_DEBUG` environment variable. For production deployments, you should remove it from the configuration.
+
+## Monitoring
+
+When you start all the component using `docker-compose up`, you'd be able to check the Replicator web UI by visiting [http://localhost:5000](http://localhost:5000), as well as Grafana at [http://localhost:3000](http://localhost:3000). Use `admin`/`admin` default credentials for Grafana. The Replicator dashboard is included in the deployment, so you can find it in the [dashboards list](http://localhost:3000/dashboards).
+
+{{< alert >}}Watch out for the replication gap and ensure that it decreases.{{< /alert >}}
+
+![Grafana dashboard](grafana.png)
+

docs/content/docs/Deployment/Kubernetes/_index.md

@@ -0,0 +1,14 @@
+---
+title: "Kubernetes"
+linkTitle: "Kubernetes"
+date: 2021-03-05
+description: >
+  Deploying Replicator to a Kubernetes cluster
+---
+
+You can run Replicator in a Kubernetes cluster in the same cloud as your managed EventStoreDB cloud cluster. The Kubernetes cluster workloads must be able to reach the managed EventStoreDB cluster. Usually, with a proper VPC (or VN) peering between your VPC and Event Store Cloud network, it works without issues.
+
+We provide guidelines about connecting managed Kubernetes clusters:
+- [Google Kubernetes Engine](https://developers.eventstore.com/cloud/use/kubernetes/gke.html)
+- [AWS EKS](https://developers.eventstore.com/cloud/use/kubernetes/eks.html)
+- [Azure AKS](https://developers.eventstore.com/cloud/use/kubernetes/aks.html)