Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Editorial pass over README #177

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Editorial pass over README #177

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 68 additions & 54 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,24 +1,25 @@
# nomad-driver-containerd

[![CI Actions Status](https://github.com/Roblox/nomad-driver-containerd/workflows/CI/badge.svg)](https://github.com/Roblox/nomad-driver-containerd/actions)
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/Roblox/nomad-driver-containerd/blob/master/LICENSE)
[![Release](https://img.shields.io/badge/version-0.9.3-blue)](https://github.com/Roblox/nomad-driver-containerd/releases/tag/v0.9.3)
[![Release](https://img.shields.io/github/release/Roblox/nomad-driver-containerd.svg)](https://github.com/Roblox/nomad-driver-containerd/releases/latest)
[![Docs](https://img.shields.io/badge/docs-website-green.svg)](https://www.nomadproject.io/docs/drivers/external/containerd)

<img src="images/nomad.png" width="232" height="100" />&nbsp;<img src="images/docker.png" width="116" height="100" />&nbsp;<img src="images/containerd.png" width="285" height="100" />

We are actively looking for contributors and maintainers for this project.
If you have experience in container internals e.g. cgroups, namespaces, or have contributed to any open source projects
around containers e.g. [`docker`](https://github.com/moby/moby), [`containerd`](https://github.com/containerd/containerd), [`nerdctl`](https://github.com/containerd/nerdctl), [`podman`](https://github.com/containers/podman) etc or build tooling which involves dealing with
around containers e.g. [`docker`](https://github.com/moby/moby), [`containerd`](https://github.com/containerd/containerd), [`nerdctl`](https://github.com/containerd/nerdctl), [`podman`](https://github.com/containers/podman) etc. or build tooling which involves dealing with
container internals, and are interested in contributing to this project, I would love to talk to you!
Golang experience is preferred but not required.

Please reach out to me [`@_shishir_m`](https://twitter.com/_shishir_m) or open an issue in this repository with your contact details, if you are interested in contributing to this project.

## Overview

Nomad task driver for launching containers using containerd.

**Containerd** [`(containerd.io)`](https://containerd.io) is a lightweight container daemon for
running and managing container lifecycle.<br/>
**Containerd** [`(containerd.io)`](https://containerd.io) is a lightweight container daemon for running and managing container lifecycle.<br/>
Docker daemon also uses containerd.

```
Expand All @@ -29,6 +30,7 @@ dockerd (docker daemon) --> containerd --> containerd-shim --> runc
Docker daemon is not required on the host system.

## nomad-driver-containerd architecture

<img src="images/nomad_driver_containerd.png" width="850" height="475" />

## Requirements
Expand All @@ -41,51 +43,52 @@ Docker daemon is not required on the host system.

## Building nomad-driver-containerd

Make sure your **$GOPATH** is setup correctly.
```
$ mkdir -p $GOPATH/src/github.com/Roblox
$ cd $GOPATH/src/github.com/Roblox
Make sure your **$GOPATH** is set up correctly.

```shell
$ git clone [email protected]:Roblox/nomad-driver-containerd.git
$ cd nomad-driver-containerd
$ make build (This will build your containerd-driver binary)
$ make build # This will build your containerd-driver binary
```

If you want to compile for `arm64`, you can run:

```
make -f Makefile.arm64
```shell
$ make -f Makefile.arm64
```

## Screencast

[![asciicast](https://asciinema.org/a/348173.svg)](https://asciinema.org/a/348173)

## Wanna try it out!?

```
```shell
$ vagrant up
```
or `vagrant provision` if the vagrant VM is already running.

Once setup (`vagrant up` OR `vagrant provision`) is complete and the nomad server is up and running, you can check the registered task drivers (which will also show `containerd-driver`) using:
```
$ nomad node status (Note down the <node_id>)
Once setup is complete and the nomad server is up and running, you can check the registered task drivers (which will also show `containerd-driver`) using:
```shell
$ nomad node status # Note down node ID
$ nomad node status <node_id> | grep containerd-driver
```

**NOTE:** [`setup.sh`](vagrant/setup.sh) is part of the vagrant setup and should not be executed directly.

## Run Example jobs.
## Run example jobs

There are few example jobs in the [`example`](https://github.com/Roblox/nomad-driver-containerd/tree/master/example) directory.

```shell
$ nomad job run <job_name>.nomad
```
$ nomad job run <job_name.nomad>
```
will launch the job.<br/>
will launch the job.

More detailed instructions are in the [`example README.md`](https://github.com/Roblox/nomad-driver-containerd/tree/master/example)
More detailed instructions are available in the [`example README.md`](https://github.com/Roblox/nomad-driver-containerd/tree/master/example).

To interact with `images` and `containers` directly, you can use [`nerdctl`](https://github.com/containerd/nerdctl) which is a docker compatible CLI for `containerd`. `nerdctl` is already installed in the vagrant VM at `/usr/local/bin`.
To interact with `images` and `containers` directly, you can use [`nerdctl`](https://github.com/containerd/nerdctl) which is a docker compatible CLI for `containerd`.
`nerdctl` is already installed in the vagrant VM at `/usr/local/bin`.

## Supported options

Expand Down Expand Up @@ -132,11 +135,12 @@ To interact with `images` and `containers` directly, you can use [`nerdctl`](htt
&emsp;&emsp;&emsp;- **type** (string) (Optional): Supported values are `volume`, `bind` or `tmpfs`. **Default:** volume.<br/>
&emsp;&emsp;&emsp;- **target** (string) (Required): Target path in the container.<br/>
&emsp;&emsp;&emsp;- **source** (string) (Optional): Source path on the host.<br/>
&emsp;&emsp;&emsp;- **options** ([]string) (Optional): fstab style [`mount options`](https://github.com/containerd/containerd/blob/master/mount/mount_linux.go#L187-L211). **NOTE**: For bind mounts, atleast `rbind` and `ro` are required.<br/>
&emsp;&emsp;&emsp;- **options** ([]string) (Optional): fstab style [`mount options`](https://github.com/containerd/containerd/blob/master/mount/mount_linux.go#L187-L211). **NOTE**: For bind mounts, at least `rbind` and `ro` are required.<br/>
&emsp;&emsp;\}

**Bind mount example**
```

```hcl
mounts = [
{
type = "bind"
Expand All @@ -147,18 +151,18 @@ mounts = [
]
```

In additon to the `mounts` option in `Task Config`, you can also mount your volumes into the container using nomad [`volume_mount stanza`](https://www.nomadproject.io/docs/job-specification/volume_mount)
In addition to the `mounts` option in `Task Config`, you can also mount your volumes into the container using nomad [`volume_mount stanza`](https://www.nomadproject.io/docs/job-specification/volume_mount)

See [`example job`](https://github.com/Roblox/nomad-driver-containerd/blob/master/example/volume_mount.nomad) for `volume_mount`.

**Custom seccomp profile example**

The default `docker` seccomp profile found [`here`](https://github.com/moby/moby/blob/master/profiles/seccomp/default.json)
can be downloaded, and modified (by removing/adding syscalls) to create a custom seccomp profile.<br/>
The default `docker` seccomp profile found [`here`](https://github.com/moby/moby/blob/master/profiles/seccomp/default.json) can be downloaded and modified (by adding/removing syscalls) to create a custom seccomp profile.
The custom seccomp profile can then be saved under `/opt/seccomp/seccomp.json` on the Nomad client nodes.

A nomad job can be launched using this custom seccomp profile.
```

```hcl
config {
seccomp = true
seccomp_profile = "/opt/seccomp/seccomp.json"
Expand All @@ -167,7 +171,7 @@ config {

**Sysctl example**

```
```hcl
config {
sysctl = {
"net.core.somaxconn" = "16384"
Expand All @@ -183,9 +187,10 @@ an image from a private repository in docker hub.<br/>
`auth` stanza can be set either in `Driver Config` or `Task Config` or both.<br/>
If set at both places, `Task Config` auth will take precedence over `Driver Config` auth.

**NOTE**: In the below example, `user` and `pass` are just placeholder values which need to be replaced by actual `username` and `password`, when specifying the credentials. Below `auth` stanza can be used for both `Driver Config` and `Task Config`.
**NOTE**: In the example below, `user` and `pass` are just placeholder values which need to be replaced by actual `username` and `password`, when specifying the credentials.
`auth` stanza can be used for both `Driver Config` and `Task Config`.

```
```hcl
auth {
username = "user"
password = "pass"
Expand All @@ -198,34 +203,39 @@ auth {

**NOTE:** `host` and `bridge` are mutually exclusive options, and only one of them should be used at a time.

1. **Host** network can be enabled by setting `host_network` to `true` in task config
of the job spec (see under [`Supported options`](https://github.com/Roblox/nomad-driver-containerd#supported-options)).
1. **Host** network can be enabled by setting `host_network` to `true` in task config of the job spec (see under [`Supported options`](#supported-options)).

2. **Bridge** network can be enabled by setting the `network` stanza in the task group section of the job spec.

```
```hcl
network {
mode = "bridge"
}
```

You need to install CNI plugins on Nomad client nodes under `/opt/cni/bin` before you can use `bridge` networks.

**Instructions for installing CNI plugins.**<br/>
```
$ curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
$ sudo mkdir -p /opt/cni/bin
$ sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz
```
Also, ensure your Linux operating system distribution has been configured to allow container traffic through the bridge network to be routed via iptables. These tunables can be set as follows:
**Installing CNI plugins**

[//]: # (TODO: Replace this with containernetworking-plugins Ubuntu package)

```shell
$ curl -L -o cni-plugins.tgz https://github.com/containernetworking/plugins/releases/download/v0.8.6/cni-plugins-linux-amd64-v0.8.6.tgz
$ sudo mkdir -p /opt/cni/bin
$ sudo tar -C /opt/cni/bin -xzf cni-plugins.tgz
```

Also, ensure your Linux operating system distribution has been configured to allow container traffic through the bridge network to be routed via iptables.
These tunables can be set as follows:

```shell
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-arptables
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables
$ echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables
```
To preserve these settings on startup of a nomad client node, add a file including the following to `/etc/sysctl.d/` or remove the file your Linux distribution puts in that directory.

```
```toml
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
Expand All @@ -235,10 +245,10 @@ net.bridge.bridge-nf-call-iptables = 1

nomad supports both **static** and **dynamic** port mapping.

1. **Static ports**
### Static ports

Static port mapping can be added in the `network` stanza.
```
```hcl
network {
mode = "bridge"
port "lb" {
Expand All @@ -250,10 +260,10 @@ network {
Here, `host` port `8889` is mapped to `container` port `8889`.<br/>
**NOTE**: static ports are usually not recommended, except for `system` or specialized jobs like load balancers.

2. **Dynamic ports**
### Dynamic ports

Dynamic port mapping is also enabled in the `network` stanza.
```
```hcl
network {
mode = "bridge"
port "http" {
Expand All @@ -263,11 +273,13 @@ network {
```
Here, nomad will allocate a dynamic port on the `host` and that port will be mapped to `8080` in the container.

You can also read more about `network stanza` in the [`nomad official documentation`](https://www.nomadproject.io/docs/job-specification/network)
You can also read more about `network stanza` in the [`nomad official documentation`](https://www.nomadproject.io/docs/job-specification/network).

## Service discovery

Nomad schedules workloads of various types across a cluster of generic hosts. Because of this, placement is not known in advance and you will need to use service discovery to connect tasks to other services deployed across your cluster. Nomad integrates with Consul to provide service discovery and monitoring.
Nomad schedules workloads of various types across a cluster of generic hosts.
Because of this, placement is not known in advance, and you will need to use service discovery to connect tasks to other services deployed across your cluster.
Nomad integrates with Consul to provide service discovery and monitoring.

A [`service`](https://www.nomadproject.io/docs/job-specification/service) stanza can be added to your job spec, to enable service discovery.

Expand All @@ -277,34 +289,36 @@ The service stanza instructs Nomad to register a service with Consul.

If you are running the tests locally, use the [`vagrant VM`](Vagrantfile) provided in the repository.

```
```shell
$ vagrant up
$ vagrant ssh containerd-linux
$ sudo make test
```
**NOTE**: These are destructive tests and can leave the system in a changed state.<br/>
It is highly recommended to run these tests either as part of a CI/CD system e.g. circleci or on
a immutable infrastructure e.g vagrant VMs.
an immutable infrastructure e.g. vagrant VMs.

You can also run an individual test by specifying the test name. e.g.

```
```shell
$ cd tests
$ sudo ./run_tests.sh 001-test-redis.sh
```

## Cleanup
```
make clean

```shell
$ make clean
```
This will delete your binary: `containerd-driver`

```
vagrant destroy
```shell
$ vagrant destroy
```
This will destroy your vagrant VM.

## Currently supported environments

Ubuntu (>= 16.04)

## License
Expand Down
Loading