Skip to content

Creating a new package

Jump to bottom
Pål Karlsrud edited this page Aug 7, 2019 · 39 revisions

Before reading this, you should know ...

What is a package?

A package is a collection of files describing how to construct an application. These files describe how the different parts of the application should be created (for instance which webserver to use) and how it is allowed to interact with other applications (ex. if the application should be available to the outside world, which ports it should use etc.).

Technically packages uses an extension of the Helm chart packaging format, which provides a generic way of describing Kubernetes objects.

When installing a package through the appstore, some additional values are automatically filled in. The section Mocking values provide by the appstore UI and API shows which values are provided by the UI and API.

TL;DR

github.com/paalka/helm-skel provides a basic skeleton for creating new packages, and it should be possible to quickly create a new package by solving the TODO's in the previously referenced repository. It is also recommended to read some of the charts in github.com/uninett/helm-charts, as they show how the charts should be structured.

Creating a new package from scratch

The different elements of a package is best shown through an example. The following section will describe how to create a package capable of creating a Jupyter notebook.

Usually packages have the structure seen below

File structure

jupyter/
    Chart.yaml              # metadata describing the package/chart
    README.md               # (Optional) providees general information about the chart
    resources.yaml          # (Optional) describes which third-party resources this package supports
    values.yaml             # default configuration values
    templates/              # (will be described below)
    templates/_helpers.tpl

If you are following this example from home, you are should create a similar file tree now.

Defining package metadata (Chart.yaml)

Some metadata is required for a package to function correctly. A file called Chart.yaml is used to store this metadata. TheChart.yaml that will be used in the Jupyter package is shown below.

# filename: Chart.yaml

apiVersion: v1                     # chart API version, always "v1"
name: jupyter                      # package name
description: Jupyter               # a short sentence describing the package.
version: 1.0.0                     # package version (in SemVer 2 format)
maintainers:
  - name: My Name
    email: [email protected]
home: https://package.com          # package homepage
icon: https://package.com/pic.png  # package icon

User-friendly package information (README.md)

In order to make the package more user-friendly, a README.md is recommended. This file provides general information about the package, such as what application the package creates and how to use it.

The README.md for our Jupyter notebook may look like the following file:

# Jupyter notebook

[Jupyter Notebook](http://jupyter.org/) is an open-source web application that
allows you to create and share documents that contain live code, equations,
visualizations and narrative text. Uses include: data cleaning and transformation,
numerical simulation, statistical modeling, data visualization, machine learning, and much more.

Providing default user input (values.yaml)

We want the user to be able to specify certain parts of the application configuration. A package contains a values.yaml file that contains default values that can be overridden by the user.

When creating the Jupyter notebook, we want the user to be able to allocate a custom amount of resources, as well as specifiying which host the application will be using.

The following values.yaml file contains values that does this

# filename: values.yaml

ingress:
  host: "local-chart.example.com"
resources:
  requests:
    cpu: 100m
    memory: 512Mi
  limits:
    cpu: 300m
    memory: 1Gi
dockerImage: quay.io/uninett/jupyterlab:20180501-6469a2f

So, if specifies that the host should be foo.bar.interwebz.cat when installing the application, the value of ingress.host: "local-chart.example.com" will be replaced with the user input. So, all the values in this file is optional, and a user does not specify a value, the value present in values.yaml will be used as defaults.

when later creating the Kubernetes object templates, these values can be referenced like this {{ .Values.ingress.host }}.

Creating Kubernetes objects (templates/)

As we want make it possible for the user to specify how certain aspects of a application should be created, we need generic /templates/ that can be used to create Kubernetes objects. These templates are stored in the templates/ directory.

Templates are written in Go template syntax with some additional functions added by Helm. See the Helm template guide for more information.

A template is mostly just a definition of a Kubernetes object, but with the ability to insert variables into parts of the template. Below an example of a template is shown

apiVersion: v1
kind: Secret
metadata:
  name: secret
type: Opaque
data:
  psst: {{ .Values.very_secret | quote }} # <-- this uses the Go templating engine to insert some secret data

Given the preceeding values.yaml file

very_secret: "Pink is pretty cool"

the Kubernetes secret above will be rendered as

apiVersion: v1
kind: Secret
metadata:
  name: secret
type: Opaque
data:
  psst: "Pink is pretty cool"

A quick detour to user defined functions (templates/_helpers.tpl)

In order to make the package easier to maintain, it is useful to define functions or commonly used variables.

Below the _helpers.tpl file we will use for the Jupyter notebook is shown.

{{/* filename: templates/_helpers.tpl */}}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes
name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

The template above defines a variable called fullname which ensures that the application name is valid in Kubernetes. Another variable called "oidcconfig" is also defined, which contains the JSON config required to use a component we will create soon. To use this variable, the following syntax is used {{ template "fullname" . }}.

Knowing that we can use functions and variables from templates/_helpers.tpl, we can continue creating our regular templates. Our Jupyter notebook will consist of the following components:

  1. The Jupyter notebook
  2. An ingress which exposes the application to the outside world
  3. A proxy which provides Dataporten authentication

We will begin by creating the Jupyter notebook. In order to make the application easy to manage and scale, we will use a Kubernetes Deployment object as shown below.

# filename: templates/jupyter-deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "fullname" . }}
  labels:
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: {{ template "fullname" . }}
    spec:
      containers:
      - name: jupyter
        image: {{ .Values.dockerImage }}
        resources:
{{ toYaml .Values.resources | indent 10 }}
        ports:
        - containerPort: 8888

this file creates a Kubernetes deployment using an image provided by the user (through the dockerImage value in values.yaml) and exposes it on port 8888.

Next, we want to create the ingress exposing the application to the outside world. To do so, we first need a service which exposes the deployment to the cluster

# filename: templates/jupyter-svc.yaml

apiVersion: v1
kind: Service
metadata:
  name: {{ template "fullname" . }}
  labels:
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
spec:
  ports:
  - port: 8888
    targetPort: 8888
    protocol: TCP
    name: {{ template "fullname" . }}-service
  selector:
    app: {{ template "fullname" . }}

then, we can create the ingress as follows

# filename: templates/jupyter-ing.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme-staging: "true"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
spec:
  tls:
    - secretName: {{ template "fullname" . }}-tls
      hosts:
         - {{ .Values.ingress.host }}
  rules:
    - host: {{ .Values.ingress.host }}
      http:
        paths:
          - path: /
            backend:
              serviceName: {{ template "fullname" . }}
              servicePort: 8888

As the application by default is isolated from all the others in the clusters, we now need to create a NetworkPolicy allowing traffic between the notebook and the ingress.

# filename: templates/jupyter-network-policy.yaml

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  name: {{ template "fullname" . }}
spec:
  podSelector:
    matchLabels:
      app: {{ template "fullname" . }}
  ingress:
    - from:
      - namespaceSelector:
          matchLabels:
            name: kube-ingress
      ports:
        - protocol: TCP
          port: 8888

If you have copied the YAML files from this tutorial, you should now have a filetree similar to this:

jupyter/
    Chart.yaml
    README.md
    values.yaml
    templates/
        _helpers.tpl
        jupyter-deploy.yaml
        jupyter-svc.yaml
        jupyter-ing.yaml
        jupyter-network-policy.yaml

these templates is sufficient to create a Jupyter notebook hosted at the value given to ingress.host in values.yaml. If you have Helm installed, you should now be able to install this package by following the instructions in the installing a package section.

Limiting access to authorized users

Currently our application does not require login, so the next step is to ensure that only authorized users are allowed to access the application. We will use goidc-proxy to ensure that only authenticated users are allowed to use the application.

First, we will modify templates/jupyter-deploy.yaml so that the proxy runs in the same pod as jupyter.

# filename: template/jupyter-deploy.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
spec:
  replicas: 1
  template:
    metadata:
      annotations:
        checksum/config: {{ include "oidcconfig" . | sha256sum }}
      labels:
        app: {{ template "fullname" . }}
    spec:
      volumes:
        - name: oidcconfig
          secret:
            secretName: {{ template "fullname" . }}
      containers:
      - name: auth-proxy
        image: quay.io/uninett/goidc-proxy:v1.1.2
        imagePullPolicy: Always
        ports:
          - containerPort: 8888
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8888
          initialDelaySeconds: 30
          timeoutSeconds: 30
        volumeMounts:
          - name: oidcconfig
            mountPath: /conf
        workingDir: /conf
      - name: jupyter
        image: {{ .Values.dockerImage }}
        resources:
{{ toYaml .Values.resources | indent 10 }}
        ports:
        - containerPort: 8889

a new container called auth-proxy was added in the deployment specification above. This container will mount a Kubernetes secret containing a specification of how the proxy should be configured. The template used to create this secret is shown below

# filename: templates/jupyter-secret.yaml

apiVersion: v1
kind: Secret
metadata:
  name: {{ template "fullname" . }}
  labels:
    app: {{ template "fullname" . }}
    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  goidc.json: {{ include "oidcconfig" . | b64enc }}

The secret, among other things, includes a variable called oidcconfig from our templates/_helpers.tpl file. Our new templates/_helpers.tpl is reproduced below.

{{/* filename: templates/_helpers.tpl */}}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes
name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{- define "oidcconfig" -}}
{
  "proxy": {
    "target": "http://localhost:8889"
  },
  "engine": {
    "client_id": "<client id here>",
    "client_secret": "<client secret here>",
    "issuer_url": "https://auth.dataporten.no",
    "redirect_url": "https://{{ .Values.ingress.host }}/oauth2/callback",
    "scopes": "<scopes>",
    "signkey": "{{ randAlphaNum 60 }}",
    "groups_endpoint": "https://groups-api.dataporten.no/groups/me/groups",
    "authorized_principals": "<authorized principals here>",
  },
  "server": {
    "port": 8888,
    "health_port": 1337,
    "readtimeout": 10,
    "writetimeout": 20,
    "idletimeout": 120,
    "ssl": false,
    "secure_cookie": false
  }
}
{{- end -}}

Improving user-friendliness (resources.yaml)

Notice that the user would have to manually specify the client ID and secret. If the package is installed through the appstore it is however possible to fill these automatically using third-party resources, thus making the installation process somewhat more user-friendly.

The resources.yaml file makes it possible to specify that a package follows a certain expectations on how values are passed to the package. We will use the dataporten third-party resource in order to automatically register a new dataporten client when installing the package. Begin by specifying that the package will use the dataporten resource.

# filename: resources.yaml

dataporten:
  Options:
    ScopesRequested: # specify the required oauth scopes
      - profile
      - openid
      - groups
    RedirectURI: 
       # specify the URL to redirect back to after log-in
       # ingress.host will be used as the prefix.
      - /oauth2/callback

After using a resource, the values that was generated by the resources are passed using the key appstore_generated_data and the name of the resource as the subkey. Thus, values generated by the dataporten resource will be passed to values.yaml as appstore_generated_data.dataporten. Since we want to use these values in the previously created secret, we need to add them to values.yaml.

Our new values.yaml will thus be

# filename: values.yaml

ingress:
  host: "local-chart.example.com"
resources:
  requests:
    cpu: 100m
    memory: 512Mi
  limits:
    cpu: 300m
    memory: 1Gi
dockerImage: quay.io/uninett/jupyterlab:20180501-6469a2f

appstore_generated_data:
  dataporten:
    scopes:
      - "scope"
    id: "0000-default-id"
    client_secret: "0000-not-very-secret"
    authorized_groups:
      - ""

and then making sure that these values are inserted into the "oidcconfig"

{{/* filename: templates/_helpers.tpl */}}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes
name fields are limited to this (by the DNS naming spec).
*/}}
{{- define "fullname" -}}
{{- $name := default .Chart.Name -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}

{{- define "oidcconfig" -}}
{
  "proxy": {
    "target": "http://localhost:8889"
  },
  "engine": {
    "client_id": "{{ .Values.appstore_generated_data.dataporten.id }}",
    "client_secret": "{{ .Values.appstore_generated_data.dataporten.client_secret }}",
    "issuer_url": "https://auth.dataporten.no",
    "redirect_url": "https://{{ .Values.ingress.host }}/oauth2/callback",
    "scopes": "{{- join "," .Values.appstore_generated_data.dataporten.scopes -}}",
    "signkey": "{{ randAlphaNum 60 }}",
    "groups_endpoint": "https://groups-api.dataporten.no/groups/me/groups",
    "authorized_principals": "{{- join "," .Values.appstore_generated_data.dataporten.authorized_groups -}}"
  },
  "server": {
    "port": 8888,
    "health_port": 1337,
    "readtimeout": 10,
    "writetimeout": 20,
    "idletimeout": 120,
    "ssl": false,
    "secure_cookie": false
  }
}
{{- end -}}

Thus, when using this package with the appstore, values under the appstore_generated_data.dataporten key will automatically be filled. When using helm manually, you will have to fill these (in values.yaml) on your own.

Installing a package

The helm client uses tiller to install a package, thus we first need a way of accessing tiller. One option is to run tiller locally in the following way

TILLER_NAMESPACE="<default namespace here>" tiller

If you have access to a Kubernetes cluster where tiller is running, you can also forward tiller's port to your local machine

kubectl port-forward <tiller-pod-name> 44134:44134 -n <tiller namespace>

which will forward port 44134 of the pod with name to your local port 44134.

When tiller is available, you can install a package by running

HELM_HOST="<tiller-local-ip:tiller-local-port>" \
helm install <package name or directory> --namespace <installation namespace>

ex.

HELM_HOST="127.0.0.1:44134" helm install . --namespace scratch

to install the package stored in the current directory in the scratch namespace.

Debugging and testing

Checking packages for errors

In order to see whether the packages is valid, you can use

helm lint --strict <chart directory>

which will notify you of any errors.

This repository (that is, uninett/helm-charts) also contains a script called lint-chart.sh which uses kubeval and kubetest to determine whether the package is valid. This script can be run in the following way: ./lint-chart.sh <chart directory>.

Viewing generated Kubernetes objects

In order to see what the Kubernetes objects will look like after having been passed through the template engine, you can use

helm template <chart directory>

which will output all the generated files to stdout.

Mocking values provide by the appstore UI and API

The following YAML file can be used to mock values provided by the appstore UI

# filename: ui-values.yaml

resources:
  requests:
    cpu: 1
    memory: 1G
    gpu: 0
  limits:
    cpu: 2
    memory: 1G
    gpu: 0
persistentStorage:
  - existingClaim: ""
    existingClaimName: ""
    subPath: ""

uid: 999
gid: 999
supplementalGroups:
  - name: "foo"
    gid: "999"
username: foo
authGroupProviders:
  - url: "https://groups-api.dataporten.no/groups/me/groups"
    scope: groups
userInfoURL: "https://auth.dataporten.no/openid/userinfo"

and the following file can be used to mock values provided by the appstore API

# filename: api-values.yaml
# Note that you have to fill the Dataporten values on your own

appstore_generated_data:
  appstore_meta_data:
    contact_email: "[email protected]"
  dataporten:
    scopes:
      - "required-scope"
      - "profile"
    id: "your-dataporten-client-id-here"
    owner: "dataporten-user-id-here"
    client_secret: "your-dataporten-client-secret-here"
    authorized_groups:
      - ""

You can use these files by using the -f flag when installing or rendering templates.

Thus, you will be able to perform a test installation by runninghelm install jupyter -f ui-values.yaml,api-values.yaml, which will attempt to install Jupyter using the combined values from the charts default values.yaml file, as well as ui-values.yaml and api-values.yaml.

Clone this wiki locally