Feature request: Containerized secret server for more accessibility

[mumoshu]

Problem

Currently, Habitus' secret server runs on the node on which habitus command runs.

This becomes a problem when:

• Your node on which habitus runs is behind a kind of NAT and
• Your docker daemon is running remotely

and hence the secret server is inaccessible from builds.

I believe this is a common situation seen in recent container-native CI systems like e.g. CircleCI 2.0 w/ its container executor, which you have to rely on remote docker daemons due to the gotcha that those container executor doesn't give us root access, while rootless docker build is still a WIP project in our container landscape.

Suggested solution

Similarly to what is shown in #94, we can theoretically run a containerized secret server within the same docker network the docker build is run.
With this setup, the above problem disappears, hence more use-cases habitus supports.

Implementation notes

As we can see from the current implementation of habitus' api server(which I call "secrets server"), the sever must have both envvars and files it serves inside the container. Envvars is not a problem as it can passed at runtime of the container.

But for secret files, we have two choices: one is to use a host volume mount, one is to build a temporary docker image for the secret server on demand. The latter is slightly complicated but a general solution, because it doesn't require host volume mounts. The docker daemon can be running locally or remotely. When it is running remotely, host volume mount is impossible by its nature.

Therefore, If I were to implement this feature, I'd rather make habitus build a temporary secret server image from secrets: definitions in build.yml, and runs a temporary container from it, and then docker builds, and finally destroy the container to minimize the risk of leaking secrets.