Enhancements to Git integration

[deviantony]

I'm opening this ticket to discuss and track update on the Git integration within Portainer in general.

We originally added support to deploy a stack from Git a few years back now and we recently turned our focus on being more Git/GitOps centric when it comes to workload deployment.

As such, we've recently added another highly requested feature with the ability to perform a manual git pull and redeploy: #1753

Now the item above has stirred up discussion about a lot of use cases and potential enhancements to that "deploy from Git" capability in Portainer at the moment and we'll be using the state of this capability as it will be released in CE-2.6.0 as our basis for future enhancements.

Open for discussion

Here are the ideas/areas/enhancements we're currently thinking about that are open for discussions:

• Persisting git credentials to provide a better UX when using the manual redeploy from Git action
• Have a proper Git provider integration to support authenticating against a provider and listing repositories to provide an enhanced deployment UX (think login with Github, click on my repository, select a branch and a Compose file, deploy...)
• The ability to push changes up when editing a stack within Portainer
• The ability to use the prune action against a stack that was deployed from Git
• Add support for relative path when deploying a workload from Git - Failed to mount file on new stack creation #2046
• Persist git credentials in secrets (Swarm/Kubernetes native secrets or integration with secret providers such as Vault)
• Support SSH authentication to deploy/redeploy a stack from a git repository (original discussion in Deploy a stack from a git url with ssh auth #1752)

Here are the ideas/areas/enhancements we're currently looking at or working on:

• Support the ability to deploy a workload in a Kubernetes environment as well as manually pull and redeploy it

This is currently in progress.

• Periodically check for changes to the git repo and automatically redeploy an application

We're working on this! We're also bringing webhook support with it. More information on this one can be found in that comment: #1753 (comment)

• Support multiple Compose/manifest paths when deploying a workload

This is currently in progress.

[tfenster]

@deviantony do you also plan to add an API for it? Scenario would be a fully automated initial deployment of a stack from a private Git repo with env variables and then later user-triggered updates

[chrisbecke]

enhancements I could think of would be to:

• use kubernetes or swarm secrets (platform dependent of course) to store the git credentials.
• and/or a hashicorp vault integration.

Other than that, my only concern as the swarm owner is, the number of times people submit service files at the moment that don't have restart-max-attempts set, or are lacking labels.

So the ability to have some code - perhaps a designated image - that Portainer can be configured pre-validate deployments might be a nice to have.

[deviantony]

@tfenster

Yes, the current API already supports stack creation, it will be enhanced to support all of the Git options, this is part of our current work :-)

@chrisbecke

Cool, I'll register your request regarding persisting the git credentials in the secret layer.

For your other request though, just to make sure that I understand the problem behind this is that you would like to have a "custom rule validator" that could be applied on Compose files right? In that case I believe that you might want to open a separate feature request as it would also target Compose files that are deployed outside of Git.

[IARI]

For the sake of completeness: I don't see ssh auth (#1752) mentioned on the list yet.
I suppose a proper Git provider integration would make the ssh-key auth less important, but maybe for starters this is easier to implement?

[PANiCnz]

Persisting git credentials would be a really helpful quality of life improvement. Also curious how "Periodically check for changes to the git repo and automatically redeploy an application" will work without persistent creds?

[deviantony]

@PANiCnz yup, we're going to persist the credentials with the introduction of the periodic check.

@IARI after the troubles we had with a potential implementation of SSH auth in #1752 we've pushed it out of the roadmap at the moment but I'll actually re-add it to the open for discussion items.

I'll also close #1752 and refer this issue for discussion.

[ethancedwards8]

enhancements I could think of would be to:

* use kubernetes or swarm secrets (platform dependent of course) to store the git credentials.

* and/or a hashicorp vault integration.

I for one vote for Vault integration, I think that could really tie everything together. Also, managing ENV with Vault would be helpful, as secret management without it is... lackluster.

[alphafalcon]

Just a short note to everyone looking to save git credentials:
For http(s) urls the format http://user:pass@server/path works quite well without entering credentials again in the authentication section.

It does display the credemtials in cleartext in the webinterface though.

[huib-portainer]

We currently have a preview version available for the first iteration of this initiative: portainerci/portainer:pr5340.
It allows you to pull compose files from Git and either poll for changes or use a webhook to pull the latest compose file from Git and redeploy the stack. And it will remember the credentials as well.
Let us know how it's working.
Note that this is a development build and should not be used in a production environment.

[tfenster]

@huib-portainer has the API changed for this one? We are deploying through an API call and it works fine on 2.6.1 but the exact same call on the pr5340 image returns the following error message:

{"message":"read C:\\data/compose/4: The handle is invalid.\n","details":"read C:\\data/compose/4: The handle is invalid.\n"}

The portainer log shows basically the same:

2021/07/27 06:47:22 http error: read C:\data/compose/4: The handle is invalid.
 (err=read C:\data/compose/4: The handle is invalid.
�
) (code=500)

[huib-portainer]

Hi, are you able to share how you're calling the API?
As well as provide some info with regards to your docker version, OS details etc

[tfenster]

This is the call (in PowerShell), formatted for easier readability

Invoke-RestMethod 
    -Uri "$baseUrl/stacks?type=1&method=repository&endpointId=$($endPointsResponse.Id)" 
    -Method POST 
    -Headers $headers 
    -ContentType 'application/json' 
    -Body "{
        `"composeFilePathInRepository`": `"stacks/docker-automation/docker-compose.yml`",
        `"env`": [
            {
                `"name`": `"SWARM_NAME`",
                `"value`": `"$($SwarmName)`"
            },
            {
                `"name`": `"COSMO_INTERNAL`",
                `"value`": `"$($CosmoInternal)`"
            },
           {
                `"name`": `"EXTERNAL_DNS`",
                `"value`": `"$($SwarmName).$($AzureRegion).cloudapp.azure.com`"
            },
            {
                `"name`": `"CLEANUP_THRESHOLD_GB`",
                `"value`": `"$($DocuumThresholdInGb)`"
            }
        ],
        `"name`": `"$($StackName)`",
        `"repositoryAuthentication`": true,
        `"repositoryPassword`": `"$($GithubToken)`",
        `"repositoryReferenceName`": `"refs/heads/$($BranchName)`",
        `"repositoryURL`": `"https://github.com/cosmoconsult/azure-swarm-bc`",
        `"repositoryUsername`": `"$($GithubUsername)`",
        `"swarmID`": `"$($swarmResponse.ID)`"
    }"

Docker version: 20.10.5
OS: Windows Server 2019 (2004)

Please let me know if you need anything else

[huib-portainer]

As you suspected, the API did change.
ComposeFilePathInRepository was renamed to ComposeFile.

And the following non-mandatory fields were added:

• AdditionalFiles
• AutoUpdate

Can you please give that a try?

[tfenster]

Great, that works! Not a big thing but maybe something to document: If I create a stack with "name": "docker-automation", it becomes "dockerautomation", so it seems like the "-" is removed?

And it seems like I now can't make portainer "forget" the authorization? What I mean is that if I deploy a stack with auth information, but don't want portainer to store it, that currently seems impossible?

[huib-portainer]

The dash one may behave as described by #4835.

For automatically pulling changes from Git we need to remember the credentials, so that's why we recommend using a PAT.

[ismailyenigul]

That would be great if we can specify git credentials under settings like we do for registries configuration. Therefore, we can use same git creds in many stacks without typing user and password for each.
When we have to change password, we have to update all stacks which is a bit annoying.

[yelodevopsi]

I just tested that the auth with username and password works with Bitbucket too.
However, only with account login. Using SSH keys is greatly needed, especially when 2.step auth is mandatory in most repo services.

[huib-portainer]

Wouldn't you be able to use an access token?
https://confluence.atlassian.com/bitbucketserver/http-access-tokens-939515499.html

[yelodevopsi]

Wouldn't you be able to use an access token? https://confluence.atlassian.com/bitbucketserver/http-access-tokens-939515499.html

I see the page you're referring too was updated Des 21. Has the feature been removed?
Edit: Just found out this is a Bitbucket Data Center feature. That might be it why I don't see it.

Anyways, this option is not available to me.
But an equivalent solution was to use "App Passwords":
https://bitbucket.org/account/settings/app-passwords/

I was able to log in with the username and app-password, by using a restricted service/read-only account.

[huib-portainer]

Ah, that explains it ;)
https://community.atlassian.com/t5/Bitbucket-questions/Personal-Access-Token-for-Bitbucket-Cloud/qaq-p/677663

[AnysSido]

Another +1 for storing git credentials. Currently I have to create a new personal access token in my GitHub account whenever I deploy a new stack as you cannot view existing tokens to reuse them. Typically a single access token would be made for a single application but my GitHub account is becoming full of individual portainer stack tokens.

[tfenster]

@huib-portainer I had in mind that the "prune" option (remove services/containers no longer in the stack) was available previously when doing a redeploy. Was that remove? Or was it never there and I mix it up with something else?

[huib-portainer]

@tfenster have a read through #5838

[shawnweeks]

With #5979 closed and listed as being included in this issue has there been any further discussion. I just got burned on this issue because there are now blog posts saying that you can do that but you really can't For example see https://tobiasfenster.io/docker-container-stack-deployments-with-dependent-files-through-portainer (not my blog) that lead me to believe you were checking out the whole repo on the target when your really just pulling the compose file. This is important for things where you might need to create some database users prior to deploying your stack like the official postgres image does.

[jdenson]

That would be great if we can specify git credentials under settings like we do for registries configuration. Therefore, we can use same git creds in many stacks without typing user and password for each. When we have to change password, we have to update all stacks which is a bit annoying.

Along this same line, how about a way to save all the git repo information and reuse it quickly in a new stack? For those of us running a lot of stacks (like me) having to copy and paste all those fields over and over is....tedious, to be generous.

[chrisbecke]

The other thing lacking from the existing git integration is that it needs to be supplemented with an external CI/CD pipeline such as Jenkins or GitLab to do the actual image building, as the Portainer GitOps will, well, deploy a stack.

However in a lot of cases this is overkill. I propose a simple CI Pipeline in Portainer that uses Docker only:

A Portainer Pipeline project would be built around docker compose, and would checkout a complete git repo when triggered, and run docker compose build docker compose push on all, or specified services, to build Dockerfiles and push built images to registries. and or a docker compose run.

Being able to get Portainer to build, push and/or run a compose file,can enable far more complex behaviours far outside just preparing images for a stack deployment :- Pipelines could also be used to run admin tasks against a swarm or cluster - i.e. the pipeline could just trigger a service job that runs docker system prune -af on each node. Or run periodic audits to ensure that all deployed services have a node.role placement constraint.

[huib-portainer]

Currently you can build a local image from your compose file if you're using docker standalone.
But it won't work for Swarm, and won't push the image to a registry for instance.

In your Git repo you'd have:

docker-compose.yml

version: '3'
  services:
    test-build-image:
      build: ./

Dockerfile

FROM nginx:latest
ADD https://quirky-beaver-325c33.netlify.app/hello.html /usr/share/nginx/html/index.html
RUN chown nginx:nginx /usr/share/nginx/html/*
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]

And then build and deploy via Portainer as follows

[z0rgster]

Feature request: Repo Caching

Hi all,
I stumpled upon some kind of catch-22 situation. Whenever I want to (re-)start a stack, I seemingly must have connection to the git repo at that moment. I want to host all my compose files in seperate repos on a private GitLab instance. That instance contains it's own compose file in a repo. Until now that was fine because the actual compose file was handled by portainer.
If I wanted to launch GitLab with a compose file that is on that GitLab I would run into a problem. This is probably a weird and maybe even dumb edge case but here I am.

A possible solution would be that Portainer caches that repo (or at least the actually used files) and keeps them as known-good if all the containers start successfully. Alternatively the user can mark them manually as such.

I imagine that as follows:
When I'd start the GitLab stack, Portainer would try and fetch the 'fresh' files from the repo and obviously fail. Then the last known-good compose file is used to start GitLab.
When I'd update GitLab, the new files would be fetched first. The the containers are re-created with the fetched compose file. If everything starts up successfully, I have an updated GitLab instance and can mark that latest cached version as know-good. Else GitLab would start up with the old gnown-good compose file and I could make changes and try again.

[Stitch10925]

Some comments on this feature:

• Auth Passwords are not encrypted in the database
• There is no way to change the git URL or path of a stack. That means that when you rearrange your git repo, you need to recreate all the stacks again

Otherwise I am really loving the ability to pull stacks from a git repo!

[stevenanthonyrevo]

Adding some notes to this feature:

• The image associated with the stack does not get rebuilt unless the original image is manually deleted first.

Git connected to a stack work successfully in deployment but changes are only applied when the docker-compose file is modified not the repository.

• Any containers previously built via a Dockerfile into an image and tagged with the 'latest' tag will not be updated nor automatically rebuilt unless you have deleted the original image first.

[jerrac]

I'll second @Stitch10925 's point about changing the git url. I run my own GitLab instance. Needed to move it a while back. Which meant recreating everything.

Actually, in general, it should be possible to change everything about a Stack's git integration. The url, the auth, the branch, all of it should be editable.

Even better would be if we could detach from git to make a quick test change, then reattach later on.

I also would love to see the branch listed on the details screen. It is not shown on a Swarm Stack details screen, no idea about Kube/etc.

[jerrac]

I also would love to see the branch listed on the details screen.
In 2.19.4, you can open the "Advanced configuration" collapsed div and what branch you set is visible, and editable.

So that is fixed.

Still no way to change the actual git url though.