How to deploy Nebari on existing Azure Kubernetes cluster

[Traychy]

Hey all, I am working on a project which needs to test Nebari on AKS which already exists. I saw that there is a specific article in the Nebari's official documentation for deploying on pre-existing kubernetes cluster - https://www.nebari.dev/docs/how-tos/nebari-kubernetes/
In the article only AWS is mentioned so I was wondering if there is any difference for Azure and if yes what is it? Should we wait for updated version of the article including the Azure/GCP deployment on pre-existing kubernetes clusters?

[rsignell]

@Traychy thanks for raising the issue here! I don't know much about Azure either, but @Adam-D-Lewis provided this info to me offline:

Nebari runs on a managed kubernetes cluster (AKS on azure). Azure requires that a new VNet is created during the creation of the AKS cluster so that won't work if you require pre existing Vnets. Additionally Nebari will try to create several Vnets by default. I believe you can use pre-existing vnets for all the vnets ( except for the AKS one) using Terraform overrides. Additionally, if you are deploying a private AKS cluster (aks nodes don't have a public IP address), then you'll need to deploy from a VM on Azure that can still reach the nodes.

[rsignell]

@Adam-D-Lewis I'm pretty sure it's the latter, right? If so, I'm sure we could work around that.

Also, I believe you said you helped one of your clients deal with similar issues on Azure and might have a sample config that could be useful to us?

[Adam-D-Lewis]

Yeah, As long as the machine deploying nebari can reach the nodes, then the nodes don't need a public IP address. I can share an example configuration that may help, but I'm traveling at the moment and won't have access to it until next week.

[rsignell]

Okay, thanks @Adam-D-Lewis ! Have a great weekend!

[Traychy]

We get to a point where Nebari is reaching the cluster and passing stage2 but it is failing on stage3-kubernetes-initialize with this error 'TypeError: Object of type KeyValueDict is not JSON serializable'. Tried to look at which exact module is failing but still no success. Namespace and traefik CRDs are deployed so I am wondering if the script is complaining about the node-selectors or some of the modules in stage3. Can you guide us in any direction ?

[Traychy]

It appeared that this is a bug in the code. We managed to fix it on our end and there is a pull request for it on your repo. Other than that, there is an issue with this - Error: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev1loadbalancernopublic-601d80ac4cae0639f9c6] Load Balancers should not have public IPs. azure-load-balancer-internal annotation is required for nebari-traefik-ingress. Ingress section is configured in the nebari-config.yaml with the needed information but still the same error.

[dharhas]

@Traychy I'm converting this from a discussion to an issue so we can track it better.

[rsignell]

#2131