Use secrets to store wehbook urls [ Kubernetes ]

[mehrdad-khojastefar]

For security reasons I want to store my webhook urls in a secret. Is it possible now ? I've read through the chart and it seems like it is not.
So are there any points to consider for this feature because if not I would be happy to develop it.
And it would be great if you share your ideas too.

[jertel]

If you're asking about a method of storing most of the rule content in configMaps but then having some sensitive data omitted from the rule configMap and instead placed in a secret, the answer is no, it's not supported. If you'd like to store the entire rule in a k8s secret then that is supported via the secretRulesName configuration value.

As I don't know what you're trying to protect I can't offer ideas on improvement.

For example, your recent PR suggests you are storing rules locally in your chart subdirectory. And then at deploy time you are now reading those local rules and generating a large configMap which is then stored in the k8s cluster. So if you're wanting to keep the webhooks out of the local rule files but you're ok with having the webhook be in the k8s configMap then that requires a specific solution that is more suited for the Helm template syntax.

However, if you are not concerned with webhooks being in your local rule files prior to deployment, but instead are more concerned with not having webhooks be stored in a configMap inside the k8s cluster, then that requires a different solution. You could store the entire rule in a secret, instead of a configMap, to solve that.

[mehrdad-khojastefar]

name: ADMIN_LOGIN_FAILED
type: any
index: aduit
timestamp_field: timestamp
realert:
  minutes: 0
filter:
  - term:
      event.keyword: "ADMIN_FAILED"
alert:
- "rocketchat"
alert_text_type: alert_text_only
alert_subject: "admin login failure - #{}"
alert_subject_args:
  - username
alert_text: |-
  {{ "username": "{}" "timestamp": "{}", "rule_key": "ADMIN_FAILED" }}
alert_text_args:
  - username
  - timestamp

this is one of our rules, now in the alert section we defined rocketchat and I would like to store this rocket chat webhook in a secret.
the webhook is defined in the values.yaml now and it is a security concern for us at the moment.

extraConfigOptions:
  discord_webhook_url: https://discord.com/api/webhooks/...
  rocket_chat_webhook_url: https://chat.rocket.com/hooks/...

and then in the config.yaml we have

...
{{- if .Values.extraConfigOptions }}
{{ toYaml .Values.extraConfigOptions | indent 4 }}
{{- end }}
...

It would be very convenient I guess to store each webhook in a secret
I am aware that we can create a secret config but the problem with that method is that we have more than one tool that need the rocket chat webhook. so we are trying to use vault and store each webhook in a secret and have our applications read from the vault.

[jertel]

It sounds like you're more interested in securing the webhook URL pre-deployment, but once it's in the cluster it's ok to be merged in with the rule. Is that correct?

If so, and since you are now storing your rules in a directory prior to helm deployment, my suggestion is to write a small Bash, Python, or ytt script that can fetch variables from the Vault and substitute them into temporary rule files on disk, using 600 permissions. Then the script would execute the helm install command and afterwards remove the merged rules from disk.

[mehrdad-khojastefar]

How about using environment variables? with that we can use vault's external secrets to fetch the webhook url

[jertel]

I'm not following.

[mehrdad-khojastefar]

Using external secrets operator we can fetch secrets from our vault, we can store hooks in the vault. Then if we can provide our webhooks url via environment variables we can use this operator to populate the environment variables at deployment time. Using this method since we can store each webhook uri in a secret separately they can be used multiple times in multiple applications without redefining them for each application.

[jertel]

Using external secrets operator we can fetch secrets from our vault, we can store hooks in the vault.

Ok, I understand this.

Then if we can provide our webhooks url via environment variables we can use this operator to populate the environment variables at deployment time.

So you somehow retrieve the URLs from the vault and store them in an environment variable. Then while generating the rules configMap, you want to replace some variables inside the source rule files with the contents of the webhook URL environment variables? Sure, you can do this. But I'm not sure how this relates to ElastAlert 2. This seems to be a custom script you would create that is integrated to your vault and possibly to Helm.

Using this method since we can store each webhook uri in a secret separately they can be used multiple times in multiple
applications without redefining them for each application.

Right, you want to store the webhook URLs in one place and reuse them for other projects. I understand this.

[mehrdad-khojastefar]

I do not need to generate ConfigMap, we do not provide the webhooks in the rules. As the example I provided suggests we are only referring to the type of notification we want to use. so in my original rule example we have rocketchat but where do we provide the webhook urls? in the config.yaml and with the values provided in the Values.extraConfigOptions.
I just think that if we can re-write this so that we can use environment variables to store hooks for notifications it would be better, also is there anything special about this that elastAlert reads webhooks urls this way?

[jertel]

I'm confused because it's unclear if you are trying to read environment variables at runtime, or if you're wanting to prepare a custom config using Vault secrets during deploy time. You stated deploy time earlier but your later posts make me think you are looking at runtime.

If you want to read environment variables at runtime, yes ElastAlert 2 does do this today, but only a select set of variables are supported. See config.py::18. This could be expanded to include additional the webhook settings you suggested. For this to work with K8s, you would generate a K8s secret containing your webhooks and then define env variables exposed to your deployment, and those would pull from those k8s secrets in the chart configuration.

If you want to read a vault and generate a special values-webhook.yml variable file for use with the extraConfigOptions Chart property, to be used during deploy time, then that is not really in scope of this project. It would require a custom Bash script.

[mehrdad-khojastefar]

Sorry for not clarifying earlier.

If you want to read environment variables at runtime, yes ElastAlert 2 does do this today, but only a select set of variables are supported. See config.py::18. This could be expanded to include additional the webhook settings you suggested. For this to work with K8s, you would generate a K8s secret containing your webhooks and then define env variables exposed to your deployment, and those would pull from those k8s secrets in the chart configuration.

Yes this is what we needed. So I will work on it.
Thanks