Allow to create dynamic index patterns for elasticsearch output

[123BLiN]

Is your feature request related to a problem? Please describe.
I want to forward logs to different indexes in elasticsearch based on namespace/pod name/etc. dynamically

Describe the solution you'd like
It seems that it is possible with another type of elasticsearch output plugin: @type elasticsearch_dynamic
https://github.com/uken/fluent-plugin-elasticsearch#dynamic-configuration

Describe alternatives you've considered
Did not found any

Additional context
The goal is to have separate indexes per app in order to mitigate index mapping issues when same field has different type in different apps and to have smaller problem scope, ideally only single app with broken json scheme should have issues with log forwarding after developers error etc.

[123BLiN]

Hm, maybe I missunderstood elasticsearch plugin documentation and just need to try placeholders ( https://github.com/uken/fluent-plugin-elasticsearch#placeholders ) any working examples with logging-operator maybe?

[123BLiN]

It works with placeholders, but I have another issue with tag format ( https://github.com/banzaicloud/fluent-plugin-tag-normaliser/issues/2) , anyway closing this, sorry for disturbing.

[123BLiN]

For someone who will look for a workaround.
Provide lua script with custom fluent-bit config:

fluent-bit.conf: |-
...
    [FILTER]
        name                  lua
        match                 *
        script                filters.lua
        call                  set_kube_app_name_field
...
  filters.lua: |-
    function set_kube_app_name_field(tag, timestamp, record)
        if record["kubernetes"]["labels"]["app.kubernetes.io/name"] then
          record["kube_app_name"] = record["kubernetes"]["labels"]["app.kubernetes.io/name"] -- copy nested field to root, remove forward slash
        else
          record["kube_app_name"] = "not_defined"
        end

        return 2, timestamp, record
    end
...
and use this field in elasticsearch output plugin like so:

index_name: k8s.${$.kubernetes.namespace_name}.${kube_app_name}-%Y.%m.%d
buffer:
  tags: tag, time, $.kubernetes.namespace_name, kube_app_name

[vavdoshka]

Thanks for the hints, I followed pretty much same path as you did and saw the other approach with tag_nomalizer which eventualy led me to this soltuion:

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: test
  namespace: logging
spec:
  filters:
  - record_modifier:
      records:
        - kube_app_name: ${record.dig('kubernetes', 'labels', 'app.kubernetes.io/name') || "unknown"}

and

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
  name: test
  namespace: logging
spec:
  awsElasticsearch:
    endpoint:
      url: https://test.com
    index_name: logs-${$.kubernetes.namespace_name}-${kube_app_name}-%Y.%m.%d
    buffer:
      tags: tag,time,$.kubernetes.namespace_name,kube_app_name