Add support to create default Instrumentation for operator based auto…

…-instrumentation

CHANGELOG.md

@@ -7,6 +7,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 ### Added
 
 - Update PodDisruptionBudgets API version to allow both `policy/v1beta1` and `policy/v1` [#835](https://github.com/signalfx/splunk-otel-collector-chart/pull/835)
+- Add support to create default Instrumentation for operator based auto-instrumentation [#836](https://github.com/signalfx/splunk-otel-collector-chart/pull/836)
 
 ## [0.80.0] - 2023-06-27
 

docs/auto-instrumentation-install.md

@@ -30,10 +30,28 @@ If a cert-manager is not available in the cluster (or other TLS certificate sour
 using `certmanager.enabled=true`. The cert-manager issues TLS certificates the operator requires. You can use the
 commands below to run these steps.
 
-_Note: In order for trace telemetry data to be properly ingested, the attribute `deployment.environment` must be on board
-the exported traces. There are two ways to set this attribute._
-- _Use the values.yaml optional `environment` configuration_
-- _Use the Instrumentation spec with the env var OTEL_RESOURCE_ATTRIBUTES_
+<details>
+<summary>Expand for extended Instrumentation configuration details</summary>
+
+- An [opentelemetry.io/v1alpha1 Instrumentation](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#instrumentation)
+object is used to configure the auto-instrumentation of your applications. To successfully enable instrumentation, the
+target pod must have an Instrumentation object available.
+- When `operator.enabled=true` and `operator.instrumentation.create=true`, the helm chart deploys a
+[default](https://github.com/signalfx/splunk-otel-collector-chart/blob/main/examples/enable-operator-and-auto-instrumentation/rendered_manifests/operator/instrumentation.yaml)
+opentelemetry.io/v1alpha1 Instrumentation object.
+  - The default Instrumentation supports [AlwaysOn Profiling](https://docs.splunk.com/Observability/apm/profiling/intro-profiling.html) when `splunkObservability.profilingEnabled=true`.
+    - Examples
+      - [Enable always-on profiling](../examples/enable-operator-and-auto-instrumentation/instrumentation_enable_profiling.yaml)
+  - Users can override the specifications of the default deployed instrumentation by setting override values under `operator.instrumentation.spec`.
+    - Examples
+      - [Add custom environment span tag](../examples/enable-operator-and-auto-instrumentation/instrumentation_add_custom_environment_span_tag.yaml)
+      - [Add trace sampler](../examples/enable-operator-and-auto-instrumentation/instrumentation_add_trace_sampler.yaml)
+- To view a deployed Instrumentation, you can use the command: `kubectl get otelinst {instrumentation_name} -o yaml`.
+- For proper ingestion of trace telemetry data, the `deployment.environment` attribute must be present in the exported traces. There are two ways to set this attribute:
+  - Use the optional `environment` configuration in `values.yaml`.
+  - Use the Instrumentation spec (`operator.instrumentation.spec.env`) with the environment variable `OTEL_RESOURCE_ATTRIBUTES`.
+
+</details>
 
 ```bash
 # Check if cert-manager is already installed, don't deploy a second cert-manager.
@@ -43,31 +61,13 @@ kubectl get pods -l app=cert-manager --all-namespaces
 helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
-### 2. Deploy the opentelemetry.io/v1alpha1 Instrumentation
-
-The [Instrumentation](https://github.com/open-telemetry/opentelemetry-operator/blob/main/docs/api.md#instrumentation)
-object is used to configure how your applications will be auto-instrumented. An Instrumentation object must be
-available to the target pod being auto-instrumented in order for instrumentation to be successful.
-
-For Splunk Instrumentation configurations, please see or use this [instrumentation.yaml](../examples/enable-operator-and-auto-instrumentation/instrumentation.yaml)
-in our examples directory. The Splunk Instrumentation object should be installed in the namespace target applications
-resides in. For example, for instrumenting applications in a namespace called `my-app-namespace` the Instrumentation
-object could be installed with the following command.
-
-```bash
-# Install
-kubectl apply -f instrumentation.yaml -n my-app-namespace
-# Check the current deployed values
-kubectl get otelinst -o yaml -n my-app-namespace
-```
-
-### 3. Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully
+### 2. Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully
 
 <details>
 <summary>Expand for sample output to verify against</summary>
 
 ```bash
-kubectl get pods -n default
+kubectl get pods
 # NAME                                                            READY   STATUS
 # splunk-otel-collector-agent-lfthw                               2/2     Running
 # splunk-otel-collector-cert-manager-6b9fb8b95f-2lmv4             1/1     Running
@@ -76,19 +76,19 @@ kubectl get pods -n default
 # splunk-otel-collector-k8s-cluster-receiver-856f5fbcf9-pqkwg     1/1     Running
 # splunk-otel-collector-opentelemetry-operator-56c4ddb4db-zcjgh   2/2     Running
 
-kubectl get mutatingwebhookconfiguration.admissionregistration.k8s.io -n default
+kubectl get mutatingwebhookconfiguration.admissionregistration.k8s.io
 # NAME                                      WEBHOOKS   AGE
 # splunk-otel-collector-cert-manager-webhook              1          14m
 # splunk-otel-collector-opentelemetry-operator-mutation   3          14m
 
-kubectl get otelinst -n my-app-namespace
-# NAME                          AGE   ENDPOINT
-# splunk-instrumentation        3m   http://$(SPLUNK_OTEL_AGENT):4317
+kubectl get otelinst
+# NAME                    AGE   ENDPOINT
+# splunk-otel-collector   3s    http://$(SPLUNK_OTEL_AGENT):4317
 ```
 
 </details>
 
-### 4. Instrument application by setting an annotation
+### 3. Instrument application by setting an annotation
 
 An _instrumentation.opentelemetry.io/inject-{instrumentation_library}_ annotation can be added to the following:
 - Namespace: All pods within that namespace will be instrumented.
@@ -106,7 +106,7 @@ The instrumentation annotations can have the following values:
 - "my-other-namespace/my-instrumentation" - name and namespace of Instrumentation CR instance in another namespace to use.
 - "false" - do not inject.
 
-### 5. Check out the results at [Splunk Observability APM](https://app.us1.signalfx.com/#/apm)
+### 4. Check out the results at [Splunk Observability APM](https://app.us1.signalfx.com/#/apm)
 
 The trace and metrics data should populate the APM dashboard.To better
 visualize this example as a whole, we have also included an image

examples/enable-operator-and-auto-instrumentation/README.md

@@ -32,19 +32,7 @@ kubectl get pods -l app=cert-manager --all-namespaces
 helm install splunk-otel-collector -f ./my_values.yaml --set operator.enabled=true,environment=dev splunk-otel-collector-chart/splunk-otel-collector
 ```
 
-#### 2.2 Deploy the opentelemetry.io/v1alpha1 Instrumentation
-
-The Instrumentation object is a spec to configure what instrumentation libraries to use in the target namespace
-to use for instrumentation. This Instrumentation will be used to know how to instrument the spring-clinic applications
-in the spring-petclinic namespace.
-
-```bash
-curl https://raw.githubusercontent.com/signalfx/splunk-otel-collector-chart/main/examples/enable-operator-and-auto-instrumentation/instrumentation.yaml | kubectl apply -n spring-petclinic -f -
-# Note: You can use this command if you want to use a local custom instrumentation file.
-# kubectl apply -f examples/enable-operator-and-auto-instrumentation/instrumentation.yaml -n spring-petclinic
-```
-
-#### 2.3 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully
+#### 2.2 Verify all the OpenTelemetry resources (collector, operator, webhook, instrumentation) are deployed successfully
 
 <details>
 <summary>Expand for kubectl commands to run and output</summary>
@@ -129,6 +117,7 @@ kubectl patch deployment customers-service -p '{"spec": {"template":{"metadata":
 kubectl patch deployment vets-service -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"false"}}}} }' -n spring-petclinic
 kubectl patch deployment discovery-server -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"false"}}}} }' -n spring-petclinic
 kubectl patch deployment visits-service -p '{"spec": {"template":{"metadata":{"annotations":{"instrumentation.opentelemetry.io/inject-java":"false"}}}} }' -n spring-petclinic
+
 ```
 </details>
 

examples/enable-operator-and-auto-instrumentation/enable-operator-and-auto-instrumentation-values.yaml

@@ -4,11 +4,14 @@ splunkObservability:
   logsEnabled: true
   accessToken: CHANGEME
   realm: us0
+  # Enables Always on Profiling
+  # profilingEnabled: true
 clusterName: CHANGEME
 environment: CHANGEME
 
 operator:
   enabled: true
+
 certmanager:
   enabled: true
 

examples/enable-operator-and-auto-instrumentation/instrumentation/instrumentation_add_custom_environment_span_tag.yaml

@@ -0,0 +1,34 @@
+---
+# Source: splunk-otel-collector/templates/operator/instrumentation.yaml
+# Description: Set a custom value (dev) for the deployment.environment tag on exported spans
+# Input values.yaml:
+# operator:
+#   enabled: true
+#   instrumentation:
+#     spec:
+#       env:
+#         - name: OTEL_RESOURCE_ATTRIBUTES
+#           value: "deployment.environment=dev"
+# Output instrumentation.yaml:
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: splunk-instrumentation
+spec:
+    dotnet:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318
+    env:
+    - name: OTEL_RESOURCE_ATTRIBUTES
+      value: deployment.environment=dev
+    exporter:
+      endpoint: http://$(SPLUNK_OTEL_AGENT):4317
+    propagators:
+    - tracecontext
+    - baggage
+    - b3
+    python:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318

examples/enable-operator-and-auto-instrumentation/instrumentation/instrumentation_add_trace_sampler.yaml

@@ -0,0 +1,40 @@
+---
+# Source: splunk-otel-collector/templates/operator/instrumentation.yaml
+# Description: Add a trace sampler to control how many traces are exported
+# Input values.yaml:
+# operator:
+#   enabled: true
+#   instrumentation:
+#     spec:
+#       sampler:
+#         type: traceidratio
+#         argument: "0.95"
+# Output instrumentation.yaml:
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: splunk-instrumentation
+spec:
+    dotnet:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318
+    env:
+    - name: SPLUNK_OTEL_AGENT
+      valueFrom:
+        fieldRef:
+          apiVersion: v1
+          fieldPath: status.hostIP
+    exporter:
+      endpoint: http://$(SPLUNK_OTEL_AGENT):4317
+    propagators:
+    - tracecontext
+    - baggage
+    - b3
+    python:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318
+    sampler:
+      argument: "0.95"
+      type: traceidratio

examples/enable-operator-and-auto-instrumentation/instrumentation/instrumentation_enable_profiling.yaml

@@ -0,0 +1,59 @@
+---
+# Source: splunk-otel-collector/templates/operator/instrumentation.yaml
+# Description: Enable Java always on profiler
+# Documentation: https://docs.splunk.com/Observability/gdi/get-data-in/application/java/configuration/advanced-java-otel-configuration.html#java-settings-for-alwayson-profiling
+# Input values.yaml:
+# splunkObservability:
+#   profilingEnabled: true
+# operator:
+#   enabled: true
+#   instrumentation:
+#     spec:
+#       java:
+#         env:
+#           - name: SPLUNK_PROFILER_ENABLED
+#             value: "true"
+#           - name: SPLUNK_PROFILER_MEMORY_ENABLED
+#             value: "true"
+# Output instrumentation.yaml:
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.80.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.80.0"
+    app: splunk-otel-collector
+    component: otel-operator
+    chart: splunk-otel-collector-0.80.0
+    release: default
+    heritage: Helm
+    app.kubernetes.io/component: otel-operator
+spec:
+  dotnet:
+    env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318
+  env:
+    - name: SPLUNK_OTEL_AGENT
+      valueFrom:
+        fieldRef:
+          apiVersion: v1
+          fieldPath: status.hostIP
+    - name: SPLUNK_PROFILER_ENABLED
+      value: "true"
+    - name: SPLUNK_PROFILER_MEMORY_ENABLED
+      value: "true"
+  exporter:
+    endpoint: http://$(SPLUNK_OTEL_AGENT):4317
+  propagators:
+    - tracecontext
+    - baggage
+    - b3
+  python:
+    env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318

examples/enable-operator-and-auto-instrumentation/rendered_manifests/operator/instrumentation.yaml

@@ -0,0 +1,39 @@
+---
+# Source: splunk-otel-collector/templates/operator/instrumentation.yaml
+apiVersion: opentelemetry.io/v1alpha1
+kind: Instrumentation
+metadata:
+  name: default-splunk-otel-collector
+  labels:
+    app.kubernetes.io/name: splunk-otel-collector
+    helm.sh/chart: splunk-otel-collector-0.80.0
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/instance: default
+    app.kubernetes.io/version: "0.80.0"
+    app: splunk-otel-collector
+    component: otel-operator
+    chart: splunk-otel-collector-0.80.0
+    release: default
+    heritage: Helm
+    app.kubernetes.io/component: otel-operator
+spec:
+    dotnet:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318
+    env:
+    - name: SPLUNK_OTEL_AGENT
+      valueFrom:
+        fieldRef:
+          apiVersion: v1
+          fieldPath: status.hostIP
+    exporter:
+      endpoint: http://$(SPLUNK_OTEL_AGENT):4317
+    propagators:
+    - tracecontext
+    - baggage
+    - b3
+    python:
+      env:
+      - name: OTEL_EXPORTER_OTLP_ENDPOINT
+        value: http://$(SPLUNK_OTEL_AGENT):4318

helm-charts/splunk-otel-collector/templates/operator/_helpers.tpl

@@ -0,0 +1,32 @@
+{{- define "validation-rules" -}}
+{{- $tracesEnabled := or (include "splunk-otel-collector.platformTracesEnabled" .) (include "splunk-otel-collector.o11yTracesEnabled" .) -}}
+{{- $endpointOverridden := and .Values.operator.instrumentation.spec .Values.operator.instrumentation.spec.exporter .Values.operator.instrumentation.spec.exporter.endpoint (ne .Values.operator.instrumentation.spec.exporter.endpoint "") -}}
+{{- if and .Values.operator.enabled .Values.operator.instrumentation.create $tracesEnabled (not $endpointOverridden) -}}
+  {{/*  If no endpoint override was provided, the environment variable/tag must be set by the agent, gateway, or instrumentation.*/}}
+  {{- if or (not .Values.environment) (eq .Values.environment "") -}}
+    {{- $envSet := false -}}
+    {{- if and .Values.operator.instrumentation.spec .Values.operator.instrumentation.spec.env -}}
+      {{- range .Values.operator.instrumentation.spec.env -}}
+        {{- if and (eq .name "OTEL_RESOURCE_ATTRIBUTES") .value -}}
+          {{- $envSet = true -}}
+        {{- end -}}
+      {{- end -}}
+    {{- end -}}
+    {{- if not $envSet -}}
+      {{- fail "When operator.enabled=true, operator.instrumentation.create=true, (splunkPlatform.tracesEnabled=true or splunkObservability.tracesEnabled=true), (agent.enabled=true or gateway.enabled=true), and .Values.operator.instrumentation.spec.exporter.endpoint is not set, either environment must be a non-empty string or operator.instrumentation.spec.env must contain an item with {name: OTEL_RESOURCE_ATTRIBUTES, value: non-empty string}" -}}
+    {{- end -}}
+  {{- end -}}
+{{- end -}}
+{{- end -}}
+
+{{- define "splunk-otel-collector.operator.instrumentation.exporter.endpoint" -}}
+{{- if and .Values.operator.instrumentation.spec .Values.operator.instrumentation.spec.exporter .Values.operator.instrumentation.spec.exporter.endpoint (ne .Values.operator.instrumentation.spec.exporter.endpoint "") }}
+  "{{ .Values.operator.instrumentation.spec.exporter.endpoint }}"
+{{- else if .Values.agent.enabled }}
+  "http://$(SPLUNK_OTEL_AGENT):4317"
+{{- else if .Values.gateway.enabled }}
+  "http://{{ include "splunk-otel-collector.fullname" . }}:4317"
+{{- else -}}
+  {{- fail "When operator.enabled=true, operator.instrumentation.create=true, (splunkPlatform.tracesEnabled=true or splunkObservability.tracesEnabled=true), (agent.enabled=true or gateway.enabled=true), and .Values.operator.instrumentation.spec.exporter.endpoint is not set, either environment must be a non-empty string or operator.instrumentation.spec.env must contain an item with {name: OTEL_RESOURCE_ATTRIBUTES, value: non-empty string}" -}}
+{{- end }}
+{{- end }}