Skip to content

Commit

Permalink
Adding s3 sqs sink (#8)
Browse files Browse the repository at this point in the history 
🐍 adding s3 sqs sink

Signed-off-by: Matthias Wessendorf <[email protected]>
  • Loading branch information
@matzew
matzew authored Oct 2, 2024
1 parent dce829d commit 865adda
Show file tree
Hide file tree
Showing 14 changed files with 949 additions and 0 deletions.
176 changes: 176 additions & 0 deletions aws-sqs-sink/README.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
= Knative connector aws-sqs-sink

Knative eventing connector based on https://camel.apache.org/camel-kamelets/[Apache Camel Kamelets].
The connector project creates a container image that is pushed into a registry so the image can be referenced in a Kubernetes deployment.

== Kamelet properties

The kn-connector source images provides these properties that you can set (e.g. via environment properties on the deployment).

.Kamelet properties
include::properties.adoc[]

== Kubernetes manifest

The build produces a Kubernetes manifest in (`target/kubernetes/kubernetes.yml`).
This manifest holds all resources required to run the application on your Kubernetes cluster.

You can customize the Kubernetes resources in link:src/main/kubernetes/kubernetes.yml[src/main/kubernetes/kubernetes.yml].
This is being used as a basis and Quarkus will generate the final manifest in `target/kubernetes/kubernetes.yml` during the build.

The final Kubernetes manifest includes:

* Service
* Deployment
* Trigger

== Kamelet sink Pipe

The sink consumes events from the Knative broker.
It uses an Apache Camel Pipe resource as the central piece of code to define how the Knative events are consumed and where the events get forwarded to.

The Pipe is a YAML file located in link:src/main/resources/camel/kn-connector-sink.yaml[src/main/resources/camel/kn-connector-sink.yaml]

.kn-connector-sink.yaml
[source,yaml]
----
apiVersion: camel.apache.org/v1
kind: Pipe
metadata:
name: kn-connector-aws-sqs-sink
spec:
source:
ref:
kind: Broker
apiVersion: eventing.knative.dev/v1
name: default
properties:
type: ""
sink:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1
name: aws-sqs-sink
----

This connector uses the https://camel.apache.org/camel-kamelets/aws-sqs-sink.html[aws-sqs-sink] Kamelet and consumes events from the Knative broker.

The Pipe references a Knative broker as a source and connects to a Kamelet as a sink.

The name of the broker is always `default` because the Knative Trigger resource is responsible for connecting the application to the Knative broker.
The Trigger decides when to call the application as it provides the events to the application based on the trigger configuration.

This way the same container image can be used with different brokers and events (e.g. by adding filter criteria to the Trigger).
It is only a matter of configuring the Trigger resource that connects the application with the Knative broker.

You can find a sample Trigger in link:src/main/kubernetes/kubernetes.yml[src/main/kubernetes/kubernetes.yml]

[source,yaml]
----
apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
annotations:
eventing.knative.dev/creator: kn-connectors
labels:
eventing.knative.dev/connector: aws-sqs-sink
eventing.knative.dev/broker: default
name: kn-connector-aws-sqs-sink
spec:
broker: default
subscriber:
ref:
apiVersion: v1
kind: Service
name: kn-connector-aws-sqs-sink
uri: /events
----

== Configuration

Each Kamelet defines a set of properties.
The user is able to customize these properties when running a connector deployment.

=== Environment variables

You can customize the properties via environment variables on the deployment:

.Environment variables
include::properties.adoc[]

The environment variables that overwrite properties on the Kamelet sink follow a naming convention:

* CAMEL_KAMELET_{{KAMELET_NAME}}_{{PROPERTY_NAME}}={{PROPERTY_VALUE}}

The name represents the name of the Kamelet sink as defined in the https://camel.apache.org/camel-kamelets/[Kamelet catalog].

The environment variables may be set as part of the Kubernetes deployment or as an alternative on the Knative ContainerSource:

[source,yaml]
----
apiVersion: sources.knative.dev/v1
kind: ContainerSource
metadata:
name: kamelet-sink
namespace: knative-samples
spec:
template:
spec:
containers:
- image: quay.io/openshift-knative/kn-connector-aws-sqs-sink:1.0
name: log
env:
- name: CAMEL_KAMELET_{{KAMELET_NAME}}_{{PROPERTY_NAME}}
value: "true"
- name: CAMEL_KAMELET_{{KAMELET_NAME}}_{{PROPERTY_NAME}}
value: "INFO"
sink:
ref:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: default
----

You can also set the environment variable on the running deployment:

[source,shell]
----
kubectl set env deployment/kn-connector-{{sink-name}} CAMEL_KAMELET_{{KAMELET_NAME}}_{{PROPERTY_NAME}}="true"
----

=== ConfigMap and Secret

You may also mount a configmap/secret to overwrite Kamelet properties with values from the configmap/secret resource.

As the Kamelet properties are configured viw environment variables on the ContainerSource you can also use values referencing a configmap or secret.

[source,yaml]
----
apiVersion: sources.knative.dev/v1
kind: ContainerSource
metadata:
name: kamelet-sink
spec:
template:
spec:
containers:
- image: quay.io/openshift-knative/kn-connector-aws-sqs-sink:1.0
name: log
env:
- name: CAMEL_KAMELET_{{KAMELET_NAME}}_{{PROPERTY_NAME}}
valueFrom:
secretKeyRef:
name: my-secret
key: my-property-value
sink:
ref:
apiVersion: eventing.knative.dev/v1
kind: Broker
name: default
----

The example above references a secret called `my-secret` and loads the key `my-property-value`.

== More configuration options

For more information about Apache Camel Kamelets and their individual properties see https://camel.apache.org/camel-kamelets/.
140 changes: 140 additions & 0 deletions aws-sqs-sink/pom.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>dev.knative.eventing</groupId>
<artifactId>kn-connectors</artifactId>
<version>1.0-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>kn-connector-aws-sqs-sink</artifactId>
<name>Knative Connectors :: aws-sqs Sink</name>

<dependencies>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-platform-http</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-microprofile-health</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-kubernetes</artifactId>
</dependency>

<!-- Kubernetes extension -->
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-container-image-jib</artifactId>
</dependency>
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-kubernetes</artifactId>
</dependency>

<!-- Kamelet -->
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-kamelet</artifactId>
</dependency>

<!-- Camel components -->
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-aws2-sqs</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-yaml-dsl</artifactId>
</dependency>

<!-- Camel Knative -->
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-knative</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.quarkus</groupId>
<artifactId>camel-quarkus-knative-producer</artifactId>
</dependency>

<dependency>
<groupId>org.apache.camel.kamelets</groupId>
<artifactId>camel-kamelets-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.camel.kamelets</groupId>
<artifactId>camel-kamelets</artifactId>
</dependency>
<dependency>
<groupId>io.fabric8</groupId>
<artifactId>kubernetes-client</artifactId>
</dependency>

<!-- Test dependencies -->
<dependency>
<groupId>io.quarkus</groupId>
<artifactId>quarkus-junit5</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.citrusframework</groupId>
<artifactId>citrus-quarkus</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.citrusframework</groupId>
<artifactId>citrus-http</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>dev.knative.eventing</groupId>
<artifactId>kn-connector-maven-plugin</artifactId>
<version>${project.version}</version>
<configuration>
<kameletName>aws-sqs-sink</kameletName>
</configuration>
<executions>
<execution>
<id>generate-connector-spec</id>
<phase>process-classes</phase>
<goals>
<goal>generate</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>${quarkus.platform.group-id}</groupId>
<artifactId>quarkus-maven-plugin</artifactId>
</plugin>
<plugin>
<artifactId>maven-compiler-plugin</artifactId>
<version>${compiler-plugin.version}</version>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire-plugin.version}</version>
</plugin>
<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>${surefire-plugin.version}</version>
</plugin>
</plugins>
</build>
</project>
4 changes: 4 additions & 0 deletions aws-sqs-sink/properties.adoc
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
|===
|Property |Required |EnvVar |Description

|===
Loading

0 comments on commit 865adda

Please sign in to comment.