aws-samples · indranil-banerjee-aws · Oct 21, 2024
diff --git a/msk-lambda-iam-java-sam/.gitignore b/msk-lambda-iam-java-sam/.gitignore
@@ -0,0 +1,23 @@
+node_modules
+npm-debug.log
+package-lock.json
+package
+*out.yml
+out.json
+bucket-name.txt
+target
+build
+.gradle
+*.zip
+bin
+obj
+Gemfile.lock
+lib
+__pycache__
+*.pyc
+.classpath
+.factorypath
+.project
+.settings/*
+.aws
+.sam
diff --git a/msk-lambda-iam-java-sam/README.md b/msk-lambda-iam-java-sam/README.md
@@ -0,0 +1,134 @@
+# msk-lambda-iam-java-sam
+# Java AWS Lambda Kafka consumer with IAM auth, using AWS SAM
+
+This pattern is an example of a Lambda function that consumes messages from an Amazon Managed Streaming for Kafka (Amazon MSK) topic, where the MSK Cluster has been configured to use IAM authentication. This pattern assumes you already have an MSK cluster with a topic configured, if you need a sample pattern to deploy an MSK cluster either in Provisioned or Serverless modes please see the [msk-cfn-sasl-lambda pattern](https://serverlessland.com/patterns/msk-cfn-sasl-lambda). 
+
+This project contains source code and supporting files for a serverless application that you can deploy with the SAM CLI. It includes the following files and folders.
+
+- kafka_event_consumer_function/src/main/java - Code for the application's Lambda function.
+- events - Invocation events that you can use to invoke the function.
+- kafka_event_consumer_function/src/test/java - Unit tests for the application code. 
+- template.yaml - A template that defines the application's AWS resources.
+
+The application uses several AWS resources, including Lambda functions and an MSK event source. These resources are defined in the `template.yaml` file in this project. You can update the template to add AWS resources through the same deployment process that updates your application code.
+
+Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
+
+## Requirements
+
+* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
+* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
+* [Git installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
+* [AWS Serverless Application Model](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html) (AWS SAM) installed
+* Create MSK cluster and topic that will be used for testing. It is important to create the topic before deploying the Lambda function, otherwise the event source mapping will stay disabled. 
+
+
+Before proceeding with the next step, please make sure you have Java JDK and Maven installed on your machine
+
+For the latest version of Amazon Corretto JDK (at the time of publishing), please go the following link:
+
+https://docs.aws.amazon.com/corretto/latest/corretto-21-ug/downloads-list.html
+
+Please follow the instructions to download and install the JDK for your Operating System
+
+Note that you don't have to use Amazon Corretto JDK but can use JDK from another source as well
+
+For the latest version of Maven (at the time of publishing) please go the following link:
+
+https://maven.apache.org/download.cgi#:~:text=Apache%20Maven%203.9.9%20is,recommended%20version%20for%20all%20users.
+
+Please follow the instructions to download and install Maven and then add the location to the bin folder of Maven in your System PATH
+
+To ensure Java and Maven are correctly installed, run the commands:
+
+java --version
+
+mvn --version
+
+
+## Deploy the sample application
+
+The AWS SAM CLI is a serverless tool for building and testing Lambda applications. It uses Docker to locally test your functions in an Amazon Linux environment that resembles the Lambda execution environment. It can also emulate your application's build environment and API.
+
+To use the AWS SAM CLI, you need the following tools.
+
+* AWS SAM CLI - [Install the AWS SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
+* Docker - [Install Docker community edition](https://hub.docker.com/search/?type=edition&offering=community)
+
+1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
+    ``` 
+    git clone https://github.com/aws-samples/serverless-patterns.git
+    ```
+1. Change directory to the pattern directory:
+    ```
+    cd msk-lambda-iam-java-sam
+    ```
+
+## Use the SAM CLI to build and test locally
+
+Build your application with the `sam build` command.
+
+```bash
+sam build
+```
+
+The SAM CLI installs dependencies defined in `kafka_event_consumer_function/pom.xml`, creates a deployment package, and saves it in the `.aws-sam/build` folder.
+
+Test a single function by invoking it directly with a test event. An event is a JSON document that represents the input that the function receives from the event source. Test events are included in the `events` folder in this project.
+
+Run functions locally and invoke them with the `sam local invoke` command.
+
+```bash
+sam local invoke --event events/event.json
+```
+
+## Deploy the sample application
+
+The Serverless Application Model Command Line Interface (SAM CLI) is an extension of the AWS CLI that adds functionality for building and testing Lambda applications. It uses Docker to run your functions in an Amazon Linux environment that matches Lambda. It can also emulate your application's build environment and API.
+
+To use the SAM CLI, you need the following tools.
+
+* SAM CLI - [Install the SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/serverless-sam-cli-install.html)
+* Docker - [Install Docker community edition](https://hub.docker.com/search/?type=edition&offering=community)
+
+To build and deploy your application for the first time, run the following in your shell:
+
+```bash
+sam build
+sam deploy --guided
+```
+
+The first command will build the source of your application. The second command will package and deploy your application to AWS, with a series of prompts:
+
+* **Stack Name**: The name of the stack to deploy to CloudFormation. This should be unique to your account and region, and a good starting point would be something matching your project name.
+* **AWS Region**: The AWS region you want to deploy your app to.
+* **Parameter MSKClusterName**: The name of the MSKCluster
+* **Parameter MSKClusterId**: The unique ID of the MSKCluster
+* **Parameter MSKTopic**: The Kafka topic on which the lambda function will listen on
+* **Confirm changes before deploy**: If set to yes, any change sets will be shown to you before execution for manual review. If set to no, the AWS SAM CLI will automatically deploy application changes.
+* **Allow SAM CLI IAM role creation**: Many AWS SAM templates, including this example, create AWS IAM roles required for the AWS Lambda function(s) included to access AWS services. By default, these are scoped down to minimum required permissions. To deploy an AWS CloudFormation stack which creates or modifies IAM roles, the `CAPABILITY_IAM` value for `capabilities` must be provided. If permission isn't provided through this prompt, to deploy this example you must explicitly pass `--capabilities CAPABILITY_IAM` to the `sam deploy` command.
+* **Disable rollback**: Defaults to No and it preserves the state of previously provisioned resources when an operation fails
+* **Save arguments to configuration file**: If set to yes, your choices will be saved to a configuration file inside the project, so that in the future you can just re-run `sam deploy` without parameters to deploy changes to your application.
+* **SAM configuration file [samconfig.toml]**: Name of the configuration file to store configuration information locally
+* **SAM configuration environment [default]**: Environment for storing deployment information locally
+
+You should get a message "Successfully created/updated stack - <StackName> in <Region>" if all goes well
+
+
+## Test the sample application
+
+Once the lambda function is deployed, send some Kafka messages on the topic that the lambda function is listening on, on the MSK server.
+
+Either send at least 10 messages or wait for 300 seconds (check the values of BatchSize: 10 and MaximumBatchingWindowInSeconds: 300 in the template.yaml file)
+
+Then check Cloudwatch logs and you should see messages for the Cloudwatch Log Group with the name of the deployed Lambda function.
+
+The lambda code parses the Kafka messages and outputs the fields in the Kafka messages to Cloudwatch logs
+
+A single lambda function receives a batch of messages. The messages are received as a map with each key being a combination of the topic and the partition, as a single batch can receive messages from multiple partitions.
+
+Each key has a list of messages. Each Kafka message has the following properties - Topic, Partition, Offset, TimeStamp, TimeStampType, Key and Value
+
+The Key and Value are base64 encoded and have to be decoded. A message can also have a list of headers, each header having a key and a value.
+
+The code in this example prints out the fields in the Kafka message and also decrypts the key and the value and logs them in Cloudwatch logs.
diff --git a/msk-lambda-iam-java-sam/events/event.json b/msk-lambda-iam-java-sam/events/event.json
@@ -0,0 +1,31 @@
+{
+   "records":{
+      "myTopic-0":[
+         {
+            "topic":"myTopic",
+            "partition":0,
+            "offset":250,
+            "timestamp":1678072110111,
+            "timestampType":"CREATE_TIME",
+            "value":"Zg==",
+            "headers":[
+
+            ]
+         },
+         {
+            "topic":"myTopic",
+            "partition":0,
+            "offset":251,
+            "timestamp":1678072111086,
+            "timestampType":"CREATE_TIME",
+            "value":"Zw==",
+            "headers":[
+
+            ]
+         }
+      ]
+   },
+   "eventSource":"aws:kafka",
+   "eventSourceArn":"arn:aws:kafka:us-west-2:123456789012:cluster/MSKWorkshopCluster/a93759a9-c9d0-4952-984c-492c6bfa2be8-13",
+   "bootstrapServers":"b-2.mskworkshopcluster.z9kc4f.c13.kafka.us-west-2.amazonaws.com:9098,b-3.mskworkshopcluster.z9kc4f.c13.kafka.us-west-2.amazonaws.com:9098,b-1.mskworkshopcluster.z9kc4f.c13.kafka.us-west-2.amazonaws.com:9098"
+}
diff --git a/msk-lambda-iam-java-sam/example-pattern.json b/msk-lambda-iam-java-sam/example-pattern.json
@@ -0,0 +1,90 @@
+{
+    "title": "AWS Lambda function subscribed to an Amazon MSK topic using IAM auth",
+    "description": "Creates a Lambda function that uses an Amazon MSK topic as an event source with IAM authentication.",
+    "language": "Java",
+    "level": "200",
+    "framework": "SAM",
+    "introBox": {
+        "headline": "How it works",
+        "text": [
+            "This pattern provides a Lambda function along with an Event Source Mapping to a Kafka topic.",
+            "It requires that you already have an Amazon Managed Streaming for Kafka (Amazon MSK) cluster setup with a topic created. ",
+            "If you don't already have an MSK cluster, you can use the example in this pattern https://serverlessland.com/patterns/msk-cfn-sasl-lambda (linked in the resources) to deploy a cluster.",
+            "This pattern works with either a Provisioned or Serverless MSK cluster as long as the cluster is configured to use IAM authentication. ",
+            "For detailed deployment instructions instructions see the README."
+        ]
+    },
+    "gitHub": {
+        "template": {
+            "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/msk-lambda-iam-java-sam",
+            "templateURL": "serverless-patterns/msk-lambda-iam-java-sam",
+            "projectFolder": "msk-lambda-iam-java-sam",
+            "templateFile": "template.yaml"
+        }
+    },
+    "resources": {
+        "bullets": [
+            {
+                "text": "Amazon MSK cluster pattern",
+                "link": "https://serverlessland.com/patterns/msk-cfn-sasl-lambda"
+            },
+            {
+                "text": "Using AWS Lambda with Amazon MSK",
+                "link": "https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html"
+            },
+            {
+                "text": "AWS CloudFormation Provisioned MSK cluster reference",
+                "link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-cluster.html"
+            },
+            {
+                "text": "AWS CloudFormation Serverless MSK cluster reference",
+                "link": "https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-msk-serverlesscluster.html"
+            }
+        ]
+    },
+    "deploy": {
+        "text": [
+            "sam deploy --guided"
+        ]
+    },
+    "testing": {
+        "text": [
+            "See the GitHub repo for detailed testing instructions."
+        ]
+    },
+    "cleanup": {
+        "text": [
+            "Delete the template: <code>sam delete</code>."
+        ]
+    },
+    "authors": [
+      {
+        "name": "Indranil Banerjee",
+        "bio": "AWS - Senior Solutions Architect",
+        "linkedin": "https://www.linkedin.com/in/indranil-banerjee-b00a261/"
+      },
+      {
+        "name": "Vaibhav Jain",
+        "bio": "AWS - Sr. Application Architect",
+        "image": "https://media.licdn.com/dms/image/C4E03AQEqzZWHGT4dBQ/profile-displayphoto-shrink_800_800/0/1580165399872?e=1687392000&v=beta&t=zdxENLnqCpqCz9i1Uf5Yx4YXlR9EYvgxP8N5UTsy6J8",
+        "linkedin": "https://www.linkedin.com/in/vaibhavjainv/"
+      },
+      {
+        "name": "Paveen Allam",
+        "bio": "Senior Solutions Architect",
+        "image": "https://www.fintail.me/images/pa.jpg",
+        "linkedin": "https://www.linkedin.com/in/pallam/"
+      }, 
+      {
+        "name": "Suraj Tripathi",
+        "bio": "AWS - AppDev Cloud Consultant",
+        "linkedin": "https://www.linkedin.com/in/suraj-tripathi-01b49a140/"
+      },
+      {
+        "name": "Adam Wagner",
+        "bio": "AWS - Principal Serverless Solutions Architect",
+        "linkedin": "https://www.linkedin.com/in/adam-wagner-4bb412/"
+      }
+    ]
+  }
+
diff --git a/msk-lambda-iam-java-sam/kafka_event_consumer_function/.gitignore b/msk-lambda-iam-java-sam/kafka_event_consumer_function/.gitignore
@@ -0,0 +1 @@
+/target/
diff --git a/msk-lambda-iam-java-sam/kafka_event_consumer_function/.settings/org.eclipse.jdt.core.prefs b/msk-lambda-iam-java-sam/kafka_event_consumer_function/.settings/org.eclipse.jdt.core.prefs
@@ -0,0 +1,8 @@
+eclipse.preferences.version=1
+org.eclipse.jdt.core.compiler.codegen.targetPlatform=11
+org.eclipse.jdt.core.compiler.compliance=11
+org.eclipse.jdt.core.compiler.problem.enablePreviewFeatures=disabled
+org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning
+org.eclipse.jdt.core.compiler.problem.reportPreviewFeatures=ignore
+org.eclipse.jdt.core.compiler.release=disabled
+org.eclipse.jdt.core.compiler.source=11
diff --git a/msk-lambda-iam-java-sam/kafka_event_consumer_function/pom.xml b/msk-lambda-iam-java-sam/kafka_event_consumer_function/pom.xml
@@ -0,0 +1,67 @@
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
+	<modelVersion>4.0.0</modelVersion>
+	<groupId>com.amazonaws.services.lambda.samples.events.msk</groupId>
+	<artifactId>MSKConsumer</artifactId>
+	<version>1.0</version>
+	<packaging>jar</packaging>
+	<name>A sample Lambda MSK consumer</name>
+	<properties>
+		<maven.compiler.source>11</maven.compiler.source>
+		<maven.compiler.target>11</maven.compiler.target>
+	</properties>
+
+	<dependencies>
+		<dependency>
+			<groupId>com.amazonaws</groupId>
+			<artifactId>aws-lambda-java-core</artifactId>
+			<version>1.2.1</version>
+		</dependency>
+		<dependency>
+			<groupId>com.amazonaws</groupId>
+			<artifactId>aws-lambda-java-events</artifactId>
+			<version>3.11.0</version>
+		</dependency>
+        <dependency>
+    <groupId>com.google.code.gson</groupId>
+    <artifactId>gson</artifactId>
+    <version>2.10.1</version>
+</dependency>
+
+		<dependency>
+			<groupId>org.apache.kafka</groupId>
+			<artifactId>kafka-clients</artifactId>
+			<version>3.4.0</version>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter-api</artifactId>
+			<version>5.6.0</version>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.junit.jupiter</groupId>
+			<artifactId>junit-jupiter-engine</artifactId>
+			<version>5.6.0</version>
+			<scope>test</scope>
+		</dependency>
+	</dependencies>
+
+	<build>
+		<plugins>
+			<plugin>
+				<groupId>org.apache.maven.plugins</groupId>
+				<artifactId>maven-shade-plugin</artifactId>
+				<version>3.2.4</version>
+				<configuration></configuration>
+				<executions>
+					<execution>
+						<phase>package</phase>
+						<goals>
+							<goal>shade</goal>
+						</goals>
+					</execution>
+				</executions>
+			</plugin>
+		</plugins>
+	</build>
+</project>