KCL Protocol

Some protocol documentation is here: https://github.com/awslabs/amazon-kinesis-client/blob/master/src/main/java/com/amazonaws/services/kinesis/multilang/package-info.java

Input/output

• consume from stdin
• control responses to stdout
• error output to stderr

The protocol itself is request/response based. Except when checking pointing, the client always responds with a status action.

A typical request/reply flow:

1. REQ: action=processRecords
2. RESP: action=status

A checkpointed request/reply flow:

1. REQ: action=processRecords
2. RESP: action=checkpoint
3. REQ: action=checkpoint
4. RESP: action=status

Requests from Daemon

Initialize processor

{"action":"initialize","shardId":"shardId-123"}

Process batch of records

{"action":"processRecords","records":[{"data":"bWVvdw==","partitionKey":"cat","sequenceNumber":"456"}]}

Graceful terminate processor. Should checkpoint final work.

{"action":"shutdown","reason":"TERMINATE"}

Rough terminate processor. Do not checkpoint final work.

{"action":"shutdown","reason":"ZOMBIE"}

Checkpointing

Checkpoint all consumed messages

{"action": "checkpoint", "checkpoint": null}

Checkpoint to sequence number

{"action": "checkpoint", "checkpoint": 1234}