Java Syslog-RELP Developer Example

Overview

Java syslog-RELP Developer Example project guides through design considerations with an actual example. The example uses syslog with Reliable Event Logging Protocol (RELP) in Java programming language.

The project also utilizes Teragrep Java RELP Library (rlp_01). Rlp_01 ensures reliable record processing in Java applications. It uses optional TLS encryption and public-key certificate verification.

Background

Syslog is a record format standard for message logging. It allows separation of:

• the software that produces message records,

• the system that stores them, and

• the software that reports and analyzes them.

When you use syslog as a record format, you can ensure the produced records are accessible.

RELP is a transport protocol. It allows reliable, encrypted and robust transmission of syslog records. With RELP, you can integrate with many syslog based systems. The mechanism is reliable for transporting the syslog messages over network.

Syslog and RELP allow a real-time record streaming, from an application to an archival and analytics system. Take Teragrep as an example.

Example about application logging

Application is producing many record streams containing valuable information about:

• requests, including transaction processing

• authentication

• authorization

• session handling

• functional errors

• technical errors

• debugging

• runtime metric data

All these are separate record streams from a single application. The application could, for example, handle web-shop orders.

Syslog record format allows capturing all these into their own streams. Meanwhile, the origin of the information is kept in the record metadata.

Example records for preceding example

Requests

<15>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-requests - - - 127.0.0.1 - - [01/Jan/2023:00:00:00 +0300] "GET /webshop/ HTTP/1.1" 200 1995 "https://referer.example.com" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0"

Authentication

<15>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-auth - - - {"event": "login", "source": "127.0.0.1:38238", "username": "user1", "authentication method": "password"}

Authorization

<15>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-authz - - - {"SUBJECT": "user:admin1", "VERB": "ASSUME", "OBJECT": "role:Administrators", "STATUS": "assume-ok"}

Session handling

<16>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-session - - - login <[user1]>, operation <create>, object_type <session>

Functional error

<12>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-logic-err - - - Checkout amount negative, calculated as <-100€>. discount coupons given: "2023FREE" by user: "user1" rejecting purchase;

Technical error

<13>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-tech-err - - - Database connection to 127.0.0.2:3306 timed out.

Debugging

<17>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-debug - - - Ensuring my code works properly enuf

Runtime metric data

<17>1 2023-01-01T00:00:00.123456+00:00 app-server1.example.com web-shop-metric - - - "time_taken":348

As seen above, syslog is able to capture all information. At the same time, the payload part of the record keeps information as human-readable. Delivery and retention of this kind of information is essential for the business.

By following this guide, you can ensure that records:

• are handled in a proper manner,

• contain essential metadata and

• are delivered without loss and in real-time from the application for further processing.

Ready-Made Application Integrations for Syslog-RELP

For logging use, Teragrep ships following libraries:

• Relp Logging plugin for Logback

• Java Util Logging RELP Handler

• Log4j 1.x Log4j RELP Plugin

• Log4j 2.x Log4j2 RELP Plugin

You can use these libraries as part of the relevant logging framework. They integrate with their relevant logging framework by extending existing capabilities.

Note	Ready made libraries work well for the logging use. However, you may want more control over the syslog and RELP in other integrations (i.e. send the set the timestamp or uuid of the record). For this, see the next section.

Syslog-RELP Integration Guidelines

General

• test cases must be executable

• errors (executions) must not be hidden

• server port needs to be configurable

• server address needs to be configurable

Syslog

Headers

• Wrap messages in syslog envelope

• Syslog-hostname must be configurable

• Syslog app-name must be configurable

Note	Sharing same syslog-hostname and syslog-app names allow parallel deployments of the application to share the same processing rules.

// see SyslogRecordTest.java

Structured-data

• Include origin data in syslog structure-data

• Include original uuid in syslog structured-data

Note	Structured-data is a managed schema. Adding new fields with our 48577 is not a good idea.

// see SyslogRecordWithSDTest.java

RELP

Non-functional requirements

• Retry always and do not lose messages

• Processing mustn’t hang infinitely due to technical issues, but rather retry new connection

// see CompleteExampleTest.java

RELP TLS

See TlsClientTest.java on rlp_03 repository.

Contributing

You can involve yourself with our project by opening an issue or submitting a pull request.

Contribution requirements:

1. All changes must be accompanied by a new or changed test. If you think testing is not required in your pull request, include a sufficient explanation as why you think so.

2. Security checks must pass

3. Pull requests must align with the principles and values of extreme programming.

4. Pull requests must follow the principles of Object Thinking and Elegant Objects (EO).

Read more in our Contributing Guideline.

Contributor License Agreement

Contributors must sign Teragrep Contributor License Agreement before a pull request is accepted to organization’s repositories.

You need to submit the CLA only once. After submitting the CLA you can contribute to all Teragrep’s repositories.