AWS Kinesis Logging with Logback Classic and RBN’s aws­kinesis­logback

Amazon enables you to log to its streaming service, AWS Kinesis. Using Logback Classic and SLF4J, we developed a maven plugin for Spring Boot Framework.

AWS Kinesis is Amazon’s streaming service that scales elastically to provide unlimited storage. Behind AWS Kinesis is a DynamoDB table, AWS’ NoSQL solution. A Kinesis stream of one shard can accommodate 1,000 writes per second with a total capacity of 1MB/s. The total capacity of the stream is the sum of the capacities of its shards. The stream never stops accumulating data, which is likely to be used for analytics and reporting. With tools like Splunk, the data can be viewed in real time. Splunk has an AWS Kinesis Module that’s installed on Amazon’s EC2 instances and allows real-time reporting and analytics on data in selected Kinesis Streams.

At RightBrain Networks we deal with Java logging in frameworks like Spring. We use Logback Classic with Simple Logging Facade for Java (SLF4J), which enables us to switch between logging frameworks (Logback, Log4J, etc) at deployment time. Amazon had developed the Kinesis appender for Log4J. We adapted the code used to stream to Kinesis to work with Logback Classic.

Logback Kinesis Appender appends application log messages to Kinesis Stream using an asynchronous thread pool. The Kinesis Appender is highly configurable. Parameters such as max thread count and buffer size are passed through xml configuration file logback.xml. It can be configured to work with a stand-alone Java application as well.

As I was developing, I has to use the classes within different applications, so I packaged them into a maven repo and published them.

RBN’s aws-kinesis-logback has included dependency for aws-java-sdk-kinesis-1.10.2.jar that contains AWS Kinesis Client jar.

Resources for aws-kinesis-logback:

Java sources and Java docs


Install with Gradle:
compile group: ‘com.rightbrainnetworks’, name: ‘aws-kinesis-logback’, version: 1.0
Logging with AWS Kinesis, Logback Classic and RBN’s aws-kinesis-logback