Overview
Message Oriented Middleware (MOM) infrastructures facilitate the sending and receiving of messages between distributed systems. Messages typically get routed to Queues(point to point) or Topics(publish/subscribe) for clients to subscribe to, receive the messages and process them.
In many respects you can think of MOM as the glue that stitches heterogeneous enterprise computing environments together.
Now why am I so interested in this ? Well, MOM and the messages transported represent a massive source of machine data that Splunk can index and resolve into operational visibility on many different levels..core operations , business analytics, transaction tracing etc..
So for some time now I have been pondering creating a solution for Splunk to tap into this source.
Most folks I have engaged with , and myself included, have in the past typically coded up ad-hoc solutions to Splunk messages from Queues and Topics , solutions that are very proprietary to a specific messaging provider and architecture.Not much value in re-use and often rather “string and chewing gum” approaches.
Modular Inputs
With the release of Splunk 5 came a great new feature called Modular Inputs.As soon as I started digging into this feature , I soon realized that this was the perfect platform to build a Messaging Modular Input.
Modular Inputs extend the Splunk framework to define a custom input capability.In many respects you can think of them as your old friend the “scripted input” , but elevated to first class citizen status in the Splunk Manager. Splunk treats your custom input definitions as if they were part of Splunk’s native inputs and users interactively create and update the input via Splunk manager just as they would for native inputs (tcp, files etc…) The Modular Input’s lifecycle, schema, validation, configuration is all managed by Splunk. This is the big differentiator over scripted inputs which are very loosely coupled to Splunk.
Creating a Java Framework
So I chose to implement the Messaging Modular Input in Java.Typically I would have used Python , but as I was planning on using the JMS(Java Message Service) API , Java it was to be.
The first task was to create a Java Modular Inputs framework.This is published on github and freely available for reuse by anyone.Basically the framework takes care of the Modular Input’s plumbing and XML dialogue between Splunk for various states such as validation, schema introspection and the Modular Input’s execution.
Now with the framework in place I coded the JMS Messaging Modular Input.
This is also published on github(if you want the source code) , and now officially released to SplunkBase.
Why choose JMS ?
JMS is simply a messaging interface that abstracts your underlying MOM provider implementation and most MOM vendors support JMS.
So this allowed for creating 1 single modular input that can index messages from :
- MQ Series / Websphere MQ
- Tibco EMS
- ActiveMQ
- HornetQ
- RabbitMQ
- SonicMQ
- JBoss Messaging
- Weblogic JMS
- Native JMS
- StormMQ
- MSMQ (using native bindings via JMS)
- Any other MOM with a JMS provider
The Modular Input is very simple to install and is platform independent.
You simply download it from Splunkbase, drop in your apps directory & restart Splunk.
The configuration of your Queue and Topic data inputs is then performed via Splunk Manager , or directly in inputs.conf where the Splunk UI is not enabled.
Any custom jar files for the JMS provider can simply be dropped in the Messaging Modular Input’s “lib” directly and they will be automatically picked up and loaded.
Manage your JMS Messaging Topic and Queue Inputs via Splunk Manager
Create/Update Messaging Input
Browse Existing Inputs
Operational Visibility from the indexed messages
