Send JSON objects to HTTP Event Collector using our .NET Logging Library

Recently we shipped a bunch of logging libraries at the same time our new HTTP Event Collector hit the streets: http://blogs.splunk.com/2015/10/06/http-event-collector-your-direct-event-pipe-to-splunk-6-3/

One of the questions I’ve heard from customers using the libraries, is “Can I send JSON objects with the .NET logging library?”

Yes, you can. To do it, you need to use our Splunk.Logging.Common library which our other loggers depend on. Interfaces like TraceListener were designed for sending strings not objects.

For example TraceSource has a TraceData method which accepts objects and which it appears should work. However (at least based on my testin)g the objects are serialized to strings and then passed on as such to the listeners. Thus by the time we get it we have a string message, not an object. We considered trying to detect if the string is a JSON object by trying to deserialize it, but that felt messy and against the spirit of the TraceListener interface.

You can however send objects using our underlying Splunk.Logging.Common library. The library also contains all the robust retry and batching logic. Our TraceListener and SLAB libraries are literally facades on top.

This library fully supports serailizable objects including Strongly typed objects, Anonymous types, and dynamic/JObject.

Here’s a snippet showing sending an Anonymous type and a dynamic.

var middleware = new HttpEventCollectorResendMiddleware(100);

var ecSender = new HttpEventCollectorSender(new Uri(“https://localhost:8088″), 

    “3E712E99-63C5-4C5A-841D-592DD070DA51″, 

    null,

    HttpEventCollectorSender.SendMode.Sequential, 

    0,

    0,

    0,

    middleware.Plugin

);

ecSender.OnError += o => Console.WriteLine(o.Message);

ecSender.Send(Guid.NewGuid().ToString(), “INFO”, null, new { Foo = “Bar” });

dynamic obj = new JObject();

obj.Bar = “Baz”;

ecSender.Send(Guid.NewGuid().ToString(), “INFO”, null, (JObject) obj);

await ecSender.FlushAsync();

Assuming you have the proper Uri and Token and Event Collector is reachable, running the code will send events similar to the following into Splunk. Notice below the “data” field’s value is a JSON object, not a string 😉

In order to use the library directly you have to pass a bunch of information which we handle for you when you use the higher level libraries. We’ll working on making this easier.

As to what the code is doing:

• Creates middleware. In the lower levels, our logging lib uses middleware to handle automatically resending if it is not able to send. We ship a default HttpEventCollectorResendMiddleware in the box which uses an incremental back-off retry policy. Here we are creating that middleware and configuring it to do 100 retries.
• We pass the Uri and Token.
• The metadata param is set to null as it is optional.
• We set the send mode to sequential. Setting it to parallel will send at a higher throughput rate, but the events may not show up in sequence in Splunk. Sequential is the default that we use for our HttpEventCollectorTraceListener and our HttpEventCollectorSink
• The next 3 parameters relate to batching, which can all be defaulted to 0.
• The last parameter accepts a delegate which the middleware exposes a Plugin property.
• It wires up to the OnError event so that you can see any errors that might occur.
• Calls Send passing in an anonymous object.
• Creates a Object and passes it calling send.
• Calls FlushAsync to force the sender to flush the events to HttpEventCollector.

Using this approach you can easily send JSON objects to Splunk!

You can download the code for this project by cloning my repository here: https://github.com/glennblock/SendJsonToHec

Let us know if it works for you.