I’m somewhat of a Heroku fan boy. I’ve been using it for some time because it is just so simple to deploy applications. However, I’ve never really looked too deeply into the logs produced by my apps via the command line.
Que Spunk.
In this post we’ll look at how you can start Splunking data from apps deployed in Heroku, and some recipes to visualise it using the SPL.
Prepare Our Indexer
We will send logs from Heroku to our Splunk Indexer via syslog. To configure the Indexer to accept these logs we’ll need to enable receiving:
Settings > Forwarding and Receiving > Recieve Data > Add New
Now all we need to do is select the port via which we want to receive this data. We’ll use port:9997 for this example but you can select an open port of your choice.
Get Heroku Logging
Heroku provides comprehensive app, system, and API logging by default. But lets turn things up a bit and include some debug and runtime logs to help us with app development, after all this is Splunk. Assuming you have the Heroku Toolbelt installed, you can run the command below to enable logging:
heroku config:add LOG_LEVEL=DEBUG --app <YOUR APP NAME>
And…
heroku labs:enable log-runtime-metrics --app <YOUR APP NAME>
Restart your app:
heroku restart --app <YOUR APP NAME>
Then add a syslog drain to send these logs to your indexer:
drains:add syslog://<YOUR INDEXER'S IP>:9997 --app <YOUR APP NAME>
Using AWS? When you use EC2 security groups, the hostname used when you add the drain must resolve to the 10/8 private IP address of your instance (which must be in the us-east Amazon region). If you use the EC2 public IP address, or a name that resolves to the public IP address, then logplex will not be able to connect to your drain.
Heroku meet Splunk, Splunk Heroku
Assuming everything is working correctly we should start to see data being indexed using a simple search:
sourcetype="syslog"
SPL Recipes
@chuckg has written a great set of recipes to start searching and visualising your data. You can find them all on Github here. From application performance metrics to Heroku errors, it’s a really comprehensive list.
Not using Heroku? Not a problem. Most PAAS providers offer logging functionality and are therefore Splunkable. A quick read of their docs should answer this question. I’d be interested to hear from people who have successfully Splunked data from other providers too.