First bite of the Cherry(py)
I didn’t always work at Splunk. In fact, many moons ago I used to be a Splunk customer. At the time we were simply looking for a means to better consolidate our enterprise’s numerous sources of log data into a centralized repository. A colleague of mine mentioned this product called Splunk , and hence the journey began. Like many, this started with getting some log files indexed into Splunk and creating some trivial searches and Simple XML dashboards. This very quickly led to more data sources and more elaborate dashboards. Then the bloke sitting next to me saw what I was doing and wanted in on the action, then the adjacent team and then the floor. This internal viral growth required setting up a simple Splunk cluster under my desk with user/role access controls, data retention policies, backups etc., But more importantly, this was when I really started to become a Splunk Developer.
Appetite for construction
I’m a coder. I like making things. And when it comes to working with other products, platforms & frameworks, I am naturally drawn to those that allow me to open them up and extend, customize and augment them to my specific needs. This doesn’t necessarily mean open source, but they need to have an open architecture with as many developer “hooks” as possible. This openness is also what leads to and drives community and collaboration. A thriving community is the bedrock of a platform’s developer ecosystem, and within this realm great ideas are seeded and emerge.
Even though my Developer journey started with Splunk several versions ago, there were already enough developer hooks in the product to satisfy my needs and allow me to shape Splunk to suit the dynamics of the environment in which I was utilizing it. Proprietary closed platforms force you to adapt to their way. It should be the opposite. You should be able to adapt the platform to your way, your data , your requirements. And this adaptation should be as simple and timely as possible to accomplish. Splunk ticked all my boxes.
All you can eat buffet
The Splunk Developer landscape today is vast and growing, with hooks into numerous areas of the platform.
What follows is an overview of all the main areas where you can currently develop atop the Splunk platform.
In many ways it is great to be spoiled for choice, it adds to the agility of the platform. But sometimes, so much choice can be overwhelming, especially to the newbie developer. So I will provide a brief overview of each development hook, with links to more substantial documentation as well as my take as to why you might consider that particular development option.
CLI (Splunk’s Command Line Interface)
You can use the Splunk CLI to monitor, configure and search Splunk via a terminal/shell interface or wrapping the commands in a shell script.
Consider using this if you are not able to program to the REST API or use a language SDK and are more suited to simple shell commands.
Furthermore, the CLI has some functionality over and above what you can do with REST i.e.: start/stop the server, clean indexes, additional troubleshooting tools and more.
http://docs.splunk.com/Documentation/Splunk/5.0.3/Admin/AbouttheCLI
REST API
You can interact with most of the functionality of Splunk via the exposed REST API. This will typically be managing, searching and inputting data. Just think of your experience using Splunk Web. What you can do there, you can perform programmatically directly via REST.
You can perform this interaction from the command line using a program such as CURL, or programmatically in your code.
You might want to use the REST API directly if you are unable to use our language SDK’s or perhaps you are using a language that we don’t currently have an SDK for i.e.: R, Perl, or you might be hitting a custom REST endpoint that isn’t available via an SDK.
http://dev.splunk.com/view/rest-api-overview/SP-CAAADP8
SDKs (Software Development Kits)
Our language SDK’s build upon the underlying REST API by providing a higher level language interface, currently in 6 different language offerings (Python, Java, JavaScript, PHP, Ruby, C#). They make it even simpler to manage, search and input data into Splunk by abstracting all the REST plumbing so you can instead focus your efforts on productive coding in the language that best suits your development needs.
You’ll typically want to use an SDK by leveraging your existing developer language skills to utilize the Splunk REST API to code solutions that integrate with Splunk, be it a data only integration or perhaps a custom user interface, or you may have a requirement to utilize the core Splunk platform to build a standalone big data solution on top off it. The reality is, the potential use cases are numerous.
http://dev.splunk.com/view/sdks/SP-CAAADP7
Splunkbase Apps and Add-ons
At a very simple level Splunk apps & add-ons are a packaging up of the various configurations, searches, knowledge objects, UI components and customizations, inputs, role definitions, field extractions etc. that you might typically create via Splunk Web. An App will typically have a user interface that sits atop multiple other Splunk objects. An add-on typically serves a single reusable purpose to extend the Splunk platform i.e.: a custom input, and won’t have a UI or a setup screen. Apps are typically themed around a specific use case i.e.: Splunk for Active Directory, whereas an add-on will generally be generic and reusable across many diverse use cases i.e.: SNMP Modular Input.
You’ll want to create an app or add-on if you have something you want to share with the community (free or charged) on Splunkbase.
The power of the community working together is a great thing, you might contribute an app, and conversely someone else might have contributed an app that you can benefit from. Everyone wins. Furthermore, it promotes modularity and reuse, all good things for Splunk productivity. Also , many Splunk partners create apps to integrate with their products , hence leveraging these connected communities for the market.
You might also choose to create apps at your organization and share these internally. Building out your internal Splunk apps not only promotes modularity but makes it easier to secure access to a multi user/multi departmental environment via Splunk user/role based permissions and access policies.
You can also bundle up code that isn’t a traditional Splunkbase app by definition, but is something you want to share on Splunkbase i.e.: a code library or perhaps something you have created using an SDK.
http://docs.splunk.com/Documentation/Splunk/5.0.3/AdvancedDev/AppIntro
Scripted inputs
Out of the box, Splunk has simple generic input options available for getting data from a file or receiving data over TCP/UDP.
But you can also create your own custom scripts, in any language, to obtain data from any source. These are called scripted inputs.
You can then bundle these up as Splunk add-ons, share them on Splunkbase and also browse Splunkbase for any scripted inputs others have created.
You’ll want to create a scripted input when the data is not available by file or UDP/TCP, or perhaps you want to perform some additional pre-processing of the raw data before sending it to Splunk. Scripted inputs have been largely superseded since Splunk 5.0 with Modular Inputs, but their bare bones simplicity and speed of development still make them an effective tool in the developer’s arsenal. And if you have a Splunk environment pre 5.0, then Modular Inputs are obviously not available.
http://docs.splunk.com/Documentation/Splunk/5.0.3/AdvancedDev/ScriptedInputsIntro
Modular Inputs
Modular Inputs build upon Scripted Inputs by elevating the creation of custom input add-ons to first class citizen status in Splunk. By that I mean that once you install them , it is as if they are natively part of Splunk, just like File or TCP inputs. You are still achieving the same basic premise as with Scripted Inputs , providing some custom, reusable way of getting at data , but a Modular Input is tightly integrated into the Splunk lifecycle(install , logging, validation , runtime, setup page in the Splunk Manager UI, manageable via REST etc..) and the experience for the end user is much simpler and easier to configure access to their data.
I would advocate creating a Modular Input if you are using Splunk 5+ and wish to provide the simplest and most seamless experience for your users in setting up access to their data sources. And just like with Scripted Inputs , you can use any language you like , although I’d generally recommend Python as Splunk comes with it’s own Python interpreter and this should provision you Modular Input as being platform independent.
http://docs.splunk.com/Documentation/Splunk/5.0.3/AdvancedDev/ModInputsIntro
Custom search commands
Splunk’s Search language is incredibly powerful and extensive for deriving a wide range of analytical insights from your indexed data. But there may well be times when a particularly Search command doesn’t quite do what you want or you have a need for an entirely new search command. The good news is that you can extend the Splunk search language with your own custom commands. These can also be bundled up as Splunk add-ons and shared on Splunkbase. Search commands will be written in Python.
I would create a custom Splunk command when you have need for search processing that is not available out of the box, have a need for frequent reuse of the custom search command logic, and you want to maintain an integrated approach for your users to processing your data by keeping the processing logic as single searches/saved searches. By “integrated approach” I mean that it could alternatively be possible to perform part of the search, output the results via REST and then perform the additional processing from some custom code.
http://docs.splunk.com/Documentation/Splunk/5.0.3/AdvancedDev/SearchScripts
Custom alert scripts
Splunk alerting channels provided by defaults are Email and RSS. But let’s say for example that you wanted to send alerts via SMS, to a Messaging Queue, as an SNMP or directly to a trouble ticket system. Then you can code you own scripted alerts to perform this functionality in any language.
You will pretty much always need to go down this path if you have an alerting requirement beyond Email/RSS. However it would also be possible to code some external alerting program that is scheduled and executes searches / looks up saved search results via the REST API and then sends alerts to some channel based on the your alerting criteria being met.
http://docs.splunk.com/Documentation/Splunk/5.0.3/Alert/Configuringscriptedalerts
Custom REST endpoints
Splunk’s REST API is very thorough, but this can also be extended with your own custom REST endpoints that you could then integrate with programmatically. You would have to use the REST API directly as custom REST endpoints wouldn’t have interfaces in our language SDKs. Custom REST endpoints can also be accessed via an Apps setup page in Splunk Web.
You will typically write a custom REST endpoint when then is specific server side functionality you require that is not exposed or available via the standard REST API.
http://blogs.splunk.com/2011/08/16/bulding-custom-rest-endpoints-conf-2011-demo/
http://docs.splunk.com/Documentation/Splunk/5.0.3/AdvancedDev/SetupExampleCustom
http://docs.splunk.com/Documentation/Splunk/5.0.4/admin/restmapconf
Custom authentication handlers
Splunk ships with 2 authentication mechanisms, internal Spunk authentication and LDAP.
But Splunk also has a scripted authentication API for use with an external authentication system, so you can develop you own authentication handlers.
When you require authentication to external systems other than LDAP, you will need to create a custom authentication handler.
The App Framework
The Splunk App Framework allows you build your own custom MVC apps and components that run inside the Python based Splunk Web Application Server.
The App Framework layer API provides the abstraction needed to easily integrate your application into core Splunk software. The API provides access to the same libraries that power Splunk Web, including the REST API, CherryPy application server, and Mako templating facilities.
You should consider going down this path if you need to create custom Advanced XML modules and View Templates (for presenting a custom UI experience), creating custom controllers(for interactions from the browser) or create custom model components (for interacting with SplunkD). The code you develop can also be bundled up as an app/add-on and shared on Splunkbase i.e.: a package of custom D3 visualizations.
http://dev.splunk.com/view/app-framework/SP-CAAADVF
The “NEW” App Framework
The “NEW” App framework (currently in beta), allows developers to leverage their existing skills with languages and web frameworks that they are already familiar with to create custom Splunk web apps. It is built on top of the Django web framework and incorporates the Splunk Javascript and Python SDKs. The Splunk Application Framework is really a set of mostly-optional layers from which you can pick and choose. The goal of the framework is to provide you with tools to make working with searches and results quick and easy, while offering the greatest flexibility for different developers and their environments.
Choosing this approach is really a question of is this going to make it simpler for developers to create Splunk web apps and is it going to lead to greater productivity for you. The App Framework is very powerful but requires a particularly proprietary skillset to get productive. The “NEW” App Framework is about allowing you to take advantage of cross platform skills you may already have.
http://dev.splunk.com/view/app-framework/SP-CAAAEMA
Coding outside the square
You can also create whatever other tools, utilities & libraries related to Splunk that you want, you don’t have to rely directly on Splunk hooks. And there are already some cool projects on Github created internally and by the developer community. Here’s a sampling of them.
IOS Logging Libraries : https://github.com/carasso/splogger
Splunk Storm Mobile Analytics : https://github.com/nicholaskeytholeong/splunk-storm-mobile-analytics
Android SDK for Splunk : https://github.com/damiendallimore/splunk-sdk-android
SplunkJavaLogging : https://github.com/splunk/splunk-library-javalogging
SplunkJavaAgent : https://github.com/damiendallimore/SplunkJavaAgent
Powershell Resource Kit : https://github.com/splunk/splunk-reskit-powershell
FluentD Connector : https://github.com/parolkar/fluent-plugin-splunk
Flurry Connector : https://github.com/splunk/splunk-flurry
Apache Camel Component for Splunk : https://github.com/pax95/camel-splunk
Spring Integration Adaptors for Splunk : https://github.com/SpringSource/spring-integration-extensions/tree/master/spring-integration-splunk
You really are only limited by your imagination !
A few more morsels to take away
That’s quite a selection of development options isn’t it? Hopefully this has given you a pretty good overview of the landscape that’s currently out there for Splunk developers.
And don’t forget, Splunk’s Worldwide User’s Conference is coming up very soon and there is some great content for developers this year.
Splunk Conf Developer Track : http://conf.splunk.com/sessions
Splunk Conf Hackathon : http://conf.splunk.com/view/conference-hackathon/SP-CAAAG7C
Splunk Conf Contests : http://conf.splunk.com/view/conference-contest/SP-CAAAG3B
Here’s a few other links/resources to further help you out in your Splunk developer journey:
Free Developer License : http://dev.splunk.com/page/developer_license_sign_up
Splunk Dev : http://dev.splunk.com
Splunk Docs : http://docs.splunk.com
Github : https://github.com/splunk
Twitter : @splunkdev , @damiendallimore