Show Me Your Viz!

Have you just download Splunk 6.4 and asked yourself what’s new and awesome? Have you ever built a dashboard with a custom visualization and wanted to share that with someone or easily replicate it somewhere else? Have Splunk’s core visualizations dulled your senses?

Reader, please meet Splunk 6.4 Custom Visualizations. Are you besties yet? If not, you two will be making sweet love by the end of this article.

I’m going to walk you through a Custom Visualization app I recently wrote and lay it all out there. I’m going to talk about why building these visualizations in Simple XML and HTML are a pain in your ass and how the API’s make your life easier. I’m going to show you techniques I learned the hard way so you can accelerate the creation of your app. Does all that sound good? Great, let’s get started.

First, a little backstory…

I recently presented a Developer session at Splunk Live San Francisco this past March. After the session I had a customer come up to me and ask a really simple question. How do you plot single values on a Splunk map? The simple answer is you don’t. But that’s a really shitty answer especially since there is no way to natively do it. Our maps were built to leverage aggregate statistics clustered into geographic bins from the geostats command. The output of that are bubbles on a map that optionally show multiple series of data (like a pie chart) if you use a split by clause in your search. If you’ve ever tried to plot massive amounts of categories using the argument globallimit=0, your system will come to a grinding halt as it spirals into javascript induced browser death. I thought about the problem for a while and figured there had to be a better way.

Less than a month later, in the span of a week, I built an app that solves this problem. The best part is it’s distributable on Splunkbase and can be used by anyone running a Splunk 6.4 search head. If you’re a Javascript pro then I’m confident you can build a visualization app in a couple days. Here’s how I did it and what I learned.

Step 0 – Follow The Docs

The docs are quite good for a new feature. Use them as a definitive reference on how to build a Custom Visualization app. Use this article to supplement the docs for some practical tips and tricks.

Step 1 – Put Splunk Into Development Mode

This will prevent splunk from caching javascript and won’t minify your JS or CSS. This allows you to just refresh the browser and re-load any changes you made without having to restart Splunk.

Create $SPLUNK_HOME/etc/system/local/web.conf with the following settings and restart Splunk.

[settings]
minify_js = False
minify_css = False
js_no_cache = True
cacheEntriesLimit = 0
cacheBytesLimit = 0
enableWebDebug = True

Step 2 – Create The App Structure

Follow this section from the docs. There’s a link to a template that has the requisite app structure and files you’ll need to be successful. The only thing it’s missing is the static directory to house your app icons if you decide to post the app on Splunkbase and want it to look pretty.

Step 3 – Create The Visualization Logic

Step 3a – Include Your Dependences

Here’s where we’ll spend the bulk of our time. This step is where you install any dependencies that your visualization relies on and write your code to display your viz.

You may be asking yourself when I’m going to get to some of the relevant points I mentioned above; specifically how using this API makes your life easier. Here’s where there rubber meets the road. We’re managing dependencies a little differently this time around. If you’ve built a custom visualization Splunk 6.0-6.3 you’ve done it using one of two methods. The first is converting your Simple XML dashboard into HTML. This works well but isn’t guaranteed to be upgrade friendly. The second method is loading javascript code from Simple XML. If you’ve used either method you’ll likely have run into RequireJS. We use RequireJS under the covers in Splunk to manage the loading of Javascript libraries. It works but it’s a major pain in the ass and it’s a nightmare when you have non-AMD compliant modules or conflicting version dependencies for modules that Splunk provides. I come from a Python world where importing dependencies (modules) is easy. Call me simplistic or call me naive, but why shouldn’t Javascript be so simple?

The Custom Visualization framework makes dealing with dependencies a lot easier by leveraging npm and webpack. This makes maintaining and building your application a lot easier than trying to do things in RequireJS. Use npm to download dependencies with a package.json (or manually install with npm install) and webpack will build your code and all the dependencies into a single visualization.js file that the custom viz leverages. This code will integrate smoothly with any dashboard and you won’t run into conflicts like you may have in the past with RequireJS.

The visualization I built requires a couple libraries; Leaflet and a plugin called Leaflet.markercluster.

Here’s what it looked like to load these libraries using RequireJS in an HTML dashboard within an app called ‘leaflet_maps’. Luckily, Leaflet doesn’t require any newer versions of Jquery or Underscore than are provided by Splunk. I’ve had to shelve an app I want to build because of RequireJS and the need for newer versions of Jquery and Lodash (modified Underscore). If you’re a RequireJS pro you may be screaming at me to use Multiversion support in RequireJS. I’ve tried it unsuccessfully. If you can figure it out, please let me know what you did to get it working.

require.config({
    baseUrl: "{{SPLUNKWEB_URL_PREFIX}}/static/js",
    paths: {
        'leaflet': '/static/app/leaflet_maps/js/leaflet-src',
        'markercluster': '/static/app/leaflet_maps/js/leaflet.markercluster-src',
        'async': '/static/app/leaflet_maps/js/async',
    },
    shim: {
        leaflet: {
            exports: 'L'
        },
        markercluster: {
            deps: ['leaflet']
        }
    }
});

This piece of code literally took me half a day to figure out. Things are easy in RequireJS if your module is AMD compliant. If it isn’t, like Leaflet.markercluster, you have to shim it. The bottom line is it’s a pain in the ass and difficult to get working. It took a lot of Google searching and digging through docs.

Here’s what it looks like using npm and webpack.

npm config – package.json

{
  "name": "leaflet_maps_app",
  "version": "1.0.0",
  "description": "Leaflet maps app with Markercluster plugin functionality.",
  "main": "visualization.js",
  "scripts": {
    "build": "node ./node_modules/webpack/bin/webpack.js",
    "devbuild": "node ./node_modules/webpack/bin/webpack.js --progress",
    "watch": "node ./node_modules/webpack/bin/webpack.js -d --watch --progress"
  },
  "author": "Scott Haskell",
  "license": "End User License Agreement for Third-Party Content",
  "devDependencies": {
    "imports-loader": "^0.6.5",
    "webpack": "^1.12.6"
  },
  "dependencies": {
    "jquery": "^2.2.0",
    "underscore": "^1.8.3",
    "leaflet": "~1.0.0-beta.2"
  }
}

This is the same package.json provided in the sample app template. The only things I modified were the name, author, license, devDependencies and dependencies. The important dependencies are imports-loader and leaflet. Leaflet.markercluster is available via npm but it’s an older version that was missing some features I needed so I couldn’t include it here. Now all I need to do is have nodejs and npm installed and run ‘npm install’ in the same directory as the package.json ($SPLUNK_HOME/etc/apps/leaflet_maps_app/appserver/static/visualizations/leaflet_maps). This creates a node_modules directory with your dependencies code.

webpack config – webpack.config.js

var webpack = require('webpack');
var path = require('path');

module.exports = {
    entry: 'leaflet_maps',
    resolve: {
        root: [
            path.join(__dirname, 'src'),
        ]
    },
    output: {
        filename: 'visualization.js',
        libraryTarget: 'amd'
    },
    module: {
        loaders: [
            {
                test: /leaflet\.markercluster-src\.js$/,
                loader: 'imports-loader?L=leaflet'
            }
        ]
    },
    externals: [
        'vizapi/SplunkVisualizationBase',
        'vizapi/SplunkVisualizationUtils'
    ]
};

Again, this is the same file provided in the template app. The difference here is the ‘loaders’ section of the ‘module’ definition. I’m using the webpack imports-loader to shim the Leaflet.markercluster module since it’s not AMD compliant. This is analogous to the RequireJS shim code I provided above. The difference here is that it’s much more intuitive (once you figure out you need imports-loader) to shim in webpack. The test key is a regex that matches the Leaflet.markercluster source file. The loader key defines the dependency on the function ‘L’ which is exported in the leaflet library.

Lastly, here’s the one small portion of RequireJS that you have to touch in your source.

define([
            'jquery',
            'underscore',
            'leaflet',
            'vizapi/SplunkVisualizationBase',
            'vizapi/SplunkVisualizationUtils',
            '../contrib/leaflet.markercluster-src'
        ],
        function(
            $,
            _,
            L,
            SplunkVisualizationBase,
            SplunkVisualizationUtils
        ) {

I’ve created a contrib directory and added some supporting Javascript and CSS files. I’ve defined my leaflet module and it’s L function as well as the leaflet.markercluster source location in contrib. Notice that since this leaflet.markercluster is not AMD compliant I don’t need to define a function.

Now all you have to do is build the code using npm.

bash-3.2$ cd $SPLUNK_HOME/etc/apps/leaflet_maps_app/appserver/static/visualizations/leaflet_maps
bash-3.2$ npm run build

> leaflet_maps_app@1.0.0 build /opt/splunk/etc/apps/leaflet_maps_app/appserver/static/visualizations/leaflet_maps
> node ./node_modules/webpack/bin/webpack.js

Hash: 9ea37b6ef76197f0a3b7
Version: webpack 1.12.14
Time: 511ms
           Asset    Size  Chunks             Chunk Names
visualization.js  649 kB       0  [emitted]  main
   [0] ./src/leaflet_maps.js 7.39 kB {0} [built]
    + 6 hidden modules

Any time you make subsequent changes to your source just re-run the build and re-fresh Splunk.

I built this app on CentOS 7 in a docker image. I have npm and node installed in the docker image but it’s also possible to leverage node that gets shipped with Splunk. You’d just tweak this section of your package.json.

"scripts": {
    "build": "$SPLUNK_HOME/bin/splunk cmd node ./node_modules/webpack/bin/webpack.js",
    "devbuild": "$SPLUNK_HOME/bin/splunk cmd node ./node_modules/webpack/bin/webpack.js --progress",
    "watch": "$SPLUNK_HOME/bin/splunk cmd node ./node_modules/webpack/bin/webpack.js -d --watch --progress"
  },

Then export the path to your Splunk 6.4 install as SPLUNK_HOME.

bash-3.2$ export SPLUNK_HOME=/opt/splunk

Your visualization code and all dependencies are now built into a single file called visualization.js.

Step 3b – Write The Code

If you’re using the app template and following the docs then you’ll be modifying the file /appserver/static/visualizations//src/visualization_source.js to place your code. There are a bunch of methods in the API that will be relevant.

The first method is updateView

The updateView method is where you stick your custom code to create your visualization. There’s nothing super fancy going on and the docs do a great job explaining what needs to be done here. One important thing to cover is how to handle searches that return > 50,000 results. If you’ve worked with the REST API or written dashboards using the SplunkJS stack you’ll know that you can only get 50,000 results at a time. Things are no different here. It just wasn’t obvious how to do it. I had to dig through the API to figure it out. Here’s how I did it so you don’t have to waste your time.

    updateView: function(data, config) {
        // get data
        var dataRows = data.rows;

        // check for data
        if (!dataRows || dataRows.length === 0 || dataRows[0].length === 0) {
            return this;
        }

        if (!this.isInitializedDom) {
	    // more initialization code here
	    this.chunk = 50000;
	    this.offset = 0;
	}

	// the rest of your code logic here

	// Chunk through data 50k results at a time
	if(dataRows.length === this.chunk) {
	    this.offset += this.chunk;
	    this.updateDataParams({count: this.chunk, offset: this.offset});
	}
    }

I initialize a couple variables; offset and chunk. I then check to see if I get a full 50k events back. If so, I increment my offset by the chunk size and update my data params. This will continue to page through my results set, calling updateView each time and synchronously running back through the code, until I get < 50k events. It's straightforward but not documented anywhere.

This leads us to the second method getInitialDataParams

This is where you set the output format of your data and how many results the search is limited to.

        // Search data params
        getInitialDataParams: function() {
            return ({
                outputMode: SplunkVisualizationBase.ROW_MAJOR_OUTPUT_MODE,
                count: 0
            });
        },

I set the count to 0 which is an unlimited amount of results. This can be dangerous and could potentially overwhelm your visualization so be sure that it can handle it before you go down this route. Here are the available options and output modes.

/**
         * Override to define initial data parameters that the framework should use to
         * fetch data for the visualization.
         *
         * Allowed data parameters:
         *
         * outputMode (required) the data format that the visualization expects, one of
         * - SplunkVisualizationBase.COLUMN_MAJOR_OUTPUT_MODE
         *     {
         *         fields: [
         *             { name: 'x' },
         *             { name: 'y' },
         *             { name: 'z' }
         *         ],
         *         columns: [
         *             ['a', 'b', 'c'],
         *             [4, 5, 6],
         *             [70, 80, 90]
         *         ]
         *     }
         * - SplunkVisualizationBase.ROW_MAJOR_OUTPUT_MODE
         *     {
         *         fields: [
         *             { name: 'x' },
         *             { name: 'y' },
         *             { name: 'z' }
         *         ],
         *         rows: [
         *             ['a', 4, 70],
         *             ['b', 5, 80],
         *             ['c', 6, 90]
         *         ]
         *     }
         * - SplunkVisualizationBase.RAW_OUTPUT_MODE
         *     {
         *         fields: [
         *             { name: 'x' },
         *             { name: 'y' },
         *             { name: 'z' }
         *         ],
         *         results: [
         *             { x: 'a', y: 4, z: 70 },
         *             { x: 'b', y: 5, z: 80 },
         *             { x: 'c', y: 6, z: 90 }
         *         ]
         *     }
         *
         * count (optional) how many rows of results to request, default is 1000
         *
         * offset (optional) the index of the first requested result row, default is 0
         *
         * sortKey (optional) the field name to sort the results by
         *
         * sortDirection (optional) the direction of the sort, one of:
         * - SplunkVisualizationBase.SORT_ASCENDING
         * - SplunkVisualizationBase.SORT_DESCENDING (default)
         *
         * search (optional) a post-processing search to apply to generate the results
         *
         * @param {Object} config The initial config attributes
         * @returns {Object}
         *
         */

Some other methods you may want to look into are initialize, setupView, formatData and drilldown. If you want to look at all the methods take a look at $SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/js/vizapi/SplunkVisualizationBase.js

Steps 4-7 – Add CSS, Add Configuration Settings, Try Out The Visualization, Handle Data Format Errors

Refer to the docs.

Step 8 – Add User Configurable Properties

You’ll most likely want to give your user an interface to tweak the parameters of the visualization. You HAVE to define these properties in default/savedsearches.conf and README/savedsearches.conf.spec. Follow the docs here and don’t skip this step!

Step 9 – Implement a format Menu

Refer to the docs. If you want to add a default value here’s an example of how you’d do it. Quick warning, I had to strip out the HTML so WordPress wouldn’t mangle it. Have a look at the code in my app if you want a full example.

splunk-radio-input name="display.visualizations.custom.leaflet_maps_app.leaflet_maps.cluster" value="true"

That’s all there is to it! I’m not a Javascript developer and this was pretty damn simple to figure out. I hope you find the experience as enjoyable as I did. If you build something cool, please contribute it back to the community and post the app on Splunkbase.com.