Exporting search results with Javascript / node.js

Recently I had a request internally for how to access the Export endpoint from Splunk from a node.js application. The Export endpoint is useful for exporting large amounts of data efficiently out of Splunk as it will stream the results directly rather than requiring you to continually poll for more results. It turns out we don’t support the Export endpoint currently in our JS SDK, but it is very easy do access it yourself using Mikael’s super simple request module.

A picture (or a snippet in this case) tells a thousand words. Below you can see how to export Splunk’s internal index. Once you start it up it will instantly start streaming. Make sure you have enough disk space, or stop it before you run out

var request = require('request');
request.get(
    {
        strictSSL: false,
        uri: 'https://localhost:8089/servicesNS/admin/search/search/jobs/
              export?output_mode=json',
        qs: {
            search: 'search index=_internal'
        }
    }
)
.auth('admin', 'changeme', false)
.pipe(process.stdout);

Here is what is going on above:

• Loads the request module.
• Calls get to issue a GET request passing in the following params:
  • strictSSL – set to false tells request to not validate the server cert returned by Splunk, which by default is not a valid cert.
  • uri – set to the Splunk host along with the path for the export endpoint. I’ve also specified in the query string to force a JSON response.
  • qs – set to supply the search param. Passing it in this way allows me to not have to URI encode the search string as request will do it.\
• Calls auth to use HTTP Basic Auth passing the Splunk username and password.
• Pipe’s the results to stdout

Any questions?