Configuring Nginx With Splunk, REST API & SDK Compatibility

Last year I posted an article on how to configure HAProxy with Splunk, REST API & SDK compatibility. Yesterday, I posted an article on how to configure Nginx as a load balancer in front of a tier of HTTP Event Collectors. Today, I want to iterate on the work I did yesterday and show a basic config for Nginx that’s compatible with Splunk, the REST API and SDK’s.

You’re going to need to build or install a version of Nginx that enables HTTPS support for an HTTP server.

./configure --with-http_ssl_module

If you install from source and don’t change the prefix then you’ll have everything installed in /usr/local/nginx. The rest of the article will assume this is the install path for Nginx.

Once you’ve got Nginx installed you’re going to need to configure a few key items. First is the SSL certificate. If you’re using the default certificate that ships with Splunk then you’ll need to copy $SPLUNK_HOME/etc/auth/server.pem and place that on your load balancer. I’d highly encourage you to generate your own SSL certificate and use this in place of the default certificate. Here are the docs for configuring Splunk to use your own SSL certicicate.

The following configuration assumes you’ve copied server.pem to /usr/local/nginx/conf.

    server {
        listen 8089 ssl;
        listen 8000;

        ssl_certificate     server.pem;
        ssl_certificate_key server.pem;

        location / {
            proxy_pass http://splunkweb;
        }

        location /services {
            proxy_pass https://splunkrest;

        }
    }

Next we’ll configure the upstream servers. If you’re using the open source version of Nginx you’ll need to use the IP Hash method for session persistence. If you’re using the commercial version Nginx Plus, you have more options for session persistence methods. Add as many servers as you have to each of the upstream blocks. I used two to illustrate that you can add N servers.

    upstream splunkweb {
        ip_hash;
        server splunk-server-1:8000;
        server splunk-server-2:8000;
    }

    upstream splunkrest {
        ip_hash;
        server splunk-server-1:8089;
        server splunk-server-2:8089;
    }

Now let’s put it all together in a working nginx.conf

worker_processes  auto;

events {
    worker_connections  1024;
}


http {
    upstream splunkweb {
        ip_hash;
        server splunk-server-1:8000;
        server splunk-server-2:8000;
    }

    upstream splunkrest {
        ip_hash;
        server splunk-server-1:8089;
        server splunk-server-2:8089;
    }

    server {
        listen 8089 ssl;
        listen 8000;

        ssl_certificate     server.pem;
        ssl_certificate_key server.pem;

        location / {
            proxy_pass http://splunkweb;
        }

        location /services {
            proxy_pass https://splunkrest;

        }
    }
}

When you start Nginx you will be prompted to enter the PEM passphrase for the SSL certificate. The password for the default Splunk SSL certificate is password.

There are a bunch of settings you may want to tweak including HTTPS Server Optimization, load balancing method, weighted load balancing and health checks.

I’ll leave those settings for you to research and implement as I’m not an expert on them all and everyone’s deployment will differ in complexity and underlying resources.

Hopefully this gives you the foundation for a reliable load balancer to use with Splunk, the REST API and SDK’s.