A common problem we see customers struggle with is how to diagnose connectivity issues with any of our SDKs. In this post, I’ll show you a few tried and true practices that can help you figure out what might be going wrong.

There are two main families of errors folks see. One has to do with general connectivity / connection info, and the other has to do with security config on the client.

General connectivity issues 

This means that you are unable to succesfully connect to the API. The best way I find to diagnose is to drop to a terminal and use curl to login to the Splunk API and see the results. The command to use is:

curl –k [server:port]/services/auth/login -d username=xxx -d password=xxx    

If you get a response like: 

<response>
   <sessionKey>G^b8zfS3YGWFSJTxY7c5fpqso4DZr6_O_z9lcXk^v...</sessionKey>
</response>

This means the API is definitely accessible. If not, then it means a number of possibilities.

• If you get this: <msg type=”WARN”>Remote login disabled by ‘allowRemoteLogin’ in server.conf</msg> then you need to tweak the server.conf setting
• If you get this: <msg type=”WARN”>Remote login has been disabled for ‘admin’ with the default password. Either set the password, or override by changing the ‘allowRemoteLogin’ setting in your server.conf file.</msg> then you are trying to connect with admin and you have not yet changed the admin password from the default of “changeme”. 
• If you get this: curl: (7) Failed connect to 10.80.9.131:8088; No error then either the URI is wrong, the port is not correct, or the port is not opened on the firewall.
• If an empty reply, then you are using the wrong scheme i.e. http when it should be https

If you get a valid response using curl yet the SDK is still failing, then the credentials / URI passed in the code that uses the SDK could be wrong. Check your app’s configuration.

Security configuration issues

The second family of issues relates to either certificate validation failing, or the security protocol configuration in Splunk.

Certificate validation

Depending on the SDK you are using, there is another kind of error you might see, which relates to certificate validation. By default most platforms will automatically throw an exception when the HTTPS cert is not valid. Splunk by default does not return a valid cert, which causes this failure.

This can be disabled within the application code. Depending on the language/runtime stack this differs, as some require it to be done in the app setup, and others (like Node.js) allow you to do it when you make the call. For example if you are using our C# SDK, then you can turn off cert validation using code like this: 

ServicePointManager.ServerCertificateValidationCallback += (sender, certificate, chain, sslPolicyErrors) =>
{
     return true;
};

Security protocol

You might also see an error that indicates the client is unable to negotiate a connection. Generally this means that the security protocol the client is using to connect, is not within the set that Splunk is configured for in server.conf. You might for example be using SSL2 while Splunk is configured for SSL3 or TLS only.

Again, each platform generally has a way to configure this protocol. Using C# again, here is how to configure the security protocol to use TLS:

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls;

Once the protocols on client and server match, you should be able to make the connection.

Does that cover everything?

I’d like to think it does, but I am almost positive it doesn’t ;-). However, it covers the most common problems I have seen over the past few years, and it may cover yours!

If you have any other tips to share, put them in the comments, and I’ll add them to the post!