How to debug Django applications with pdb, PyCharm, and Visual Studio

Using a debugger is a common way to find out what is wrong with your application, but debugging a Django application in Splunk might not be so obvious. But it is possible, and I’ll show you how using pdb, PyCharm, and Visual Studio.

Disclaimer: Don’t try this in a production environment.

Python interpreter

Splunk ships with a Python interpreter. To launch it, use the splunk cmd command (see Command line tools for use with Support):

_Windows

%SPLUNK_HOME%\bin > splunk.exe cmd python

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./splunk cmd python

To help run this command, let’s create a couple of small shell scripts under $SPLUNK_HOME/bin:

_{Windows (save it as python_splunk.cmd)}

"%~dp0\splunk.exe" cmd python %*

_{Mac OS / Linux (save it as python_splunk.sh)}

#!/bin/bash
"$(dirname "$0")/splunk" cmd python "$@"

Note: However you name the scripts, just make sure the name starts with python. Otherwise, you’ll run into this PY-11992 issue.

On Mac OS / Linux, be sure to give permissions to execute the script:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ chmod +x python_splunk.sh

Note: We need a shell script for two reasons. First, a script simplifies the way we can execute Python code with the interpreter included with Splunk. Second, we need to have an executable file to set up the Python interpreter in Visual Studio and PyCharm. These IDEs do not allow you to specify a command with an executable and a set of parameters.

Now we can run the Python interpreter from the shell with the script:

_Windows

%SPLUNK_HOME%\bin > python_splunk.cmd

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./python_splunk.sh

Discovering the start point of the SplunkWeb service

Before I show you how to debug SplunkWeb, I want to show you how to find out what you need to launch it manually so that if something changes in the future (how we launch SplunkWeb) these steps will help you to diagnose this and find new a start point.

Currently Splunk 6.0.1 is the latest stable version. If you are using this version, feel free to skip this section. If you skip these steps and if debugging does not work for you, follow the steps in this section to find what has been changed.

Note: All commands in this section are from a Mac OS terminal because Splunk for Windows has an actual SplunkWeb.exe service and it is not so easy to find out how it starts Python code.

I assume that you have Splunk installed on your development box. If Splunk is not running, start it from a terminal:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./splunk start

Now run this command:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ps x -o command= | grep "^python"

This command shows us all processes that have python as an executable. This is the output I see:

_{Mac OS / Linux}

python -O /Users/dgladkikh/dev/splunk/bin/splunk 6.0.1/lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py start

In my case I had only one result. If you have more than one result, it should be easy for you to figure out which one is SplunkWeb. The command in the output is the command that Splunk uses to launch the splunkweb service.

Let’s see if we can use it. Stop the splunkweb service:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./splunk stop splunkweb

Let’s try to launch splunkweb manually by using the script we created earlier:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./python_splunk.sh -O "./../lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py" start

Now you know how to launch SplunkWeb manually. To stop it, press Control + C in the terminal.

In Windows, Python libraries are located in a different folder relative to $SPLUNK_HOME:

_Windows

%SPLUNK_HOME%\bin > python_splunk.cmd -O ".\..\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py" start

Debugging with pdb, the Python debugger

If you like to debug in a terminal, you can debug using the pdb module. Change the command line to:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./python_splunk.sh -m pdb "./../lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py" start

_Windows

%SPLUNK_HOME%\bin > python_splunk.cmd -m pdb ".\..\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py" start

I’ve dropped the -O switch, which is responsible for optimization. There is no point using it for development. See the official Python documentation to learn more about Python command-line arguments.

After this command you should get a breakpoint on the first executable line. See the documentation for the pdb module to learn how to continue (c) execution and how to work with breakpoints.

Debugging with PyCharm

I used the free Community Edition of PyCharm. Download it, install it, and launch it.

On the Welcome screen, choose Open Directory and select your $SPLUNK_HOME directory (where you installed Splunk):

Note for Mac OS/ Linux: You might have issues if you try to open the directory, which does not allow you to write into it. Opening the SPLUNK_HOME folder is not a requirement. You can open the directory with your application. Just make sure that you set up all paths to Splunk in PyCharm as I do below.

Note for Windows: You need to launch PyCharm with elevated permissions (Run As Administrator).

Now set up the Python interpreter. We’ll use the scripts python_splunk.sh / python_splunk.cmd that we created earlier.

Open PyCharm Preferences ⌘, (Settings… in Windows Ctrl+Alt+S) and go to the Project Interpreter / Python Interpreters. Click the Add + button, select Local… and choose $SPLUNK_HOME/bin/python_splunk.sh (%SPLUNK_HOME%\bin\python_splunk.cmd in Windows). PyCharm will ask you to set it up as Project Interpreter, click Yes.

Note for Windows: If you don’t see anything on the Paths tab, make sure to launch PyCharm with elevated permissions.

Next we need to set up how to launch this project:

1. Open Run / Edit Configurations…, click the + button and select Python. Name it as you wish (SplunkWeb for example).
2. Set Working Directory to $SPLUNK_HOME/bin. In my case it is /Users/dgladkikh/dev/splunk/bin/splunk 6.0.1/bin/.
3. Set Script as ./../lib/python2.7/site-packages/splunk/appserver/mrsparkle/root.py (on Windows it is …\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py). This is the script that we found earlier.
4. Set Script parameters to start.
5. Verify that you have the right Python interpreter (the one we set in PyCharm Preferences above).

This is what you should see:

Finally, verify that splunkd is running and splunkweb is not:

_{Mac OS / Linux}

$SPLUNK_HOME/bin $ ./splunk stop splunkweb
$SPLUNK_HOME/bin $ ./splunk start splunkd

_Windows

%SPLUNK_HOME%\bin > splunk.exe stop splunkweb
%SPLUNK_HOME%\bin > splunk.exe start splunkd

In PyCharm press the Debug button to start the splunkweb server. You can place breakpoints in your source code and open Splunk in a browser.

Debugging with Visual Studio

Install Visual Studio and Python Tools for Visual Studio. In my case I installed Visual Studio 2013 Update 1 and Python Tools 2.0 for Visual Studio 2013.

Launch Visual Studio with elevated permissions (Run as Administrator).

First we need to configure the Python interpreter:

1. Go to the Visual Studio Options -> Python Tools -> Environment Options.
2. Click on Add Environment, name it as you wish, for example Python-Splunk.
3. In Environment Settings -> Path specify the path to python_splunk.cmd (see first section of this article).
4. In Environment Settings -> Windows Path specify the path to %SPLUNK_HOME%\bin%.
5. In Environment Settings -> Library Path specify the location of the Python libraries %SPLUNK_HOME%\Python-2.7\Lib.
6. Specify the architecture x64 (x86 if you are on a 32-bit machine).
7. Specify the language version, 2.7.

This is what you should see:

Let’s create a new project. Open File -> New Project… and select From Existing Python Code:

Select the path to %SPLUNK_HOME%:

Select the Python interpreter, which we created above:

Save the project:

Open Project Properties and specify Startup File on the General tab as .\Python-2.7\Lib\site-packages\splunk\appserver\mrsparkle\root.py:

On the Debug tab specify Script Arguments as start:

Save your settings, then verify that splunkd is running and splunkweb is not:

_Windows

%SPLUNK_HOME%\bin > splunk.exe stop splunkweb
%SPLUNK_HOME%\bin > splunk.exe start splunkd

Now start your project in Visual Studio. In my case Visual Studio told me that I have some errors in my project. Just ignore the errors and click Yes to launch anyway.

Place breakpoints where you want, then open Splunk in a browser.

Note: To stop debugging, close the terminal window with Python/SplunkWeb. When I tried to stop debugging from within Visual Studio, it crashed.

Enjoy!