Developer’s tool belt: debugging production issues–part 2

So your shiny new application is ready to deploy. You’ve written unit tests and have great code coverage.  What could go wrong?  Then after running for some time in production an issue crops up.  Maybe it freezes and you suspect there might be a deadlock, or it crashes without any error logged, or there’s a memory leak you can’t track down.  If the issue can’t be reproduced and the logs don’t show the issue, then digging into a memory dump might be the last resort.

This post will cover setting up WinDBG and taking memory dumps.  I’ll dive into details about analyzing memory dumps in my next post.  The first step is to install WinDBG.  This used to be a separate install but I guess someone thought that was too easy, so now it’s bundled into the Windows SDK.  Download the web installer and choose Debugging Tools for Windows from the installation options:

WinDBGsetup

Once WinDBG is installed, an environment variable named “_NT_SYMBOL_PATH” needs to be created to specify where WinDBG can find debugging symbols for modules.  Set this variable to “SRV*C:\Symbols*http://referencesource.microsoft.com/symbols;SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols”.  This instructs WinDBG to download symbols from https://referencesource.microsoft.com/symbols and http://msdl.microsoft.com/download/symbols and to save them to C:\Symbols, from which they’ll be loaded by WinDBG when needed.

Creating Dumps

There are a few ways to create memory dumps, which basically boils down to user preference.  The easiest way to create a dump is by using Task Manager, which has had the ability since Windows Vista.  Right click on the process you want to create the dump of and select Create Dump File:

CreateDumpFromTaskManager

This will take a minute or two and display the location of the dump:

CreateDumpFromTaskManagerResult

One thing to keep in mind, which I just learned when creating the dump of Visual Studio for the screenshot above, is that Task Manager can’t create dumps of 32-bit processes by default.  When trying to open the resulting dump file it errors with the message “Debugging a 64-bit dump of a 32-bit process is not supported, please collect a 32-bit dump of a 32-bit process.”  A 32-bit dump can be created by running the 32-bit version of Task Manager from C:\Windows\SysWOW64.  This blog post has more information.  Since I’ve always been creating dumps from 64-bit processes, I haven’t been affected by this before, thanks to .NET targeting Any CPU when compiling by default.

The traditional method for creating dumps is by using ADPlus, which will attach CDB (Console Debugger that’s installed with WinDBG) and create a dump file.  The great thing about ADPlus is that you can use it to create dumps immediately when the process has hung (known as “hang” dumps) or have it wait until the process crashes and create a crash dump.  To use ADPlus, it’s easiest to add “C:\Program Files\Debugging Tools for Windows (x64)” to the Path environment variable and open an administrator command prompt.  Just type “adplus” to view a summary of the options available.  Details on the various options can be found here

It might look complicated but I usually only need to use three options.  First, specify –hang or –crash to indicate whether to create a dump immediately or to just attach CDB now and wait for a crash.  Second, specify where to save the dump with –o.  And third, specify what process to attach to, which is most easily done by using –p and entering the process ID (you can get this from Task Manager using View > Select Columns… > PID) or by using –pn and entering the process file name if there’s only one running.  So, if I wanted to create a hang dump for process 7900, I’d type “adplus –hang –o C:\Dumps –p 7900”.  This opens a separate console window while the dump is being created that displays output from CDB, which will close when it’s finished.

The last method of obtaining a dump is by using the dump created by Windows Error Reporting when the process crashes, if you were lucky enough that one was created.  Look through the Application event log for an Information event from Windows Error Reporting logged just after the Error logged for the crash.  This log entry will have details about the crash, like the following example:

Fault bucket , type 0
Event Name: CLR20r3
Response: Not available
Cab Id: 0

Problem signature:
P1: 3KQHBFJJVVOHCIEKMSLMA1UKFTHQY1VE
P2: 2.1.0.63
P3: 507c3954
P4: mscorlib
P5: 4.0.0.0
P6: 4f1967ce
P7: 1f45
P8: 10
P9: System.OutOfMemoryException
P10: 

Attached files:

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportQueue\AppCrash_3KQHBFJJVVOHCIEK_aabae3283b257b1c1dde673025c79c8073f44c84_23ee8796

Analysis symbol: 
Rechecking for solution: 0
Report Id: e566d53e-24ab-11e2-bc24-005056990056
Report Status: 4

The AppCrash folder may have an .hdmp file in it, which can be copied to another folder and opened with WinDBG.  You’ll want to copy it so you can use it after the report queue is cleared.

In my next post, I’ll detail the techniques I use to triage a memory dump taken from a crash or a hang. Coming up after that, I’ll cover how to search for memory leaks in a memory dump.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s