Developer’s tool belt: debugging production issues–part 3

Before I dive into the next post in this series, I want to recommend reading Debugging Microsoft .NET 2.0 Applications.  I rely on this book whenever I can’t remember how to do something in WinDBG, and what I’ll be blogging about investigating memory issues or deadlock I learned from this book.  In addition to its great detail about using WinDBG, it has a lot of other great tips for debugging with Visual Studio.

There are a few commands I run on a dump when I first open it in WinDBG to get a quick look at the state of the process.  The first command must always run in order to analyze .NET dumps:

.loadby sos clr

This loads SOS.dll into WinDBG and enables all of the .NET related commands you’ll need to use.  If the dump was created due to a crash, something like the following might be displayed when it’s first loaded:

This dump file has an exception of interest stored in it.
The stored exception information can be accessed via .ecxr.
(80c.3fc): CLR exception - code e0434352 (first/second chance not available)
00000000`771718ca c3              ret

You can execute .excr but it isn’t aware of .NET, so it will just display the registers and unmanaged stack trace at the time of the exception.  Instead, !analyze –v will display the Windows Error Reporting information related to the exception, including the stack trace of the thread the exception occurred on.  Typically, I want to see what all of the threads looked like when the dump was made.  First, I use !threads:


Oh look, there’s the OutOfMemoryException that crashed the process.  Threads with XXXX as the thread number in the leftmost column are dead threads that haven’t been cleaned up yet.  You won’t be able to switch to those threads or view their stack, but if a lot them are listed that could be an indication of the issue.  If a particular thread doesn’t catch my eye, I’ll usually use !EEStack next.  It works the same as !DumpStack, but gives the full stack trace (managed and unmanaged) for all threads, which helps you get a good idea of what was going on at the time of the dump.  !ClrStack is similar but only displays the managed portion of the stack, but it’s useful when you want to see the parameters passed to methods and the local variable when you get deeper into the dump.  In the case of !EEStack, seeing the unmanaged stack is useful because it makes it easy to see what threads are currently sleeping, whether because they have nothing to do or they’re waiting for a lock or some other action is causing them to block.  This is usually indicated by a WaitFor… at the top of the stack trace:


For example, here’s the top of a stack trace of a thread waiting for a SQL query to complete:


You can see the difference between how managed and native methods are displayed, and at the top there’s a WaitForSingleObject in this case.

If my intent was to try to track down a memory issue, my next step would be to look at the heap with !DumpHeap –stat.  I’ll go into the details of debugging memory issues in my next post.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s