Got High-CPU Usage Problems? ProcDump 'Em!

On the Microsoft support team, one of the most common customer problems we encounter is systems experiencing high CPU usage. Solving this type of problem is often challenging because you must first determine which process or activity is responsible for consuming so much CPU time, then determine the best approach for capturing the process's activity during the problem period so that it can be analyzed for root cause. Fortunately, Microsoft provides tools available to assist with high-CPU issues. I'll give a brief rundown of these tools, then introduce you to a brand-new free tool called ProcDump that will save you much time and hassle the next time you run into a high-CPU problem.

High-CPU Usage Troubleshooting Tools
Until now, we've relied mainly upon these tools to help troubleshoot high-CPU problems on Windows systems:

Adplus.vbs. This VBscript tool comes with the Debugging Tools for Windows (www.microsoft.com/whdc/devtools/debugging/default.mspx) and is a great resource for administrators to use for dumping out a process during a high CPU occurrence. However, one of the drawbacks of Adplus is that a person usually has to be at the console to physically issue the Adplus command to dump out the process when the CPU spike occurs.

Xperf. This is a super tool for collecting process activity during a high CPU spike, and it doesn't require anyone to be physically at the console to monitor for high -CPU occurrences. (You can download Xperf at msdn.microsoft.com/en-us/performance/default.aspx.) Although Xperf isn't fully supported on Windows Server 2003, our experience with collecting stackwalk data on Windows 2003 (which is the critical piece of data for analyzing high-CPU problems) has been very positive, as long as you have the hotfix download available at support.microsoft.com/kb/938486 or a later-dated kernel installed.

One of the things to consider with XPERF is that the tool collects data about all processes and activity on the system, then lets you narrow your focus post-mortem, which means there's no way to specify, say, "I just want stackwalking for XYZ.EXE"; instead you have to turn it on for the entire system. So collecting and logging all of a system's activity for a problem that may occur once in 24 hours could be too much overhead depending on the typical workload of the systems you're monitoring. (For more information about Xperf, see "Examining Xperf" and "Under the Covers with Xperf.")

Process Explorer (procexp.exe). I highly recommend that you use Process Explorer, which you can download at technet.microsoft.com/en-us/sysinternals/bb896653.aspx, to at least look at the thread that's spiking the CPU to determine what components are involved, so that you can update them before calling tech support. If you need to investigate the problem further, though, you'll need a tool that actually dumps out the process during the high-CPU spike; Process Explorer can't do this. (For more information about Adplus and Process Explorer, see "Say 'Whoa!' to Runaway Processes.") But ProcDump can.

Introducing ProcDump
ProcDump (procdump.exe) is a new Windows Sysinternals tool from Mark Russinovich, which you can download at technet.microsoft.com/en-us/sysinternals/dd996900.aspx. Procdump.exe was created after one of the escalation engineers in my group asked Mark if he would consider adding functionality to Process Explorer to allow for capturing a dump file of a process to help troubleshoot those pesky high-CPU problems. After some thought, it was determined that the best approach was to write a new tool, and ProcDump was born.

ProcDump lets you configure how much CPU a process should consume and for how long a time period before ProcDump creates a dump of the process. What this means is that you don't have to be at the console ready to issue commands the next time the process spikes the CPU. And you get to determine at what threshold the process can consume the CPU before ProcDump captures a dump of the spiking process.

So, for example, you notice the wmiprvse.exe (the WMI Provider Host process) spikes the CPU to 90 percent at random times throughout the day, and you'd like to capture a few dumps for analysis. The following command will dump out the spooler process three times when the CPU for wmiprvse.exe is at or exceeds 90 percent for 3 seconds and store the dumps in the c:\procdumps directory that you've already created:

c:\procdump.exe -c 80 -s 3 -n 3 wmiprvse.exe c:\procdumps

The -c option is the CPU threshold parameter that you can configure. The -s option tells ProcDump how long the service needs to consume the CPU at the threshold you configured before a dump is generated. The -n option tells ProcDump how many dumps to create, and wmiprvse.exe is the process name you're asking ProcDump to monitor.

So, for the previous command line, the WMI Provider Host service will be dumped out each time the process exceeds 80 percent CPU for three seconds or more and store the dump files in the c:\procdumps directory. The name of the dump file will be in the format PROCESSNAME_DATE_TIME.dmp; the included timestamp makes it easy to identify files captured over a period of several days. The other great feature of ProcDump is that the thread that consumed the highest amount of CPU is baked into the dump file, so that when the dump file is opened in the debugger, you get a message indicating which thread consumed the CPU, as Figure 1 shows.