Executive Summary:
Xperf, a versatile Microsoft Windows event-tracing tool, provides a stackwalking feature that you can use to determine which system functions are consuming the most processor time. You can use this information to uncover the source of Windows system performance problems, such as might occur after you update an application or make a configuration change. Michael Morales shows you how to run an Xperf stackwalking trace for a specific time period, then view and interpret the trace data to pinpoint resource-consumption issues that may be impeding your system's performance. |
One of the more powerful features of Xperf is the stackwalking feature, which lets you capture and view the functions executed during a specific time period and determine the most processor-intensive operations. The stackwalking feature lets you see not only the where all the processor time is being spent but also the path of execution for a process that led up to the expensive operation. Xperf can also help uncover the impact of configuration changes to a system by collecting aggregate data and parsing it for information about performance or patterns of resource consumption. In this article, I'll walk you through step-by-step how to properly capture a stackwalking trace and how to interpret the data.
When Stackwalking Can Be Helpful
Here's an example of a scenario where Xperf stackwalking can be useful. Following a recent update to a web application, you notice that system performance deteriorates. Using Xperf and the stackwalking feature, you find out that processor is spending time trying to draw a border around a JPEG file on a website. As you examine the function calls within Xperf, you discover that each JPEG file lookup results in hundreds of registry lookups, which is where all the processor time is being spent. When you give this information to the application vendor or in-house developers, they quickly realize that all the registry lookups are unnecessary and that the information could be held in memory where access times are much faster and less processor-intensive.
Capturing a Stackwalking Trace
Here are the tasks you'll need to perform to use Xperf to capture a stackwalking trace.
Set system variables for symbol loading. On the target system, the best way to ensure that your symbols are properly configured is by setting the following two system variables. You can set these two variables via the command line; however, my recommendation is to set these variables using the Control Panel's System applet or by right-clicking My Computer and selecting Properties, clicking the Advanced tab, and clicking Environment Variables.
The first system variable you need to set is this one:
_NT_SYMBOL_PATH = srv*c:\symbols*http://msdl.microsoft.com/download/symbols*c:\APP1
If you're a developer, then you may want to add paths to your own application's symbols or to another vendor's symbols. Simply add an additional asterisk (*) to separate each path.
The second system variable to set is this:
_NT_SYMCACHE_PATH = c:\symbols
Setting this variable tells Xperf to strip each symbol of the unnecessary information (i.e., structure definitions) and create a smaller-sized symbol file. The smaller file increases the speed of symbol loading and improves the overall Xperf experience when you're viewing stacks.
Specify the stackwalk flag. There are several ways you can use the stackwalking flag in Xperf. Following is a typical command line that turns on stackwalking for kernel events and can be used to help diagnose high–CPU-usage issues:
xperf -on latency -stackwalk profile
To find a more comprehensive list of stackwalking events, use this command:
xperf -help stackwalk
Turn on symbol loading. After the trace has been collected and you're viewing the trace file, you'll need to turn on symbol loading from within the Xperf viewer, as Figure 1 shows.
Figure 1: Loading symbols from within Xperf
Those are the three main action items required to utilize the stackwalking functionality within Xperf, which leads us to a question: Is there any value in enabling the stackwalking feature if you don't have access to symbols? Answer: Yes, but it depends. Without symbols, you'll be able to see only the module within the process responsible for consuming the CPU. However, in some cases that could be enough information for you to make progress in resolving an issue, such as high CPU usage.
Capturing and Interpreting a Trace
Now let's walk through the complete set of steps you'll need to perform to first capture a stackwalking trace, then view and interpret the trace data.
Step 1: Issue the Xperf command. Following is a typical command line you can use to capture the information you'll need to diagnose a high–CPU-usage issue by using the stackwalking functionality. Once you've set up symbols (as I just described) and installed Xperf, you don't have to reboot or restart any other service. You're ready to run the following command:
xperf -on latency -stackwalk profile
This command says, "turn on tracing for the kernel events that the latency group represents, and turn on the stackwalking profiler for this session." You won't be able to view any stack traces without issuing the -stackwalk profile flag.
Step 2: View and then stop the trace. The CPU spike needs to occur sometime while tracing has been enabled. To view and stop the trace, enter the following command:
xperf -d perf.etl
By default ETW places the information into the kernel.etl. By issuing the -d command you are saying, "merge the kernel.etl with the name of the etl file you specified (e.g., perf.etl).
Step 3: Open the perf.etl file. Now all that's left to do is open the perf.etl file inside the Performance Analyzer Viewer (i.e., Xperf), by issuing the following command:
xperf perf.etl
Figure 2 shows a complete end-to-end string of command-line entries.
Figure 2: Xperf trace commands