How many times have you faced a problem where no error information was displayed on screen and related logs provided no data to help trace the failure? Here I'll introduce you to five awesome tips that intermediate problem solvers who are new to debugging can use as a starting point in learning to reverse-engineer this type of problem. You can apply these tips to any program that you're debugging.
Although these tips are generic, I'll demonstrate them by using a real application that I support—Device Manager—which you're probably familiar with. I'll spare you the mind-numbing walk-through of the entire assembly-level debug of this particular problem and instead focus on sharing some basic debugging techniques that you'll need to be aware of as you begin crossing over into the intangible binary world of debugging.
Tip 1: Open a process in the debugger.
When no information is output to the system about a problem, you can use the debugger (windbg.exe) as a powerful tool to identify what's going on in the process. (For more information about downloading and getting started using the debugger, see "Administrators' Intro to Debugging.") Before launching a process in the debugger, you'll need to obtain the command line that you will need to type into windbg to launch that process. You can find the command line by using Process Explorer; to obtain the command line, simply double-click the process, and you'll see the command line as displayed in Figure 1.
Figure 1: Command line to launch a process
After opening the Windows debugger from the Debugging Tools for Windows Start menu group, you can launch Device Manager by selecting Open Executable from the File menu, as Figure 2 shows.
Enter the command line that you'd normally use to start the process.
Tip 2: Find out as much as you can before debugging.
Before jumping into the debugger, get some basic information about the code you want to study. Determining where to start debugging often begins outside the debugger. You need a way to determine the names of functions related to your problem. For example, if your application is reporting an error stating it was unable to open a registry key, your goal is to identify the function that's used to open registry keys. So how do you know what functions are used for different tasks? Although the function names provide some clues, you can use MSDN to research what calls are available. For example, a quick MSDN search on "registry functions" would locate the MSDN documentation listing these functions at http://msdn.microsoft.com/en-us/library/ms724875(VS.85).aspx. You'd see that RegOpenKeyEx is the function used to open registry keys.
You can use the free Dependency Walker tool (depends.exe) to obtain information about relevant functions. Dependency Walker displays what DLLs a binary uses and the function names that the binary uses from the DLL. Obtaining this information is easy; launch depends.exe, then open the binary file that you're troubleshooting by using the open command from the File menu. Dependency Walker will then display the names of the functions that this application calls when it executes. This information is important to your debugging because it lets you identify interesting calls that may be related to the problem. For example, if your application is popping up a message stating that the network connection attempt failed, you'd search Dependency Walker's output for function names that appear related to opening network connections. You can then use the debugger to investigate these calls as they're made.
As an example, let's use Dependency Walker to open devmgr.dll. This is the binary comprising the code that mmc.exe uses to create the Device Manager snap-in. As you can see in Figure 3, Dependency Walker shows that devmgr.dll imports various functions related to device enumeration from setupapi.dll.
In case you're wondering how I determined that devmgr.dll is the DLL used to create Device Manager, devmgmt.msc is actually an XML file that lists devmgr.dll in the text. You can use Notepad to open it, as Figure 4 shows.
Figure 4: Devmgmt.msc contents showing devmgr.dll