Heap Corruption, Part 2

Last month, I introduced you to heap corruption and the PageHeap functionality of ntdll.dll. This month, I show you how to use Windows Debugger (WinDbg) and PageHeap to troubleshoot heap corruption. You set up PageHeap differently in Windows 2000 and Windows NT, and I address the setup in each OS separately.

Preliminaries
Be sure to set up heap troubleshooting for the first time on a test machine to familiarize yourself with the process. Some changes can adversely affect Windows if you don't make them properly. When you're finished troubleshooting the problem, be sure to reset the system to its previous state. The settings I show you how to enable use a lot of resources and affect computer performance.

Because of the random nature of heap corruption problems, I've written a simple .dll file called debug.dll that doesn't cause a crash by itself in a test environment. (If you use debug.dll in a production environment under load, however, it will probably cause a crash.) Rather, this .dll file shows you how the PageHeap functionality of ntdll.dll lets you track down a problem after you think that a program has heap corruption. (Note that these techniques let you debug any DLL or component on your server.)

Before you can begin test debugging, you need to run debug.dll. Download it from the Code Library on the IIS Administrator Web site (http://www.iisadministrator.com) to your Default Web Site's Scripts directory (or the directory you use to run scripts). Attach WinDbg to IIS. (For information about attaching WinDbg to IIS, see "Using the Windbg Debugging Tool," July 2001.) Now, run the program by opening a browser and typing the URL

http://localhost/scripts/debug.dll?overrun

You probably won't see anything unusual after entering this URL. But after you set up the heap-corruption tools, I'll instruct you to enter the URL again, and you'll see how the results differ. For now, select Break from the Debug menu to break into the program you're debugging. Then, select Debug, Stop Debugging. Click Yes when the Save base workspace information dialog box appears.

Ntdll.dll and Memory Allocation
The heap-corruption tools you're using are WinDbg and the PageHeap functionality in NT's memory manager routine, ntdll.dll. (Note that pageheap.exe is simply a tool that sets global flags to enable the PageHeap functionality in ntdll.dll.) When a request for memory comes in, ntdll.dll grants the requesting program as many bytes of memory as the program requests, usually beginning with the next available space following the previous request's allocation. When you enable PageHeap, ntdll.dll allocates memory on a much grander scale.

Ntdll.dll breaks memory down into pages. (A page of memory is 4KB.) When a program requests 200 bytes of memory, then another 200 bytes, then another 500 bytes, all three of these requests fit into one page of memory with a fair amount of memory still available (7292 bytes, to be exact). However, with PageHeap enabled, each request receives its own page of memory plus an extra page. Ntdll.dll generates the pointer for the memory by starting at the end of the first page and counting backward by the number of bytes requested, as Figure 1 shows. Ntdll.dll marks the second page of memory as no access. Thus, ntdll.dll gives the program a boundary so that if the program tries to write even 1 byte past the amount of memory it requested, the program hits the second page and immediately generates an Access Violation error.

PageHeap for Win2K
Unlike NT, Win2K has the PageHeap functionality built in. To set up PageHeap on a Win2K test system, follow these steps:

1. Increase the size of the memory pagefile to at least 512MB or double its previous size, whichever is larger. To increase the memory size, right-click My Computer, select Properties, then click the Advanced tab. Click Performance, then modify the size of the paging file.



2. Run gflags.exe from the debugging directory you chose when you installed WinDbg.



3. In the Global Flags dialog box, which Figure 2 shows, enter the name of the process that you want to monitor in the Image File Name text box. This process will be either inetinfo.exe (if you want to monitor in-process IIS) or dllhost.exe (if you want to monitor all out-of-process applications).



4. Select the Image File Options option.



5. Select the Enable heap tail checking, Enable heap free checking, Enable heap parameter checking, Enable heap validation on call, Disable heap coalesce on free, Enable page heap, and Enable heap tagging check boxes. Click Apply, then click OK. (Note that you must click Apply before you click OK.)



6. Open a command prompt and type

net stop iisadmin /y

to stop IIS. Then, restart your IIS services. (You can also use the Iisreset command to stop and restart IIS.)