The Sorcerer's Apprentices

I often hear questions about the two diagnostic and repair tools included with Exchange Server 5.5: Isinteg and Eseutil. Misunderstandings are common about what these tools do, how they work, and when—or if—you should run them on your servers. Because these tools are key parts of Exchange Server disaster recovery, the best time to learn how to use them is before you need them (as a friend of mine says, "The time to learn how to be a firefighter is not when your house is on fire"). And like firearms or prescription medications, disaster-recovery tools can be dangerous if you don't understand how they work and when to use them. (Imagine shooting a shotgun at a gallon milk jug full of water—a graphic demonstration of what can happen when you don't properly handle a powerful tool.) But before you can dig into what these tools do, you need a good understanding of the Information Store's (IS's) databases and how they work.

The Logical Structures
Let's examine the IS's private and public databases—priv.edb and pub.edb. As far as the command shell or a tool such as Windows Explorer is concerned, priv.edb and pub.edb look like any other file type, only larger. But in reality, the IS database structures are quite complex. Inside each .edb file is a complicated set of interrelated data that the IS and the client turn into what you see as a mailbox or a message.

The Exchange Server Extensible Storage Engine (ESE) stores everything in the IS databases in special structures called B-trees. These structures allow for efficient and fast indexing and retrieval of the databases' contents, which is exactly what you need for an application such as Exchange Server 5.5. An ESE table is a collection of B-trees; each ESE table is made up of rows and columns. When a column contains data that takes up less than 1KB, ESE stores the data directly in the table. When the data is longer than 1KB, ESE stores the data in separate long-value B-trees and stores a pointer to that data in the original table. This procedure makes the non-long-value tables' size predictable and provides a performance advantage because ESE always knows how much data it must read or write.

Single-instance storage is an important Exchange Server feature that depends on this process. With single-instance storage, if I send the same message to 25 recipients on one server, the IS stores only one copy of the message in the database—the other 24 recipients' mailboxes contain pointers to the appropriate B-tree. A logical error in a B-tree can affect multiple mailboxes or public folders.

The Physical Structures
An ESE database is a collection of 4KB pages. Each page can contain the same maximum amount of data, and the page's owner decides how much of that capacity to use. Depending on its size, a chunk of information (e.g., the ESE table that tracks message attachments) can span any number of pages. A 1MB Microsoft PowerPoint attachment will take up a little more than 250 pages (allowing for some overhead). ESE uses pages to efficiently read and write relatively small chunks of data on demand. Pages and the efficient indexing that the B-tree system offers provide good performance, even under heavy loads.

One page can contain data and pointers to other pages. An 8KB message will span at least two pages: The first page will contain almost 4KB of data and a pointer to the second page. Pages also have headers that contain several items, including an error-checking code (called a checksum) that you can use to verify pages during backup and restore operations. If the header's checksum doesn't match the checksum that ESE calculates when it reads or writes the page, ESE logs an error (event 1018) in the event log to warn you that the physical database structure might be corrupt.

Isinteg: The High-Level Verifier
Isinteg looks at the mailboxes, public folders, and other parts of the IS, checking for anything amiss. Think of a proofreader: Isinteg scans the tables and B-trees that organize the ESE pages into their logical structures. In addition, the tool looks for orphaned objects, objects that have incorrect values, or impossible references. Isinteg can perform 33 tests, some of which work on only a particular database.

You can run Isinteg in two modes. In default mode, the tool runs the tests you specify and reports its findings. In fix mode (which you control using the optional —fix switch), Isinteg runs the specified tests and attempts to fix whatever it can. The repair process is akin to putting together a jigsaw puzzle—Isinteg moves information around but never throws anything away. Isinteg isn't overtly destructive the way Eseutil can be. However, whenever you run isinteg —fix, you run the risk (however small) that you'll lose some data.

Isinteg has another, probably better-known function: After restoring an offline backup, you can run the tool to fix the globally unique IDs (GUIDs) for items in the IS. If you don't fix the GUIDs, your IS won't start, and you'll see an event 1011 error in the event log. (You don't need to use Isinteg when you restore an online backup because Exchange Server-aware backup software fixes the GUIDs on the fly.)

Eseutil: Down and Dirty
If Isinteg is a proofreader, then Eseutil is a plumber. Eseutil scans the physical structures in the databases, looking for and fixing physical errors such as incorrect page checksums and bogus page-to-page pointers. The tool has six modes. Eseutil can check each page's header to ensure that it matches the page's data (integrity check mode). This mode duplicates the process that takes place when you run an online backup. Eseutil can dump the databases' physical structures in a form that humans can read (dump mode). The tool can condense the databases by combining partially full pages into a reduced number of completely full pages (defragmentation mode). And Eseutil can take a database that an earlier version of ESE has generated and update that database for ESE97, which is the version that ships with Exchange Server 5.5 (update mode).

Eseutil has two distinct repair modes. Eseutil can try to fix whatever it can without touching the pages' data (recovery mode). In this mode, the tool might modify table links or B-tree entries, but it won't truncate or throw away bad pages. Or Eseutil can modify or throw out whatever necessary in a last-ditch effort to salvage your data (repair mode).

How to Use Isinteg
To run a complete suite of Isinteg tests in the optimal order for catching and fixing the maximum number of problems, use the following command:

isinteg —pri —test alltests