Before you deploy patches, you need to test them to ensure that they don't break existing functionality on your systems. But so many patches exist for so many products and product versions that you need to automate patch testing as much as you can. Here's some advice for setting up a lab and scripting patch testing to quickly shake out the bugs with as little manual labor as possible.
Build a Test Lab
Your test lab should be a microcosm of your Active Directory (AD) forest. To make the lab affordable, you can use VMware Workstation or Microsoft Virtual PC 2004 to simulate your computer hardware, then install fully functional OSs and applications. These virtual systems can communicate over your physical network and make a real AD forest. For a tutorial about using VMware Workstation, see "VMware Workstation 2.0," February 2001, InstantDoc ID 16446.
To get your lab going, purchase at least two Pentium 4 machines with 2GB of RAM and 120GB of disk space each. You'll need ample RAM and CPU horsepower because one physical box will be hosting multiple virtual systems simultaneously. The extra disk space is for the dozens of virtual machine (VM) image files you'll eventually create. To convince management that the lab is worthwhile, prepare a cost/benefit analysis that shows the savings of avoiding even one patch-management catastrophe. And don't forget to point out the other uses for the lab, such as training new systems administrators, testing software updates, modeling Group Policy changes, and performing other software trials.
Your lab should start with at least one instance of each OS in use on your LAN, then add more VMs as necessary for different service pack levels and configurations. Install copies of mission-critical software on the appropriate VM images, and expect to later add special images for your Microsoft Exchange Server, Microsoft IIS, and Microsoft SQL Server systems.
Ideally, the lab's AD forest should have the same domains, trusts, DNS zones, and Group Policy Objects (GPOs) as your real forest, but give the domains slightly different names from your real domains in case your virtual and real forests ever need to interact, and use a small number of accounts to keep the AD database size small. You can use the new Group Policy Management Console (GPMC) to export the GPOs from your production forest and import them into your lab. For more information about GPMC and its import function, see "Windows Server 2003's Group Policy Management Console," July 2003, InstantDoc ID 39190. You can also export your Internet Information Services (IIS) 6.0 metabase and import it into your lab's Web server, as "IIS 6.0 Features," May 2003, InstantDoc ID 38496, explains. You can even restore the system state backups of production servers to the replicas of these servers in the lab.
In addition to implementing your forest in your lab, you should install your patch-deployment method—for example, Microsoft Systems Management Server (SMS), Microsoft Software Update Services (SUS), custom scripts, or third-party products. Then use this patch-deployment method to install updates in your lab.
You'll typically want to test patches in groups and roll them out in groups. You should test and apply crucial patches for your Internet-attached servers as quickly as possible, of course, but applying not-so-crucial patches one at a time is impractical. Microsoft now releases important updates on the second Tuesday of each month, so coordinate your testing and deployment cycle with that schedule.
System and Network Tests
After your lab is up and running, you can begin testing new patches. The strategy is to run a script that will perform a variety of tests, pipe the script's output to a text file, apply the patches, run the tests again, pipe their ouput to a second file, then compare the two files for differences. The two output files should be identical. If they're different, it means something changed after you applied the patches and you need to investigate.
Your scripts should first test for basic network connectivity. Run baseline.bat, which Listing 1 shows, on the test machine to be patched and redirect its output to a text file with a command such as
baseline.bat > before.txt
Then apply the patches, reboot, and run the batch file again, redirecting its output to another file:
baseline.bat > after.txt
By using the file-compare command to compare the two output files
fc.exe /l /n before.txt
after.txt
you can quickly determine whether changes have occurred. Feel free to add more tools to the batch file to expand the scope of its coverage, but make sure you filter the output of these tools with a utility such as findstr.exe so that no irrelevant differences get into the output files to cause false positives. If you'd prefer to use a graphical tool instead of fc.exe to compare the output files, try windiff.exe from the Windows 2000 Server Support Tools.