At this stage, the binaries for SP1 have been applied to Node2 and the AD objects for the EVS have been updated to show the EVS as running SP1. You can verify these changes by running Exchange System Manager (ESM) and viewing the EVS's build version, as Figure 2 shows. Exchange 2003 SP1 has build number 7226.6.
Node1 is still running an earlier build of Exchange, so you can't move the upgraded EVS to that node. If you attempt to do so, you'll get the error message Version of Exchange on this machine does not match the version of Exchange server NodeName. Apply hotfix 831464 to Node1 and reboot. When Node1 comes back online, apply Exchange 2003 SP1.
To confirm that the upgrade was successful, move EVS1 back to Node1 and verify that Exchange starts correctly on that node. As it is in Exchange 2000, this failover operation is optional because all nodes in the cluster are running at the same revision level, but it helps to verify that everything is installed correctly.
Take a backup of the Exchange cluster nodes, the system state on each node, and a full Exchange database backup. You won't be able to restore database backups that you performed under earlier versions of Exchange. When Exchange 2003 mounts the databases for the first time, it stamps them with information to indicate that the database has been accessed by a new version of the Extensible Storage Engine (ESE), which connects the logic in the IS with the underlying Jet database.
The procedure I've just described is the same for clusters with more than two nodes. However, such clusters have a limitation that governs failovers. Clusters that have more than two nodes can host only one EVS per node at a time. Any attempt to move a second EVS to a node currently hosting an EVS will fail with the error message An error occurred attempting to bring the System Attendant resource online: The cluster resource could not be brought online by the resource monitor. Error ID: 5018 (0000139a). The Microsoft article "XADM: Exchange Virtual Server Limitations on Exchange 2000 Clusters and Exchange 2003 Clusters That Have More than Two Nodes" (http://support.microsoft.com/?kbid=329208) describes this failover constraint.
Let's expand our example to add two more nodes (Node3 and Node4) and a second EVS (EVS2) to our sample cluster. Node3, Node4, and EVS2 are running Exchange 2003 without any service packs. Node1, Node2, and EVS1 have been upgraded to Exchange 2003 SP1, and EVS1 is active on Node1.
Take the EVS2 Exchange cluster resources offline. Move EVS2 from Node3 to Node2. (Remember that you can't move EVS2 to Node1 because that node is currently hosting EVS1.) Right-click the Exchange System Attendant resource for EVS2 and select Upgrade Exchange Virtual Server to update the AD metadata for EVS2. When the upgrade has finished, you'll see the message The Exchange Virtual Server has been upgraded successfully.
Bring the Exchange resources back online by right-clicking them and selecting Bring Online. When all the resources are online, check the Application log for errors and take any action necessary.
EVS1 and EVS2 are now upgraded to Exchange 2003 SP1 and are running on Node1 and Node2, respectively. To complete the upgrade, you need to update the Exchange binaries on Node3 and Node4. Apply hotfix 831464 to Node3 and Node4, then reboot each node. Apply Exchange 2003 SP1 to Node3 and Node4, then reboot each node. When Node3 rejoins the cluster, move EVS2 to Node3. Check the Application log to verify that Exchange is starting correctly. Move EVS2 from Node3 to Node4 to verify that failovers are functioning correctly. Check the Application log to verify that Exchange starts correctly. Finally, perform a full database backup of EVS2 and back up Node3 and Node4.
Troubleshooting Tips
As "How to install Exchange Server 2003 Service Pack 1 in a clustered Exchange environment" mentions, you might run into a problem or two during the upgrade. Let's take a quick look at these potential gotchas.
If Cluster Administrator's Upgrade Exchange Virtual Server menu option is unavailable, first determine whether the EVS you're trying to upgrade is hosted on a node that hasn't been upgraded. If so, try moving the EVS to a node that has been upgraded. Another possibility is that Cluster Administrator is running on a node that hasn't been upgraded, so try logging on to and running Cluster Administrator on an upgraded node in the cluster. If neither of these actions does the trick, determine whether Cluster Administrator is running on a server that isn't part of the cluster.
If the Upgrade Exchange Virtual Server procedure fails with error ID c1037b44, check to be sure that the EVS is running on a node that's running the most recent Exchange service pack. Also ensure that the following Exchange cluster resources for the EVS are offline:
- Exchange System Attendant resource
- Exchange HTTP Virtual Server Instance
- Exchange Information Store Instance
- Exchange MS Search Instance
- Exchange POP3 Virtual Server Instance
- Exchange Routing Service Instance
- SMTP Virtual Server Instance
Taking the System Attendant resource offline should take the other resources, which depend on the System Attendant resource being online, offline as well. Be sure that the following cluster resources in the cluster group belonging to the EVS are online:
- IP Address resource for the EVS
- Network Name resource
- Physical disk resources used by the EVS
Finally, take a look at the Exchange setup log file (exchange server setup.log, on the same disk as the \winnt or \windows folder). This log is an invaluable tool for troubleshooting installations. A lot of the information in this file will make sense only to an Exchange engineer or developer, but at least you'll have some data about what's happening during the upgrade. Figure 3 shows a sample of the log.
Be Prepared
When you compare the cluster upgrade models for Exchange 2000 and Exchange 2003, you can reasonably argue that Exchange 2003 doesn't support rolling upgrades for Exchange service packs: Each EVS must be taken offline to update the AD metadata for the EVS, so you have additional downtime to factor into your SLAs. And even straightforward additional tasks increase the risk that something can go wrong during the upgrade. One way to reduce this risk is to create a test EVS for your cluster. Do so a day or two before you plan to upgrade the cluster so that the EVS's AD information has time to replicate across your Exchange and AD infrastructure. After you upgrade one node in the cluster, you can test the EVS upgrade by using your test EVS before you attempt to upgrade your production EVS.