You've probably heard the saying, "The more you learn, the less you know." This adage certainly applies to Exchange Server 2003 and its many complexities, such as migration, storage, administration, and disaster recovery. Combine those complexities with the need to understand related technologies, such as Active Directory (AD), SANs, and third-party tools, and the challenge is obvious. Some Exchange 2003 intricacies, such as migration, the Recipient Update Service (RUS), or securely exposing Exchange access to and from the Internet, are expected. Others are less obvious. Specifically, let's consider two Exchange 2003 components—global size limit settings and public folder replication—and how each affects the other. We'll also look at related oddities that show up in the message transport system.
The Scenario
While recently working on a consulting engagement, I was reminded of how global size limits affect public folder replication. One of my clients built a pristine Exchange 2003 organization with the intent of moving the company's legacy Exchange Server 5.5 environment to the new Exchange 2003 infrastructure. The company has multiple, distributed Exchange 5.5 organizations. As part of its messaging-system consolidation, the company plans to consolidate its two data centers; planning and design for the new environment, including a migration strategy, has been finished. Exchange 2003 is functioning as the routing hub for all environments, essentially handling all inbound and outbound mail.
This company recently set its Sending message size and Receiving message size limits to 12MB. Its strategy was that this setting alone provided the appropriate inbound and outbound message size limits. Therefore, the company set no specific limits for the Exchange connectors and SMTP virtual servers. This practice is fairly typical. Many customers set 10MB to 20MB limits because they want to restrict the size of the messages that flow in and out of their Exchange environments. Microsoft doesn't appear to be concerned about 20MB limits; the Exchange Server Best Practices Analyzer (ExBPA) tool doesn't check inbound and outbound messages limits until messages exceed 30MB.
My client has a fairly large public folder implementation. When the company created its Exchange 2003 public folder structure and content as part of its Exchange 5.5 migration, the organization ended up with nearly 20,000 Exchange 2003 public folders, 17,000 of which are email-enabled. Some individual public folder items are larger than 20MB, yielding about 150GB of public folder content.
The client consolidated five organizations into one and, like most companies, wanted to transition through the painful migration process as quickly as possible. After the migration process is finished, the company will move much of its current public folder structure to other technologies, such as Microsoft Sharepoint Portal Server. The company will also further cleanse its public folder structure and, as a result, realize a much more manageable environment. Many of the mail-enabled public folders actually generate revenue, so this client has no plans to move completely away from public folders anytime soon. Because the new environment's public folder structure represents the legacy Exchange 5.5 organizations, locating information in the new environment is fairly easy, although not ideal.
The Exchange 2003 architecture is split across two data centers that use N+1 clustering and share a high-availability network. The two data centers represent the two routing groups in the organizational structure, which lets the company control how traffic moves between data centers and provides more flexibility and control. Multiple bridgehead servers in each data center not only handle traffic between the data centers but also route and relay inbound and outbound Internet traffic. The Exchange 2003 bridgehead servers aren't directly exposed to the Internet, however. Instead, the client implemented SMTP appliances between Exchange 2003 and the Internet. These appliances, which the two data centers share, provide antispam, antivirus, and routing capabilities.
The Problem
After the migration and public folder synchronization, a major portion of the public folder content ended up on an Exchange 2003 server in the company's East Coast data center. Replication of this content targeted an Exchange 2003 server in the West Coast data center. The replication seemed to be working, and the replica server's database continued to grow. However, after several weeks of replication, the servers' databases differed in size by nearly 10GB—too large a difference to write off as an anomaly.
To determine why the database sizes were so different, we used message tracking to validate message exchanges between the two public folder servers. Because the servers are in different routing groups, replication messages traverse routing group bridgehead servers. When we examined the tracking logs (via the Exchange Message Tracking Center in Exchange System Manager—ESM), a few things were immediately obvious. First, some of the messages were huge—more than 20MB. Second, some of these large messages weren't going anywhere. As Figure 1 shows, some of the messages were larger than 20MB; others, such as the one highlighted in the figure, were never delivered to a bridgehead server and appeared to be stuck in a Started Outbound Transfer of Message state. This turned out to be true for every message larger than about 12MB. But none of the messages showed up in delivery queues. So, although the Message Tracking Center led us to think these messages were stuck somewhere, they actually just seem to disappear after being submitted for delivery.
Troubleshooting the Problem
To troubleshoot the problem, we first looked at the message size limits for public folder replication messages. (You can view these limits in ESM by drilling down to Administrative Groups, AdminGroupName, System Policies, right-clicking the Public Store policy, and selecting Properties.) The limit was set at 500KB, as Figure 2 shows.
The client initially found this 500KB limit confusing because many of the messages were larger than 500KB. Public folder replication lets Exchange bundle multiple small items into one replication message. For example, if a public folder contains ten 50KB items, one replication message can carry all the items. However, public folder replication has no way to break a 10MB Microsoft PowerPoint presentation, for example, into twenty 50KB messages. Instead, Exchange ignores the message size limit and sends one 10MB replication message. That explains why some of the messages were so large but doesn't explain why they weren't delivered.
Next, we looked at the connector between the routing groups to determine whether the connector had a message size limit. (You can check this by selecting the connector's Content Restrictions tab.) The connector didn't have a message size limit.
Then we checked the SMTP virtual servers for a specific size limit because such size limits can affect replication. (You can view the size limit by selecting the Messages tab on the Default SMTP Virtual Server Properties dialog box.) These servers didn't have size limits specified, so we checked the size limit for every SMTP virtual server that transported replication messages between data centers. We looked at the SMTP virtual server on each public folder server as well as on each bridgehead server defined in the connector. None of the SMTP virtual servers had a specific size limit. So far, so good.
Our final check uncovered the problem. The customer had set the global message settings size limit at 12MB instead of the 10MB default. The System Attendant's metabase update service (DS2MB) pushes this value to the Microsoft IIS metabase, and all SMTP virtual servers read and take on that value unless you specifically override it on a virtual server. In our case, because some of the replicated items were larger than the limit, the transfer service submitted the messages for delivery but Exchange rejected them. The Message Tracking Center doesn't produce error messages in such a case, nor does it provide a link to the subsequent nondelivery report (NDR). To see whether any NDRs have been generated, select Properties on the server, locate the Diagnostic Logging tab, drill down to MSExchangeIS, Public, Replication NDRs, and set the logging level to Maximum.