The front-end and back-end topology that Exchange Server 2003 and Exchange 2000 Server offer lets you scale your Exchange infrastructure by separating the machines that clients communicate with from the (usually larger) machines that store the mail data. However, in a typical front-end/back-end setup, the front-end server is outside the corporate network boundary, often in a demilitarized zone (DMZ), which means that the front-end server and back-end server will likely be communicating across a trusted network boundary. Unfortunately, Exchange doesn't support the use of the Secure Sockets Layer (SSL) protocol to secure this traffic. You can work around this limitation by using a firewall that can do SSL bridging (such as Microsoft Internet Security and Acceleration—ISA—Server), but that solution isn't always practical. As an alternative to SSL bridging, you can use IP Security (IPSec) to secure your Exchange network for free.
Understanding IPSec
IPSec is a set of extensions to the basic IP technology that we use for Internet communications. IPSec operates at the transport layer, so applications don't need to be aware of whether IPSec security is in effect. That approach is a major advantage over SSL, which—as an application-level protocol—requires that the application on each end know about the protocol. Another IPSec advantage is that it's made up of two separate but complementary protocols:
- The Authentication Header (AH) protocol adds a cryptographic authentication header to each IP datagram on a secured connection. The AH protocol calculates and inserts a digital signature into the packet between the original IP datagram header and the packet's payload. This approach lets the packet be routed without losing the AH data; nonIPSec-capable devices think that the AH data is part of the payload. AH provides tamper-proofing, but no confidentiality—an attacker can still read AH-protected traffic in transit.
- The Encapsulating Security Payload (ESP) protocol provides confidentiality and integrity checking. ESP uses one of two modes to encrypt the datagram's contents: In tunnel mode, packets are protected to enable connections to two separate networks; transport mode provides end-to-end security between a client and a remote network. We'll use the transport mode with Exchange; you can use tunnel mode to establish IPSec-protected VPNs.
You can use the AH and ESP protocols in conjunction with each other or independently; each protocol also supports several cryptographic algorithms. Two IPSec-capable computers begin communication by using the Internet Key Exchange (IKE) protocol to exchange cryptographic keys. The computers then negotiate to find an algorithm and key length that they both support. This process establishes a secure channel—called a security association (SA)—which protects traffic between the two machines.
Using IPSec Between Front-End and Back-End Servers
Before you begin using IPSec, you need to open several ports to provide end-to-end connectivity between your front-end and back-end servers. You must open UDP port 500, which IPSec key exchange uses for the IKE protocol. If you want to use AH for front-end and back-end communication, open IP protocol 51; for ESP, open IP protocol 50. If you're using multiple firewalls, be sure to open the ports on both sides and in both directions. Note that when you open the ports on the firewall, you need to specify the source and destination addresses of your front-end and back-end servers.
When you use IPSec for front-end and back-end communications, you use Group Policy to control which types of communications a server attempts to encrypt or authenticate. You typically specify filters based on the IP address and port that traffic is going to or coming from; the Microsoft Management Console (MMC) IP Security Policy Management snap-in lets you create flexible policy rules, if you need them. Fortunately, each Windows 2000 machine has a local IPSec policy engine (simply a small version of the IPSec Group Policy mechanism) that lets you apply IPSec policies to individual machines without requiring a full-blown Group Policy deployment. It's important to keep your policies simple so that you don't make mistakes that block network traffic. If you're not already familiar with IPSec, you might want to hire an expert to help you design an appropriate policy set.
When you deploy IPSec, you can create rules that specify which protocols and ports to use for communications (e.g., protocol 50 with port 80) and whether they require or merely permit IPSec. This process can be daunting, but you can simplify it by keeping in mind the following two caveats:
- You can tell the front-end servers to initiate IPSec connections only to the back-end servers, not to any other servers. Kerberos and IPSec don't mix particularly well in Win2K, and protecting communications between the front-end servers and the Global Catalog (GC) isn't strictly necessary, although it's possible.
- The back-end servers can accept IPSec requests they receive from the front-end servers, but the back-end servers don't need to initiate outbound IPSec traffic because the back-end server will never initiate communications to a front-end server on its own.
Given this advice, you can easily build an IPSec policy that does what you need. How you target these policies will vary according to how your front-end servers and back-end servers are separated. If they're in separate organizational units (OUs), bear in mind that you can't apply IP security policies to an OU, only to a domain. For most applications, applying IPSec policies to the individual front-end servers and back-end servers makes the most sense because you'll have relatively few servers.