Microsoft Exchange Server 2010's database availability groups (DAGs) support encryption and compression of the email data they pass around as part of the seeding and replication process. How do these capabilities work?
The short answer is, "It's a Windows thing." Windows has long included a set of base classes for encryption and authentication. (If you're old enough, you might remember that the full-strength version of Windows encryption couldn't be exported to some countries.) Rather than re-invent the wheel, Exchange uses the encryption and signing methods provided by Windows whenever possible. In the case of SSL/TLS, Exchange can do much of its own certificate management—including generating self-signed certificates—but all of the actual crypto is done by Windows.
Exchange also uses Windows' authentication methods. Windows authentication services are provided by modules known as Security Support Providers (SSP). These SSPs provide interfaces that let clients encrypt, decrypt, sign, and verify data by using a particular set of security protocols. There are SSPs for Kerberos and NTLM. For DAG encryption, Exchange uses the Kerberos SSP. After two applications or endpoints have completed a handshake process to let them come up with a shared encryption key, they can both make calls to the SSPs on their local machines to process the data blocks.
Exchange always tries to pick the strongest possible encryption algorithm from the list of those supported by the host OS. For Windows Server 2008 and Server 2008 R2, that means the 256-bit version of the US Federal Advanced Encryption Standard (AES), ideally with Secure Hash Algorithm-1 (SHA-1) used as a Hashed Message Authentication Code (HMAC). On the compression front, DAGs use the familiar LZ77 algorithm, which is used to compress GIF files and in tons of other places. Microsoft calls its implementation "XPRESS," for some reason that I haven't been able to figure out; Microsoft uses this implementation in a few other products besides Exchange.
Outlook uses many of these same mechanisms; for example, MAPI RPC compression between Exchange and Outlook uses XPRESS, and MAPI RPC encryption can use the Kerberos SSP (or the NTLM SSP, which isn't available for DAGs.) That's no big surprise, given that these capabilities are built into Windows. Why reinvent the wheel?
To control these options in a live DAG, Set-DatabaseAvailabilityGroup is your friend. By default, newly created DAGs use encryption only when replicating across different subnets, with no encryption used during seeding operations. You can change this setting with the -NetworkEncryption switch; just set it to Enabled instead of InterSubnetOnly. In like manner, compression is normally enabled only for cross-subnet replication, but the -NetworkCompression switch lets you fix that right up.
Note that these properties are associated with the DAG itself, not the underlying DAG network. For that reason, you can set these properties only when the DAG (and its underlying physical nodes) are up—the settings are node-level properties.
Alert readers will note that I haven't made any mention of using IPsec, one of my favorite technologies, to encrypt communications between DAG members. What about using IPsec with DAGs? That's a topic for another UPDATE!