You probably already know that you can't use Secure Sockets Layer (SSL) to protect traffic between front-end and back-end Exchange servers. You can, however, use IPsec to do so in two ways: You can block all other communication to the front-end server so that it communicates only with domain controllers (DCs) and the back-end server, and you can apply encryption, authentication, or both to traffic between the machines. This is an example of server isolation. You can easily apply IPsec protection to just the front-end and back-end servers by adding local IPsec policies with either the Windows GUI or a command-line tool (i.e., netsh ipsec for Windows Server 2003, ipseccmd for Windows 2000).
Domain isolation is a bit trickier. When you deploy domain isolation, the goal is usually to allow trusted machines (i.e., machines that your IT staff manages and maintains) to intercommunicate and to limit which machines your untrusted machines can talk to. Depending on your security needs, you might choose to put your Exchange servers inside the trusted group so that no untrusted machines can talk to them. Alternatively, you can put Exchange servers in a boundary group so that untrusted clients can use some protocols (IMAP and WWW Distributed Authoring and Versioning--WebDAV/HTTP only) for limited mail access without letting them use SMTP or remote procedure call (RPC).
Microsoft goes into great detail about how to structure your isolation environment; the easy route is to base the structure on your underlying Active Directory (AD) domain design, but you can also use organizational units (OUs). The latter strategy, not coincidentally, is a nice fit for Exchange because Microsoft's Exchange security operations guides have long recommended that you put your Exchange servers in their own OUs so that you can easily apply dedicated Group Policy Objects (GPOs) to them. If you've already done so, you should be able to easily apply appropriate IPsec isolation policies to them; if not, the security benefits of being able to place policies only on specific sets of Exchange servers make this something you should do even if you don't plan to use isolation.
Isolation planning and deployment requires a fair amount of up-front planning and testing before you deploy it. Microsoft's IPsec Technology Center contains a wealth of useful information to help you prepare.
http://www.microsoft.com/ipsec