The primary purpose of IPsec is to protect the content and integrity of network traffic by implementing digital signing and encryption. But when you need to restrict access to certain resources, you probably turn to Access Control Lists (ACLs) or VLANs as the most likely candidates for this purpose. But ACLs—specified in the application layer—can end up posing a great security risk. In reality, you can use IPsec for purpose of isolating specific hosts or domains from the threat of unauthorized (or unmanaged) computers.
In particular, the new IPsec-based Connection Security Rules in Windows Server 2008, Windows Vista and Windows 7—configurable through the Windows Firewall with Advanced Security Console and Group Policy—provides an excellent tool for implementing server isolation. Let's start with a little background about server isolation in general, then dive into the process of configuring it in your environment.
What Is Server Isolation?
By implementing server and domain isolation, you propagate a network policy that requires that specific servers—members of domain—accept authenticated and secured communications only from other domain-member computers. This network policy isolates specific servers from computers that aren't domain members, or computers that are domain members but don't satisfy certain criteria. For example, you can configure a policy that forces a database server to accept connections only from the servers that are members of a specific security group or that have a specific computer certificate installed.
When you implement isolation this way, there's no need to reconfigure the network or implement any third-party software. Everything you need is already present in the OS. Hosts or domains isolated in this way will require no maintenance in case of changes in network design, or if they're moved to another location or another network device. Because isolation is implemented at the OS level, it won't interfere with other levels of protection.
In Windows Server 2003, server isolation was possible by configuring the Access this computer from network Group Policy setting, but this feature's functionality was limited. It was possible only to grant users or computers the right to access a specific host; you couldn't assign additional options such as the authentication method .Also, it was possible to force an authentication protocol through Group Policy (e.g., to accept only NTLMv2), but it wasn't possible to force Kerberos or to request certificates as a means for authentication.
Server 2008, Vista, and Windows 7 provide new functionalities for server isolation through the Windows Firewall with Advanced Security Console. Aside from providing advanced firewall configuration possibilities, this console lets you implement Connection Security Rules. These rules are crucial for the implementation of server isolation. Although IPsec-based isolation was possible to achieve in earlier OSs such as Windows XP and Windows 2003, Server 2008 and Vista integrate IPsec and firewall functionality for the first time.
Requesting or Requiring?
Your first task is to identify the host you want to isolate, then determine the level of isolation to implement. In some cases, server isolation will occur on all hosts in domain, which essentially equates to domain isolation. However, more often, you'll want to isolate only specific (client or server) machines that require an additional layer of security. So, let's focus on implementing isolation on a single host. (Because Connection Security Rules exist on both Vista and Server 2008, and are configured the same way, I won't focus on a specific OS.)
You'll find the Windows Firewall with Advanced Security Console in the Control Panel Administrative Tools applet. After you open the console, right-click the Connection Security Rules node and select New Rule. Doing so starts the New Connection Security Rule Wizard, which offers several choices. Choose the first option, Isolation. The other available options let you make an exemption rule for specific hosts, implement authentication between two specific computers (the Server-to-Server option), force authentication in tunneling mode (useful for site-to-site links), or make a custom rule.
After you select Isolation and click Next, you must choose between several authentication requirements. Essentially, your choice is between requesting and requiring. If you choose a Request option, authentication will be requested (i.e., offered) for inbound or outbound traffic (or both), but it won't be forced. If the other party can't properly authenticate, traffic will still be allowed. If you choose a Require option, the OS will force authentication and will drop the connection if authentication is unsuccessful. Depending on your required level of security, you can choose Require authentication for inbound connections and request authentication for outbound connections, which is acceptable if you want only to force inbound authentication (when other hosts trying to access this isolated host), or you can choose Require authentication for inbound and outbound connections, which maximizes security by forcing authentication on both inbound and outbound traffic.
The first option, Request authentication for inbound and outbound connections, won't force authentication in any way, so it's not true isolation. The second option, Require authentication for inbound connections and request authentication for outbound connections, will keep an acceptable level of security for an isolated host while still allowing the host to communicate with all other hosts (domain and non-domain). For that reason, the second option is a good solution, so select that option and click Next.