How to Implement SDI Using IPSec in a Mixed Environment

Many enterprises proactively defend against malware outbreaks by limiting access to their key Windows servers and networks. One popular defense is Server and Domain Isolation (SDI) using IPsec. SDI is easy to implement in Windows environments. However, some enterprises maintain a heterogeneous environment of Windows, UNIX, and Linux systems. Fortunately, you can extend SDI to UNIX and Linux systems.

One reason why SDI is easy to implement in Windows OSs is that you can use Kerberos as an authentication mechanism. However, not all UNIX and Linux clients support Kerberos. Most UNIX and Linux clients do support preshared keys (e.g., passphrases, sequences of random characters) and X.509v3 certificates as authentication mechanisms. Using preshared keys is the simplest authentication method to implement, but it's generally considered the least secure. The use of certificates is more secure but requires a public key infrastructure (PKI).

To keep things simple, I'll show you how to use preshared keys to configure three popular UNIX and Linux OSs to use IPsec so that SDI is supported for both outgoing and incoming connections. I'll only cover how to configure your Windows systems to the extent necessary for interoperability with the UNIX and Linux systems. For more information about configuring IPsec to support SDI on Windows systems, see the references in the Learning Path at the top of the page.

Before I proceed, I must caution you that incorrectly configuring and deploying IPsec in your Windows environment, especially when using Group Policy, can break all communications between systems and between systems and domain controllers (DCs). I recommend you test everything in a lab environment before deploying IPsec to production systems and networks.

Preparing Your Windows Environment

Before you start to configure your UNIX and Linux systems, I recommend that you configure SDI in your Windows environment first. By default, Windows uses Kerberos as the authentication method for IPsec. If your UNIX and Linux systems don't support Kerberos as an authentication method, you don't need to stop using Kerberos with IPsec in your Windows environment. IPsec will negotiate a mutually agreeable authentication method. All that's required is that the Windows, UNIX, and Linux systems support the same authentication method.

Most SDI implementations in Windows environments divide systems into secure servers and clients:

Secure servers. Secure servers are configured to always use IPsec in system-to-system communications, typically with exceptions for communications with DCs and default gateways. If a secure server receives an inbound connection that isn't secured using IPsec and the connection isn't from a DC, a default gateway, or another exempted system, the server will reply that IPsec is required. You can view these replies using the Microsoft Management Console (MMC) IPsec Security Monitor snap-in or Network Monitor.

Note that DCs are exceptions because Windows uses Kerberos by default, and DCs authenticate users and computers. If you can't communicate to a DC without IPsec, which requires authentication, you can't be authenticated. In addition, DCs typically offer services such as DHCP and DNS. DHCP is used to obtain an IP address, so IPsec can't be used before the machine has fully started up and authenticated itself against a DC. DNS is used for DC discovery for authentication. IPsec can't be established before authentication.

Clients.Clients are typically configured to attempt all communications without IPsec. When they attempt to communicate with a secure server, the server will reply that IPsec is required. In response, the client will begin to authenticate using IPsec. Once authenticated, communications will continue.

The simplest means to configure and manage IPsec policies in your clients and secure servers is to use Group Policy Objects (GPOs). From this point on, I assume that you already have GPOs with IPsec policies for your clients and secure servers.

In the Group Policy Object Editor, you need to perform the following steps for each GPO:

1. Open the GPO, navigate to Computer Configuration\Windows Settings\Security Setting, and click IP Security Policies. In the right pane, double-click your clients' IPsec policy to open the IP Security Policy Properties dialog box.
2. Assuming you have a single default IPsec rule for your clients, select that rule and click Edit.
3. Select the Authentication Methods tab and click Add.
4. In the New Authentication Method properties dialog box, select Use this string (preshared key) and enter a key. Figure 1 shows the addition of an authentication method that uses a preshared key.
5. Click OK, then close the IP Security Policy Properties dialog box.

Changes to policies will be distributed to your Windows systems at the next policy refresh interval or when the Windows systems start up, whichever comes first. For clients with more than one IPsec rule and for secure servers, you'll need to add an authentication method for each IPsec rule.