Troubleshooting IPsec is not simple to configure. You might find that you have to troubleshoot configurations, preshared keys, and policies before you can get two systems to communicate when IPsec is required. This will likely be the case in complex configurations in which you're using AH and ESP in addition to ESP NULL.
To help you troubleshoot, you can use Network Monitor 3.2, which runs on Windows Server 2008, Windows Server 2003, Windows Vista, and Windows XP. Network Monitor lets you see IKE packets being exchanged between Windows systems and UNIX and Linux systems. You can also see the proposals being made by the end systems, which lets you see if the correct proposals are being offered based on your choices in GPOs. You can download this free utility from the Microsoft Download Center.
On Solaris, you can start in.iked in debug mode by specifying –d on the command line. In debug mode, the daemon won't disassociate from the terminal and will log output to standard output. You'll see a wealth of information that lets you troubleshoot why IPsec communications are failing.
On FreeBSD and Linux systems, you can start racoon in the foreground by specifying
racoon –F
on the command line. Adding the –d and –v flags will produce a lot of debug information that you can use. You can add –l and the name of a log file to have racoon log to a file. The –v, -d and –l flags can be used when racoon runs in the foreground or background. The returned information includes a proposal-by-proposal comparison provided by the remote system with acceptable proposals defined on the local system, which is very useful in ensuring that there are matching proposals. You can check which policies are loaded into the kernel by running the command
setkey –DP
You can also check for established Security Associations (i.e., successful IPsec associations) by running
setkey –D
from the command line.
Use Preshared Keys to Establish IPsec Communications
As you have seen, you can use IPsec with preshared keys to establish IPsec communications between Windows and common UNIX and Linux variants. Although using preshared keys isn't considered as secure as using certificates or Kerberos, preshared keys are supported by a wide range of systems. In addition, preshared keys are easier to configure than certificates or Kerberos. By using IPsec with preshared keys, you can extend SDI to your UNIX and Linux systems.