We have a WAN with connections to several business partners, and we want to protect our more critical servers from being accessed by people outside our organization but on the WAN. Can we configure a Windows server to be accessible only to clients in certain IP subnets without using a firewall?
Yes, you can easily implement such a rule with IP Security Policies on Windows 2000 Server and Windows Server 2003 machines. Note that IP Security Policies are not exactly the same as IP Security (IPSec). Windows implements IPSec through IP Security Policies, but IPSec is only part of IP Security Policies.
IP Security Policies allow you to filter traffic according to IP address, protocol, or port number. Then you can choose to allow the filtered packets, block the packets, or establish an IPSec connection to protect the packets. In your case, you'd create an IP Security Policy with one rule and configure the rule's filter list to catch all packets with a source IP address that falls within the subnets of your business partners. Then you'd select the block action for the rule. You can configure all the desired servers at the same time by creating the IP Security Policy in a Group Policy Object (GPO) that's applied to the servers. Of course, filtering based on source address can be defeated if a business partner spoofs an IP address so that it appears to originate within one of your trusted subnets, but this method is probably secure enough for most situations.
If you want rock solid security, though, consider actually implementing IPSec on sensitive servers for authentication purposes. Instead of a policy that blocks packets from untrusted subnets, create a policy with a filter that catches all traffic and then requires IPSec to be negotiated. In fact, a prebuilt IP Security Policy called Secure Server (Require Security) will work for you. You'll also need to enable the Client (Respond Only) prebuilt IP Security Policy on all legitimate clients of the sensitive servers. Only clients that are part of your forest will be able to authenticate because the default IP Security Policies base IPSec authentication on Kerberos and rely on both computers having accounts in the forest.
After you implement IPSec on a server, when a client tries to connect to the server, the server will respond with a request to negotiate IPSec. If the client fails to comply, the server will block all traffic from the client.