Microsoft's Web Capacity Analysis Tool makes
it easy and efficient
Last month, I explained that bigger is not always better, and that you can buy too much hardware for some Web site environments. I've seen several Web performance tests done on dual Pentium Pro 200 machines, and I haven't been impressed with the performance of my quad Pentium Pro 200 machine. So I decided to run some performance tests of my own. This month, I'll tell you how I tested the performance of my Web environment and show you what the results mean.
I wanted to make the tests as easy as possible, yet informative enough for
me to draw some conclusions. So I grabbed the Microsoft Windows NT Server
Resource Kit Version 4.0, Supplement 1 and loaded the Microsoft Web Capacity
Analysis Tool (WCAT). WCAT lets you run a simulated workload on a Web server and
comes with several prepared scripts that let you simulate different types of
environments.
You can run any of four types of WCAT tests:
- Basic tests: These tests consist of static pages of different file sizes.
- Common Gateway Interface (CGI) and Internet Server API (ISAPI) tests:
These tests consist of pages that you dynamically create.
- HTTP keepalive tests: These tests are variations of the basic tests and
CGI and ISAPI tests that use the HTTP keepalive feature in HTTP 1.1.
- Secure Sockets Layer (SSL) tests: These tests are variations of the first
two tests, but they use the SSL encryption protocol, which is typically slower
because of the overhead involved.
You can customize the WCAT test environment to mimic your Web environment.
For example, I could have created the scripts required to simulate the Windows
NT Magazine Web environment (i.e., most of the pages dynamically created
with Cold Fusion). But I wanted to try WCAT right out of the box to check its
ease-of-use before I started tweaking it. Besides, because I was just glad to
have access to the 22 machines I needed to run the tests when I needed them, I
wasn't about to mess around with the configuration.
To run WCAT, you need a Web server, a controller server, and some clients.
The Web server contains all the files that the WCAT tests request. The
controller server is the test administrator. It tells the clients what to do and
for how long. It then monitors the Web server and collects data from the clients
at the end of the test. The client machines produce a load on the Web server.
Each client machine can support up to 200 virtual clients.
Using WCAT, I wanted to evaluate the performance differences between a dual
Pentium Pro 200 Web server and a quad Pentium Pro 200 Web server. To minimize
the number of variables involved, I used one machine to serve as both Web
servers and set it up to be two different machines during different stages of
the testing. The Windows NT Magazine Lab lent me a quad Pentium Pro 200
with 512MB of RAM. To level the playing field for all the tests and to simulate
practical configurations, I configured the machine to use just 256MB of RAM
during the tests. Figure 1, page 211, shows my test configuration.
To set the number of processors and the amount of memory without cracking
the case on the machine and yanking out RAM and processors, I used a setting in
the Boot.ini file to specify the configuration I wanted to use. In this file,
you can specify the number of processors and the maximum amount of memory
available to NT. To do so, you need to add the line
multi(0)disk(0)rdisk(0)partition(1)\WINNTSER="Windows NT Server Version
4.00" /NUMPROCS=2 MAXMEM=256
This approach is not suitable for true benchmark testing, but for my
purposes, this configuration worked perfectly.
For the controller server, I used a second quad Pentium Pro 200. You don't
need to use a machine this big for this task, but the Windows NT Magazine
Lab already had this system set up as a controller. For the clients, I used 20
Pentium 133 machines, each with 32MB of RAM, and added virtual clients to each
of these machines. Finally, I performed the tests on a network made up of 100Mb
connections, because I knew that this many clients would easily saturate a 10Mb
network.
Following the WCAT installation directions to the letter, I first installed
WCAT on the Web server. Then I installed and configured WCAT on the controller
server and the 20 client machines. Setting up the tests was easy. However,
installing and configuring 20 client machines can get tedious. Fortunately, you
have to set them up only once.
At this point, I tested the WCAT client's ability. I didn't want to
overload the clients and have them become a potential problem or bottleneck, so
I performed a few tests to determine how many virtual clients each physical
client could sustain. For this part of the exercise, I used the ISAPI WCAT test
that I planned to use for the primary testing and kept adding virtual clients to
each physical client until I started to see errors. I discovered that each
client could sustain 60 virtual clients. So with 20 client machines, I was able
to simulate a maximum of 1200 total clients.
The Test
After you finalize your test configuration, all you must do is decide which
test to run. I wanted to make the testing realistic (i.e., I wanted a test that
combined static and dynamic pages), so I ran one of the ISAPI tests. The test I
selected consisted of half dynamic pages and half static pages. I ran the test
using 5 client machines (300 virtual clients), 10 client machines (600 virtual
clients), 15 client machines (900 virtual clients), and then 20 client machines
(1200 virtual clients). I tested these settings for both the dual Pentium Pro
200 and the quad Pentium Pro 200. I considered conducting this test with only
one Pentium Pro 200 running, but the test choked after I started the 10 client
machines. After I discovered that this configuration couldn't make the grade, I
decided to stay with the original test configuration--dual versus quad.
With all the clients running and waiting, you only need to edit the test
configuration file to tell it how many client machines to use, how many virtual
clients on each, and how long to run the test (for my tests, the duration was 10
minutes). Then, from the command prompt, you simply type
run TESTNAME
where TESTNAME is the name of the WCAT test you want to use (in my case, the
test name was isapi50). The controller server takes care of the rest and tells
the clients everything they need to know.
Although WCAT is handy for this type of testing, Microsoft never intended
WCAT to be an official benchmark or an extremely scientific test. This
limitation doesn't mean you can't get down to some specific numbers about the
performance of your Web site with WCAT; I think you can. However, for my
purposes, I just wanted to run a test with everything equal except the number of
processors until the machines choked. That way, I could see what the results
told me about the benefits of beefing up my Web server hardware.
The Results
So what did the test tell me? I learned that in this environment, a quad
Pentium Pro 200 does not have a significant advantage over a dual Pentium Pro
200. Throughout the tests, the results were almost identical for both
configurations. As you see in Graphs 1 and 2, the dual and quad Pentium Pro 200
machines have almost identical performance numbers for 300, 600, and 900
clients. Both configurations have unacceptable numbers for 1200 clients because
of several connection errors. This falloff is probably the result of saturating
the network somewhere between 900 and 1200 clients. But as you can see, both
configurations handled the saturation in the same way. The test shows that the
dual Pentium Pro 200 machine can do everything the quad Pentium Pro 200 machine
can do up to the point of saturation of the 100Mb network (translation--save
your money or spend it on several Web servers rather than one big Web server).
WCAT is an excellent tool that makes performing consistent tests a breeze.
With the proper environment and this type of tool, you can really fine-tune your
Web server. I plan to use WCAT a lot in the near future. I've found several
interesting Registry hacks I want to try out and some tuning suggestions for
Cold Fusion. With WCAT, you run the test against your Web server, make whatever
tweaks you want, and simply run the test again. The results can easily tell you
if you're headed in the right direction.