Pushing the pace for file-sharing performance
Recently, the Windows NT Magazine Lab handed me this hot assignment: Produce performance benchmarks for Microsoft Windows NT Server 4.0 and Novell NetWare 4.1. Why is this topic so hot? Because chances are really good that at least one of two industry giants will not like the results. Undaunted by the prospect of attracting the wrath of a major corporation, I gathered up my gear and headed to the Lab's speedway.
For my performance tests, I used the Lab's standard configuration: a set of
client machines on a 100Mbps Ethernet network that simulates the workload of
multiple users. (For details about the Lab's test environment, see the sidebar,
"The Benchmarking Speedway," page 65.) One server, running either NT
Server 4.0 or NetWare 4.1, tackled the workload of the client machines. Because
I wanted to test file-sharing performance, I employed Bluecurve's Dynameasure
for File Services 1.5 as the workload engine. (For information about this
product, see Lab Reports: "Dynameasure Enterprise 1.5," September
1997.) The combination of Dynameasure and the Lab's test environment let me
simulate workloads that users typically perform and pinpoint potential
bottlenecks to ensure that I stressed the server, not the network or clients.
After a month of testing and reviewing reams of data, graphs, and tables, I
determined a clear winner: The checkered flag goes to...NT Server 4.0!
Preparing the Track
I wrestled with several background issues going into these tests. My biggest
concern was how to create a fair comparative evaluation. Different people often
have different priorities when they evaluate a product. For example, when my
wife and I decide to buy a new car, we define our needs and select criteria
(e.g., price, speed, appearance, hauling capacity, headroom) to measure a car's
potential value. Even if we agree on the criteria, we don't always agree on the
importance of each item; picking a clear, overall winner is difficult.
Fortunately, we agree that one criterion outweighs all others: performance.
Performance is also a primary criterion that people use to evaluate file
servers. Although other factors (price, support, compatibility,
interoperability, etc.) matter, performance is of utmost concern. I based my
evaluation and conclusion on only how well NT Server 4.0 and NetWare 4.1
performed in the Lab's test environment under Dynameasure for File Services 1.5.
If criteria other than performance are more important to you (e.g., your primary
concern is to find a file server operating system that runs on a 386), you might
come to a different conclusion.
The Speedway Judges
In the performance tests, I measured three main areas: throughput, average
response time (ART), and motors per step (MPS). Throughput (measured in
kilobytes per second--KBps) is the total number of bytes all the motors copy
during the measurement phase of a step divided by the elapsed time of the
measurement phase. Throughput measures system capacity. The type of transaction,
the number of motors, and the hardware capacity of the system influence
throughput. Higher throughput means greater system capacity.
ART is the average time in seconds to complete a transaction during the
measurement phase of each step. ART measures the speed of the test system. The
type of transaction, the number of motors, and the hardware capacity of the
system also influence ART. Lower ART means the system is faster.
The third measurement, MPS, is the number of motors that report results for
each step of the test. MPS measures the total number of assigned motors in a
step that complete the transaction. MPS is a direct measure of load on the
system. Higher MPS means greater load.
I approached the benchmarking process as if I were testing two unique
racing teams and pit crews. Each team (NT and NetWare) used the same physical
track (the Lab's network) and the same physical cars (clients and motors). Each
team had equal time (about 16.5 minutes) to complete as many laps as possible
(throughput). I tracked average lap times (ART) and the number of cars that
completed the race in a given time frame (MPS).
Start Your Engines
Establishing a test environment that could run both NT and NetWare was the
first order of business. For the test server, I used a generic PC clone with the
following hardware configuration: a 120MHz Pentium, 64MB of RAM, a master 2.1GB
hard disk (EIDE), a slave 2.1GB hard disk (EIDE), and a Novell NE 2000 Socket EA
network adapter. Software configuration included NetWare 4.1 with 10 license
connections and NT 4.0 with Service Pack 3 (SP3). I partitioned the slave 2.1GB
hard disk into two equal areas--one for NTFS and the other for Novell's file
system. I added this test system to the Lab test environment to measure
performance with the Dynameasure software.
I chose six of the Lab's clients, running NT Workstation 4.0, as my user
testbed. I configured each client with eight or nine motors to simulate a total
of 50 network users. NWLink IPX/SPX bound all client adapters. I performed some
preliminary tests with both the Microsoft NetWare client/protocol software and
the Novell-supplied NetWare client/protocol software. I used the same testing
specifications I planned to use for the benchmark tests, and I saw no
performance difference between the two client/protocol software products.
Throughout the benchmark performance tests, I used the Microsoft client/protocol
software.
Time Trials
I ran several initial Dynameasure tests just to warm up the track. At this
point, I wanted to identify any bottlenecks that could affect the results. In
particular, I wanted to eliminate the possibility that the client systems or the
network bandwidth could degrade performance. The warm-up tests ran the
Dynameasure for File Services Copy All Bidirectional test configured for a 5.6MB
dataset; a 1KB block size; 10-second think time; and 6 steps, with the following
number of motors assigned to each step: 5, 10, 20, 30, 40, and 50.
The Copy All Bidirectional test consists of 16 different transactions in
which compressed data, uncompressed data, binary files, text files, and image
files are copied between the server and the clients. Based on this test (and
with help from Bluecurve's technical support team), I was able to ensure that I
was not overstressing the client workstations or the network. The summary
reports for NT and NetWare showed that server performance began to degrade after
reaching approximately 25 motors--throughput leveled off, ART rose, and MPS
declined.
During these warm-up tests, NT vastly outperformed NetWare, as the
Dynameasure graphs in Screen 1 and
Screen 2 show. In Screen 1, the left graph
displays throughput, and the right graph displays ART. Screen 2 displays
throughput in the left graph and MPS in the right graph. NT maintained higher
throughput, lower ART, and higher MPS in these tests.
For several reasons, the Lab had hypothesized that NetWare would have
higher throughput and be faster during certain types of transactions (such as
copying small files). For example, NetWare 4.1 includes Packet Burst technology,
which lets a server transmit several packets in a burst, without waiting for
verification that each packet has been received. NetWare 4.1 also supports Large
Internet Packets, which lets the server and workstation communicate using the
largest possible frame size.
The Copy All Bidirectional test includes eight different types of data,
with two files for each data type: one file for client-to-server transactions
and one file for server-to-client transactions. Because the type of transaction
influences the three benchmark measures, I decided to break out individual
transactions and compare the results to get additional information. This view of
the data would let me compare server performance based on the type of
transaction each server completed.