How much wood would a woodchuck chuck; If a woodchuck could
chuck wood?
The fundamentals of benchmarking are the three Rs: relevance, robustness,
and repeatability. How many times have we seen the results of a benchmark and
said, "What the heck is a Hellmark?" If the number is higher, is that
better or worse? Who is Erastosthenes anyway, and what does a Sieve have to do
with reality? Unless you're a theoretical mathematician, not much. We believe
it's important to use real-world applications in everyday situations to test
performance.
Windows NT brings new challenges to benchmarking: 32-bit programs,
preemptive multitasking, threads, security, multi-CPU support cross-platform
compatibility (Intel, PowerPC, MIPS, and Alpha), and more. The old 16-bit test
suites for DOS/Windows won't cut it anymore. Simply porting them to Windows NT
won't answer the burning questions: What does Windows NT uniquely offer to the
computing industry? Is Windows NT scaleable? Is RISC faster than Intel on NT?
Does it run on all platforms? These are the questions we are being asked.
Fortunately, the answers are being worked on.
BAPCO has announced SYSmark for Windows NT, a workstation benchmark that
rlms a set of Microsoft Test scripts against popular desktop applications: Word,
Excel, PowerPoint (16-bit), Welcom Software's Texim Projec@ and Orcad MaxEDA
(PCB design tool). In addition, BAPCO is working on a version that will test
the file and print capabilities of Windows NT Server using the SYSmark clients.
On the database server side, the TPC Transaction Processing Council)
benchmarks are the current leader. TPC-C tries to model a moderate-to-complex
OLTP system based on an order entry system. Transactions are submitted from
UNIX dumb terminals that interact with nine tables on the server database
(approximately 25OGB) and are measured in transactions per minute (tpm). TPC-D
benchmarks decision support by using a database stress test and complex queries.
Client Server Labs in Atlanta, Georgia, has announced an independent test
suite that runs three simultaneous loads on Windows NT Server. The test is
based on three benchmarks: a modified TPC-C using PCs instead of dumb terminals,
TPC-D, and BAPCO (file serving). The relative performances of these three
workloads are combined into a single measurement called an RPMark.
At Windows NT Magazine, our strategic direction is to adopt or
create benchmarks that satisfy the three Rs. However, we don't want to invent
these tools in a vacuum. We have invited representatives from the major chip
vendors (Digital, Motorola, Intel, MIPS) and database vendors to work with us in
developing these benchmarks. I invite you to participate with us in this
challenge. Please let us know the types of questions you would like answered by
benchmarks and the types of real-world measurements you would like to see used.
You can email your comments to: benchmarks@winntmag.com.