The Follies of Benchmarking
With the advent of the Internet, information on computer hardware has become widely available to anyone with Internet access. Thanks to a large variety of sources, such as manufacturers as well as third party sources like us (Games First) and Tom’s Hardware Guide, users can find information to help them decide on hardware purchases before spending their hard-earned money. This is great as long as the user can interpret the information so it translates into usable knowledge about how the hardware will be used.
I don’t know how many times I’ve purchased a new piece of hardware for my computer after carefully reading about it, taken it home and haven’t gotten the performance levels I’ve read about. A good example of this is video cards. Most of the time, manufacturers or third parties claim, oh, let’s say 50 Frames Per Second (FPS) on a P II 300. However, after taking the card home you find that the maximum you get on your P II 300 is about 46 FPS. Does it mean that their claims are wrong" The answer is no, not necessarily. Most benchmarks are made under ideal conditions, which never occur in most people’s machine. The first and most common reason that benchmarks are faster is that they are performed on a clean install of Windows. By a clean install I mean an install of windows with no other programs except the benchmarking software and the most basic necessary drivers. These conditions alone can cause an increase of about 4-7 FPS on average, and sometimes even as high as 9 or 10 on some systems that are really loaded down.
By now you should be asking, “what should I do to get those sweet benchmarks that I read about” There is a simple answer, and that is to turn off all programs running in the background besides the desired application. Doing this one thing can sometimes net you 2 or 3 FPS all by itself. Barring a clean windows install, the only other option is to expect about 3-4 FPS loss and learn to live with it (the horror!).
Now before someone jumps on my back about this, I’m not saying this is always the case, far from it. I’m just saying that the majority of gamers aren’t going to see the exact benchmark numbers that they see on the web. Most–and I stress–most of the time the numbers will be a little shy of the claimed numbers
With this in mind, I plan to test most of the hardware I review on my system at home (Pentium 2 300 w/128 meg of ram). My machine is used mostly for gaming and the Internet and is loaded down with tons o’ software. My machine most of the time will produce numbers that you should expect to see on your typical Pentium II 300. The benchmarks I plan to use include X, Forsaken, Quake 2, and Turok: Dinosaur Hunter. This should give readers a good judge of both Open GL and Direct 3D performance on a typical gaming system.