Overclocker Cafe - A Hardware Tech Site for the Rest of Us.

Applicability of Futuremark becoming hazy - Editor's Opinion

by Dean Barker (1/19/2004)

There has been quite a bit of press in the past six months about Futuremark, NVidia and the alleged “optimizing” aka cheating, that may or may not have been occurring. We have always included Futuremark’s 3DMark2003 in our testing in an effort to provide another perspective or ‘piece of the pie’ so to say, in our benchmarking. Earlier today, we received an e-mail from Futuremark requesting that we pull the 3DMark2003 results Brian graphed in his Connect3D Radeon 9600 XT review. As a reason for this, they stated that the NVidia 53.03 drivers we used were not on the “approved” driver list as documented on the Futuremark site. You may or may not recall Futuremark making an attempt to correct any problems that may or may not have been occurring with cheating by putting out an approved driver list. This was the same time lots of folks were saying shame on NVidia and Futuremark at perceived cheating. When I looked at the approved driver list on the Futuremark site, the 53.03 NEDs were not listed. The ONLY NVidia approved drivers listed were the 52.16s with this disturbing cautionary message.

Notes: The NVIDIA ForceWare 52.16 Drivers have been tested with the GeForce3 series, GeForce4 series and the GeForceFX series. The drivers have not been tested with the MX-series! There are no Win9x or WinME WHQL drivers available supporting all the GeForce series. The 52.16 drivers have 3DMark03 specific optimization for the Pixel Shader 2.0 test and that score is solely comparable between nvidia cards.

This was obviously disturbing because not only was there only ONE “approved” driver but that sole "approved" driver by Futuremark’s own admission, has ‘specific optimizations.’ How can Futuremark be used in benchmarking when they say only one driver is okay and that specific driver gives advantages to NVidia that could be exploited by use of 3D Mark 2003? What’s worse is that how does a driver that is clearly documented by Futuremark, as "optimized" make it to the “approved” list.

Taking this one step further; Futuremark made their request that we pull results generated by a non-"approved" driver on an ATi review with an NVidia product used as the comparison card. My last VGA review was on the Albatron GeForce 5950 and I used the same 53.03 driver. At that time, I remarked on how the synthetic benchmark (Futuremark) didn’t jive with the gaming benchmarks we did. Out of curiosity, I reran 3D Mark 2003 using the “approved” driver and ended up with the same results – which by the way, now are questionable given the information Futuremark themselves provided as listed above. It almost comes across as if Futuremark is happy only as long as ATi isn’t besting NVidia on its benchmark. That is of course only my opinion. You draw your own conclusions. I want to say I do not know nor have heard of any questionable ties between Futuremark and NVidia nor do I wish to suggest collusion between the two. As for us here, we are now one of the numerous other sites who will not be using 3D Mark 2003 again.

-dean

HOME