  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
| Intel Cheats in 3D Performance Benchmarks
This isn't terribly game related, since no gamer in their right mind would use an integrated Intel chipset as their main graphics card, but it's still interesting.
»techreport.com/articles.x/17732 -- It takes two to lie, Marge. One to lie, and one to listen. |
|
  Mchart Super Joe
join:2004-01-21 Gurnee, IL
·AT&T Yahoo
·RoadRunner Cable
| I don't see what the problem is. It's not like they are reducing video quality to get the FPS gains. They are doing something that actually probably took quite a bit of work to pull off and it's cool if you ask me. GPU overloaded? Offload some of that work to the CPU. Doesn't seem like cheating to me. -- THIS IS SPENCER. MISSION ACCOMPLISHED - I HAVE JOE. RETURNING TO BASE. |
|
  TK421 Premium join:2004-12-19 Canada
| Driver optimizations are common and should be considered fine if they can sqeeze a few more FPS without losing image qaulity. AMD, Nvidia, and Intel all do it.
The problem here though is manipulating a benchmark result for marketing purposes.
How do Intel's driver optimizations affect the competitive landscape? To find out, we assembled an AMD 785G-based system that's pretty comparable to the G41 rig we used for testing: Athlon II X2 250 processor, Gigabyte GA-MA785GPMT-UD2H motherboard, the same Raptor hard drive, and 4GB of DDR3 memory running at 800MHz with looser timings than the Intel system. We even disabled the board's dedicated sideport graphics memory, forcing the GPU to share system RAM like the G41.
With the Futuremark-approved Catalyst 9.9 drivers, the AMD 785G-based system scored 2161 in 3DMark Vantagenearly the same score as the 2132 3DMarks the G41 gets when it's playing by the rules, but well below the 2931 the score the G41 posts with optimizations enabled. (Renaming the Vantage executable on the AMD system had no notable effect on benchmark scores.) The app-specific optimization gives the G41 a definitive lead in 3DMark Vantage.
Here's the tricky part: the very same 785G system managed 30 frames per second in Crysis: Warhead, which is twice the frame rate of the G41 with all its vertex offloading mojo in action. The G41's new-found dominance in 3DMark doesn't translate to superior gaming performance, even in this game targeted by the same optimization.
That's no great shock, really. We've seen Intel's integrated graphics solutions thoroughly outclassed by rivals from AMD and Nvidia on multiple occasions.
All of which brings us back to the perils of using 3DMark Vantage as a substitute or proxy for testing real games. Those perils are well established by now. PC makers and others in positions of influence would do well to re-train their focus on real applicationsespecially for testing integrated graphics solutions, which have no need of advanced graphics workloads based on the latest version of DirectX to push their limits. 3DMark's traditionally purported role as a predictor of future game workloads makes little sense in the context of these modest IGPs.
In summary, Intel violated Futuremarks explicit rules prohibiting targeting 3DMark optimizing by recognizing the executable (proven by renaming the executable for testing). Intel gets a higher 3DMark score than their competition yet even with specific real game optimizations still can not get half the real performance!
Nothing really shocking but it demonstrates again that as techreport says: "3DMark's traditionally purported role as a predictor of future game workloads makes little sense in the context of these modest IGPs." |
|
  SKYHN Lu.. Lu.. Lulululu Premium join:2001-09-16 99999 | reply to knightry Nothing that ATI, AMD, Nvidia, even Matrox have not done before. -- Do you douche with coconut juice? |
|
  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
| reply to knightry The problem is that they've done something specifically banned by 3DVantage.
And just because other companies have done it means it should be acceptable? This seems like a dangerous precedent. -- It takes two to lie, Marge. One to lie, and one to listen. |
|
  Mchart Super Joe
join:2004-01-21 Gurnee, IL
·AT&T Yahoo
·RoadRunner Cable
| Precedent? It was done first by Nvidia YEARS ago and ATI.
I have absolutely NO problem with a company doing this if they can improve the performance without decreasing visual quality. There should not be any complaints if the visaul quality is not decreased.
Repeat after me. There is nothing wrong with increasing performance if visual quality is not decreased.
And FYI, no one gives a hoot or a holler about 3DMark anymore. Considering that Intel's fixes also work in Crysis and other games - Cool. It means the customer is getting more for his money. -- THIS IS SPENCER. MISSION ACCOMPLISHED - I HAVE JOE. RETURNING TO BASE. |
|
 Joe12345678
join:2003-07-22 Des Plaines, IL
| reply to knightry said by knightry :This isn't terribly game related, since no gamer in their right mind would use an integrated Intel chipset as their main graphics card, but it's still interesting. » techreport.com/articles.x/17732 apple may start putting this carp in the new minis. |
|
  elios
join:2005-11-15 Springfield, MO | reply to knightry its only a cheat if it ONLY does it on the benchmark if it does it with games to its not a cheap its not a bad plan really with multicore cpus your going to 1-2 cores not fully loaded what not use that power for some thing? |
|
  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
| said by elios :its only a cheat if it ONLY does it on the benchmark if it does it with games to its not a cheap its not a bad plan really with multicore cpus your going to 1-2 cores not fully loaded what not use that power for some thing? Look, I'm not saying for them to use it in games is a bad thing, but using it on the Vantage Benchmark is strictly prohibited (whether they do it for games too or not).
quote: With the exception of configuring the correct rendering mode on multi-GPU systems, it is prohibited for the driver to detect the launch of 3DMark Vantage executable and to alter, replace or override any quality parameters or parts of the benchmark workload based on the detection. Optimizations in the driver that utilize empirical data of 3DMark Vantage workloads are prohibited.
-- It takes two to lie, Marge. One to lie, and one to listen. |
|
  elios
join:2005-11-15 Springfield, MO | rename the benchmark see if it does it if it does its not cheating if it doesnt its cheating
pretty easy imo if it DOES IT IN EVERY THING its not cheat if its ONLY doing it based on detecting what ap is running then yes its cheating |
|
  Mchart Super Joe
join:2004-01-21 Gurnee, IL
·AT&T Yahoo
·RoadRunner Cable
| Well, it has to do it based on detecting an application as they have written application specific profiles for each application.
If you say 'cheating' in these terms that specifically is implying that the graphics are being degraded to achieve better performance - Like nvidia and ATI did a while back. However, Intel is not doing this. They are offloading some of the work to the CPU.
End of discussion. -- THIS IS SPENCER. MISSION ACCOMPLISHED - I HAVE JOE. RETURNING TO BASE. |
|
  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
| said by Mchart :Well, it has to do it based on detecting an application as they have written application specific profiles for each application. If you say 'cheating' in these terms that specifically is implying that the graphics are being degraded to achieve better performance - Like nvidia and ATI did a while back. However, Intel is not doing this. They are offloading some of the work to the CPU. End of discussion. Look, I'm sorry to belabor the point, but Vantage is very clear about this. I quote again, my point being bold:
quote: With the exception of configuring the correct rendering mode on multi-GPU systems, it is prohibited for the driver to detect the launch of 3DMark Vantage executable and to alter, replace or override any quality parameters or parts of the benchmark workload based on the detection. Optimizations in the driver that utilize empirical data of 3DMark Vantage workloads are prohibited.
Hell, I don't even care they they did it, but they broke the rules, which is cheating by my books. -- It takes two to lie, Marge. One to lie, and one to listen. |
|
  Mchart Super Joe
join:2004-01-21 Gurnee, IL
·AT&T Yahoo
·RoadRunner Cable
1 edit | said by knightry :said by Mchart :Well, it has to do it based on detecting an application as they have written application specific profiles for each application. If you say 'cheating' in these terms that specifically is implying that the graphics are being degraded to achieve better performance - Like nvidia and ATI did a while back. However, Intel is not doing this. They are offloading some of the work to the CPU. End of discussion. Look, I'm sorry to belabor the point, but Vantage is very clear about this. I quote again, my point being bold: quote: With the exception of configuring the correct rendering mode on multi-GPU systems, it is prohibited for the driver to detect the launch of 3DMark Vantage executable and to alter, replace or override any quality parameters or parts of the benchmark workload based on the detection. Optimizations in the driver that utilize empirical data of 3DMark Vantage workloads are prohibited.
Hell, I don't even care they they did it, but they broke the rules, which is cheating by my books. And?
I don't care. Nor does most anyone else. If the quality isn't being changed - No one cares. You didn't bother to bold the part of the quote that actually matters - And thats the line that says if the quality is altered to get said quality gains. -- THIS IS SPENCER. MISSION ACCOMPLISHED - I HAVE JOE. RETURNING TO BASE. |
|
  elios
join:2005-11-15 Springfield, MO
·Mediacom
| reply to knightry also who says there detecting any thing? ever think MAYBE thats just how the driver works high load on GPU -> off load to CPU?
really try it out rename the benchmark to some some thing random and see what it does
then try the same with a game if its still off loading its just how works |
|
 Margolis Premium join:2003-11-24 Saint Louis, MO
·Charter Pipeline
1 edit | reply to knightry intel didn't cheat. This is a VERY BIASED article. the original article snipped only a small part of the 3dmark rules that they thought would back up their biased claims.
The very first rule from 3dmark concerning cheating:
It is prohibited to for the driver to change the rendering quality level, rendering technique or replace or alter any shaders or other parts of the workload from that what is requested by 3DMark.
and they ignore the most important part of the quote they do use.
to alter, replace or override any quality parameters or parts of the benchmark workload based on the detection
they are NOT replacing any quality parameters or any of the workload. They are just offloading parts to the cpu to help out. |
|
  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
1 edit | reply to elios said by elios :also who says there detecting any thing? ever think MAYBE thats just how the driver works high load on GPU -> off load to CPU? really try it out rename the benchmark to some some thing random and see what it does then try the same with a game if its still off loading its just how works This is what the writers of the article did. They renamed the EXE and lost the performance increase.
After a closer read, I see the point of some of the others here. Since they didn't affect the quality, it seems like the change they made should be okay. Though I'd still like to hear 3DMark's statement on whether they feel the modification is allowed. After all, the test is supposed to be a stress-test of the GPU, which can easily be argued to not be accurate if some of the processing of the GPU is off-loaded to the CPU.
edit: fixed typo -- It takes two to lie, Marge. One to lie, and one to listen. |
|
 Margolis Premium join:2003-11-24 Saint Louis, MO | reply to knightry the thing is, ms is doing this for games also, not just benchmark programs like 3dmark. If they only did it for 3dmark and never helped other games it would be one thing, but they are helping other games perform better also. |
|
  knightry I'm Not Fat, I'm Festively Plump
join:2002-05-06 Oviedo, FL
| said by Margolis :the thing is, ms is doing this for games also, not just benchmark programs like 3dmark. If they only did it for 3dmark and never helped other games it would be one thing, but they are helping other games perform better also. True, though only for a very very small subset of the games out there. I feel like it would be more fair if their offloading were something more generally applicable, not just for the few they've picked. -- It takes two to lie, Marge. One to lie, and one to listen. |
|
 Margolis Premium join:2003-11-24 Saint Louis, MO
·Charter Pipeline
| said by knightry :said by Margolis :the thing is, ms is doing this for games also, not just benchmark programs like 3dmark. If they only did it for 3dmark and never helped other games it would be one thing, but they are helping other games perform better also. True, though only for a very very small subset of the games out there. I feel like it would be more fair if their offloading were something more generally applicable, not just for the few they've picked. It may not work for all. So they are testing and tweaking what they know it works for. I'm sure as time progresses they will add more games to the list. |
|
  Dream Killer Graveyard Shift Premium join:2002-08-09 Forest Hills, NY clubs: | reply to knightry I don't see what the problem is. Nothing is wrong with tweaking drivers for a unique executable. Both ATi and Nvidia tweaks driver profiles for each game. It's like tuning a race car's settings to the race track before each race. |
|