The Benchmark
Since we are focused on one game, our testing methodologies will be a little different this time around, and we'll take the opportunity to go in-depth on our testing method for this game as well. We will look at 6 different levels of the game, and for each Level Analysis, we will look at image quality as well as demo benchmark performance.One of the largest caveats about benchmarking in FarCry is that demos don't work like one would expect. For example, in Unreal Tournament 2004, we can start a game, play our hearts out, start and stop recording somewhere in the middle, and we have a very cool little benchmark of the action. This repeatable benchmark is a fair representative of gameplay as far as benchmarks go. This seems reasonable for a game with a built-in demo mode.
FarCry, on the other hand, will record the movement of the player through a level without recording any of the players actions (like firing a weapon or pressing a button to open a door), and none of the other characters in the level are recorded either. When a demo recorded in single player mode is viewed, all the AI controlled characters man the same posts that they would in the game; only they ignore the player moving through the level when the demo is running.
This means that some demos have instances of passing through locked doors, and AI bots making their normal rounds regardless of the player (and can be in different locations for different runs of the benchmark). And as if this wasn't enough, the worst part of the whole experience (have you figured it out yet?) is that demos are entirely absent of any fighting, conflict, or gunfire.
We tried many times to benchmark this game using FRAPS, but our ability to be repeatable was worse than what demo playback gives us.
So, why are we at all OK with using FarCry's built-in demo mode? Because much of FarCry game play has to do with sneaking around, walking through the levels, and taking in the scenery. No, it's not the all-encompassing perfect benchmark, but it isn't the worst thing that we've seen either (*cough* - 3dmark - *cough*). We've compared the demo mode to our very non-repeatable FRAPS benchmarks of walking around levels and we are comfortable with the reliability of the scores that we get from the demo for that purpose.
Also, when we were informed that this patch was coming down the pipe, NVIDIA sent along a couple demos to test the performance difference with the new SM3.0 path enabled. Ubisoft is going to include these 4 demos with their patch, but we were obviously a little wary of just throwing these numbers up. We took a close look at the demos, and we are including them alongside our original custom demo and a new custom demo that we recorded for this article. The reason why we are including the NVIDIA provided demos is that they are definitely sections of the game that are really parts of the gameplay. Whether these are representative of overall gameplay or not, there are definitely experiences in the single player mode of the game that are represented by the demos.
It is important to keep the numbers in this test in perspective. We are specifically investigating the benefit and impact of CryTek's new SM3.0 path. To explore the technology fully, it is necessary to look both at parts of the game that benefit most from the update as well as those that don't see that much impact. We trust that the demos provided by NVIDIA will highlight the best results that we can expect to see, and it just so happened that our original demo and the new one, which we recorded, see less change (as we will soon discover).
Facebook
Twitter
RSS
36 Comments
View All Comments
Illissius - Friday, July 2, 2004 - link
For these benches, were nVidia's trilinear and/or anisotropic optimizations on or off? (This would help in comparing results with other sites, for example.) I don't recall seeing them mentioned, but they're getting to be as important as the driver revision these days.DerekWilson - Friday, July 2, 2004 - link
DAPUNISHER:I'm not sure about the 64bit version of any game, as game developers are much more likely to hold everything until MS releases WinXP64 than hardware vendors. My guess is that we can expect not to see any visual improvements or differences with the 64bit move. There's much less reason to alter the graphics of the game when gaining more registers and memory address space than when you add the ability to do conditional rendering, floating point frame buffers, instancing, and all that...
Zak,
We could try to guess performance based on these numbers:
http://anandtech.com/video/showdoc.html?i=2044&...
But we didn't use those because they're based on the 1.1 version of farcry under dx9b and Catalyst 4.4 ...
Our focus was the impact of SM3.0, not on overall relative performance, but in the future we will include older generation cards even when looking at next gen features. You are right, it does provide a way to relate to the numbers, and those cards should be in there for completeness' sake as well. Thanks for the suggestion.
Zak - Friday, July 2, 2004 - link
I wish you guys would include one or two benchmarks on some older video cards to give a point of reference for those, such as myself, who still run R9800 and older generation cards. Without seeing how the game performs on R9800 or eqivalent card it's hard to relate to these benchmarks.Zak
DAPUNISHER - Friday, July 2, 2004 - link
Very impressed with the GT's performance in this version. When can we expect your preview of FarCry 64bit version with the SM3 path Derek? and will 64bit bring some new eye candy or more performance? Inquiring minds want to know :-)Warder45 - Friday, July 2, 2004 - link
Interesting. Some odd stuff like ATI's X800 line actually decreasing in performace with the 1.2 patch. I wonder if thats a driver issue that now needs to be fixed, but if I was an ATI owner I'd stick with the 1.1 version of the game. I'd really like to see someone benchmark with omega's drivers for ATI, and see if there's any difference in performace there.#9, That article at tom's is from the NV40 review months ago. This new verison, 1.2 fixes most of the IQ problems nvidia was having.
http://www.xbitlabs.com/articles/video/display/far...
So far the only IQ problem I've seen mentioned with the new version.
araczynski - Friday, July 2, 2004 - link
very nice, that 68UE sure is kicking some tail, before AND after the SM3.perhaps this will lead to developers optomizing (to some small degree at elast) their code for the 2 camps? (or at least for the camp that pays them the most...)
in any case, here's to hoping the 68U/UE are priced acceptably by xmas, or at least next tax time :)
Shad0hawK - Friday, July 2, 2004 - link
"Then, during the ATi refresh,we will all be greeted the the Geforce 6900, 6900 Ultra, 6900 Turbo and 6900 Ultra Hyper Fighting Edition."actually that will probobly be after ATI anounces the super golden/silver platinum extra extra XT edition with not only one but TWO "free" certificates for games not out yet
nserra - Friday, July 2, 2004 - link
#1, #2, #3, #4:I think the huge hit is because nvidia is not doing AA to the all scene as ati does.
The new drivers from nvidia have this ability. How do you think nvidia have come to top so soon, after some driver release ... Trilinear optimizations, Shader optimizations and now AA optimizations...
I also don't understand why only toms site notes differences between ati and nvidia image quality...
http://graphics.tomshardware.com/graphic/20040414/...
ZobarStyl - Friday, July 2, 2004 - link
Bearxor, though I agree the overclocked editions are silly, don't act like ATi doesn't do the exact same thing...Ultra Extreme = XT Platinum Edition
Ultra = XT
GT = Pro
vanilla 6800 has no direct competitor, but it held it's own occasionally against the Pro in the review.
Both of the double-named cards are just the top end overclocked, so I tend to ignore them in the reviews, but then the GT was beating all of the ATi cards in some of those demos too...
RyanVM - Friday, July 2, 2004 - link
#6, Ditto :p. Dell 2001FP for life :D