Ups and Downs in Benchmarking Far Cry 2
 

The Far Cry 2 benchmark tool owns all other built in benchmark tools around. So far there are only two issues we have with it. We want it to be able to take screenshots at a specific frame or at specific intervals, and we want it to be just a little more stable. The former issue is of less importance than the latter, as there are other ways to take screenshots. But stability is the only thing that really stands in the way of some incredible analysis. If running more than 5 to 10 different test cases, we've found the tool to be likely not to complete. If individual runs fail, we'd still like to see the benchmark continue, but unfortunately it just stops.

This is a big issue when performing 60 runs. That's right, 2 DX versions, 2 AA settings (off and 4x), 3 quality settings, and 5 resolutions. It's a data collectors dream come true. Not only do they keep CSV files with frame data for all frames from each of the 3 loops we are running per test, but they build a nice html file with all the data for easy access and even display a graph of instantaneous framerate per frame. Even though I haven't had the chance to re-run our AMD numbers, here's a sample of what we're looking at:

http://www.anandtech.com/weblog/showpost.aspx?i=514