GameBench Mobile Game Testing Methodology

What is a GameBench badge?

A GameBench badge is an assessment of the graphical performance of a particular game running on a particular device.

We don’t currently try to badge a game or device in isolation because it’s the combination of hardware and software (or what we call the “game-device pair”) that determines whether a gamer will experience smooth and responsive gameplay.

Over time, however, as we build up our database of badges, we'll be able to aggregate them to provide a more device-centric or game-centric view. For example, if the Apple iPhone 8 tends to have better frame rates (or worse battery life) than the Samsung Galaxy S8 across a sample of popular games, then we hope to be able to reflect that in simple, singular badges for those devices.

What’s the point of these badges?

GameBench is committed to raising the standard of mobile gaming by setting meaningful benchmarks for the way commercially-available games should run on commercially-available devices.

Our badges are a quick, simple way to represent these benchmarks, allowing both gamers and game developers to judge the objective quality of a particular gaming experience and, where appropriate, encouraging them to seek something better.

What do the badges mean?

Ultra performance --This badge is awarded to game-device pairings that deliver excellent graphical fluidity of at least 60 frames per second (fps), without any undue compromise on other aspects of gameplay such as graphical quality or resolution. Frame rate is the best objective correlate of perceived smoothness and responsiveness in gaming, and 60fps has long been established in the PC and console gaming worlds as a target for excellent playability.

Smooth performance -- This badge acknowledges that gamers don't always need 60fps to have a good time. Some genres of game are inherently slower-paced and strategic enough that they can be played very well at 30fps. Other games may need to animate quicker, perhaps because they involve rapid reflexes or realistic physic simulation, but they can still deliver good playability if the frame rate hovers around 40fps, for example, or if graphical fidelity or resolution has to be reduced in order to maintain this fluidity.

Basic performance -- A gaming experience may be postive even when it isn't very smooth. For example, a gamer with a low-cost, low-end device may prefer to have the option to play a particular strategy game, even if it renders at only 20fps, rather than having his or her device blacklisted and prevented from playing the game at all. So long as the gamer knows what to expect, and so long as the game never totally grinds to a halt (e.g., dropping below 18fps), this may be acceptable -- and that's what the GameBench Basic badge represents.

Poor performance -- Any gaming experience that contains touch-responsive animation, even if it's the most basic or deliberately retro-style animation, should at least deliver this animation in a way that is comfortable to view and responsive to user inputs. As frame rate drops below 20fps, both these attributes are lost.

Good battery life -- This badge is awarded to game-device pairings that last more than five hours on a charge. This longevity is in turn a product of two factors: the average current draw of the workload (in milliamps, mA) and the physical battery capacity of the device (in milliamp-hours, mAh).

Okay battery life -- This badge means that the game-device pairing lasts 3-5 hours on a charge. In other words, your battery shouldn't drain more than 10 percent during a 20-minute gameplay session.

Poor battery life -- The game-device pairing runs for less than three hours on a charge, potentially forcing the gamer to make a choice between playing the game or preserving other key phone functions (such as making and receiving calls).

How does GameBench assess each rating?

We're constantly trying to improve GameBench's rating methodology and thresholds to make them as useful as possible to gamers. In this endeavour, we actively seek feedback from enthusiasts, studios, platform-holders and hardware companies.

Our early badges, covering older devices and game versions, were assessed on a pretty simple combination of metrics:

Median frame rate - the typical frame rate during a gaming experience;
Frame rate drops - the worst frame rate that the gamer experienced, even if only momentarily.

Starting in early 2018, we started to develop algorithms that automatically assess a marked region of gameplay (excluding any menus or loading screens) and take account not only of the above two metrics, but also:

Variability - the average jump between consecutive frame rates;
Frame-to-screen sync - the percentage of a gameplay segment that is spent within a particular range of frame rate values that is suitable for the refresh rate of the device and therefore likely to be free of "judder";
Estimated resolution - games often do not render at the native screen resolution of a device, so we estimate resolution at the start of gameplay to ensure that high frame rates are not being achieved at the expense of poor visual quality.

Together, this suite of metrics helps us to ensure that our Ultra badge is only awarded to genuine 60fps+ experiences that don't achieve high frame rates at the expense of consistency, judder or reduced graphical quality.

How is testing conducted?

Our results and ratings apply to real gaming tests conducted by real people at a given point in time. GameBench ensures that its testers follow a strict testing methodology and we verify all results prior to publication.

Our methodology is designed to ensure that tests accurately reflect a real gaming experience while at the same time controlling variables that do not stem from the optimisation of the game-device pair.

Here are some of the variables we seek to capture:

The objective quality of optimisation of the game-device pair as experienced by a user;
The performance impact of thermal throttling after prolonged gameplay (which is why we test for at least 15 minutes);
Battery drain that occurred during gameplay.

Here are some of the variables we seek to exclude:

Device settings such as connectivity and brightness;
Background activity on the device (such as messaging clients etc.)
The impact of non-playable parts of a game, such as menu screens, loading screens and victory animations.

Finally, there are some variables that we cannot control and do not pretend to:

Gamer behaviour will inevitably vary slightly across tests, although we ensure that the same areas of a game are tested for the same duration, our results cannot be extrapolated beyond the specific test session they measure;
We cannot account for hidden changes or inaccuracies in the way the underlying layers the operating system on an Android or iOS device report their behaviour to developers -- just like the rest of ecosystem, we rely on such underlying tools (such as Apple Instruments) to be accurate, as an inaccuracy will lead not only in incorrect results but also to poor optimisation in the long-term.

Due to these uncontrolled variables, we offer our results and ratings with the disclaimer that, while we do our utmost to accurately reflect underlying data exposed to the GameBench tool by a test device, we cannot guarantee the accuracy of these underlying, which are outside of our control. Furthermore, we cannot guarantee that our results are perfectly repeatable or capable of being extrapolated beyond the specific test session that they describe.

How can I get badges myself?

GameBench is DIY compatible! There’s nothing to stop you predicting what badge a mobile product would get if GameBench tested it officially.

First, download our tools:

Then do the following:

Minimise connectivity and background activity on your device (stop all background apps, notifications, push/pull, only keep WiFi on);
Set a moderate screen brightness (60% brightness on an iPhone 8 is a good standard);
Play for 15 minutes of warm-up time, then profile the next 8 minutes with GameBench;
Use the marker feature in the GameBench Web Dashboard to reveal metrics for the 8 minutes of gameplay.

Note: If you’d like to publish results or ratings for your product, please get in touch with GameBench Labs via badges@gamebench.net. We’ll get back to you within 24 hours.

GameBench Testing Methodology