Further Proof Quantified Rating Systems Are Flawed


It’s no secret that there’s a definite skew in quantified ratings for video games. It’s not necessarily that they’re a bad idea. Quantified ratings seem to work quite well for movies and other forms of mass media. So why does it not work in video games?

The idea of quantifiable ratings has again been brought to the forefront because of the AAA title Uncharted 3 and the review written by Simon Parkin on Eurogamer. It was quite a well written review. It was both insightful and critical of some points of the game. He praised the feel of the game and the tremendous cinematics, yet criticized the lack of player agency and the occasional mechanic like a shotgun taking too many hits to kill an opponent. It was a very fair, impartial review when read as written word, but what all too many ended up seeing was its quantified score: 8/10.

An 8/10 was one of the lower scores Uncharted 3: Drake’s Deception received. According to Metacritic, the only score worse than that was a 5/10 from The AV Club. Naturally the critic was criticized by the general gamer community. The majority of professional critics (56/62) gave the game a 90/100 or higher to give the game a total metascore of 93. The disparity in score from expectation is what drew the most complaints.

An 8/10 in a 1-10 rating system shouldn’t be considered that big of a disparity, but in the reality of video game reviews the rating system is really based on a 7-10 scale with anything scoring below a 7 considered a game most likely not worth a purchase. The readers immediately began piping up comparing his review and its score to other websites. One commenter stated “Parkin is a bit of a tit to be honest. 5/5 from Giant Bomb.”

These comments and reader responses, coupled with the surprisingly low score for the game led Eurogamer to give the developer Naughty Dog the chance to respond and clear their good game. Naughty Dog’s response can get summed up  with “Lemarchand admitted that he cares about review scores “more than I should”, but wished they didn’t exist.” (Photo: courtesy Metafuture)

The reaction to the review from both Naughty Dog and the readers isn’t the problem with quantified rating systems for video games, but it is a symptom. The greater issue, and one that professional game reviewers often have to grapple with is a combination of issues including the time-sensitive nature of reviews, the developers controlling the flow of review games, and the cost and time commitment required to review the games.

The time commitment needed to play so many of these games is pretty astounding. RPGs are 30 hour ordeals minimum, and the average single player campaigns in other genres is around eight hours long with a sizable multiplayer portion. In order for a reviewer to put in enough time to properly review the games they either need to play constantly starting with the release date, or they need a head start in the game before the release.

Games can also be cost-prohibitive for a reviewer. At roughly $60.00 a pop, a reviewer deciding to purchase the game will come as an especially hard blow if it’s bad. In order for a reviewer to not seem like they are just playing the zero risk AAA blockbuster games or just giving all games favorable reviews, reviewers will have to risk bad games and wasted money. Trust me, it hurts. The only real way to get around this is to obtain a copy for free somehow, usually from PR contacts.

Those who control the flow of review copies are developers and their PR agents who, of course, are most interested in seeing game sales increase. Good reviews do a lot to help those sales so they naturally want to see the games go to the journalists who are most likely to review the games well. This in turn causes many journalists to want to review the games well in hopes of receiving a continuing stream of games. This, in turn, leads to the shrinking of review ranges to a 7-10 basis. There aren’t any bad people in this cycle, though. There are just people doing their jobs and attempting to do them well.

These practices are no secret, but when they reach the headlines many still feel shocked. Recently EA was caught sending a questionnaire to reviewers looking to get a hold of Battlefield 3. Four months ago Duke Nukem Forever‘s PR group threatened to not give any more review copies to reviewers who they felt were unfairly critical of the game. As that previous article shows, it’s not really uncommon for a publisher to blacklist the media or vice versa, but it is uncommon for it to be publicized as these past two incidents were.

We see these issues crop up more in the video game world than in most other forms of mass media and there are a few good reasons for this. Movies and music are a form of mass media that ask very little time and money commitment from us. Movies will be about three hours tops and cost from $10-$20 to go see on your own. Music has short, easily digestible songs and again will cost roughly $15. This means that unless you want to write a review before the album or movie comes out, it’s easy to experience entirely on the day of its release and have a thoughtful review out the day of. This means review copies aren’t in such high demand and aren’t necessary for those looking to write about them.

What about books that aren’t quite so quickly experienced? Surely journalists get early releases from publishers, right? That is true, but if you look at the most popular book review sources vs. the most popular movie review sources one thing becomes quite clear. There isn’t a quantifiable reviewing scale. Aside from small little quotes and phrases that can be pulled from book reviews, there is no easy way to gleam their meaning.

I will reiterate my previous (and I am sure obvious) thoughts now. We need to do away with the 1-10 or 1-100 scale of reviews and instead focus on real, critical insight into the game. Hundreds of people work on these games we play and they don’t deserve cold numbers to assess their works. They deserve proper journalism and critical assessment. Without these the gaming industry cannot grow as an art form. Honest critical assessment is needed and honest, critical assessment cannot come from a simple numerical score.

If we continue down this path it will be increasingly more common for AAA titles to get a 6/5 rating, or for developers to lash out at reviewers for not giving the game the score they marketed it to be. Naughty Dog’s lead designer, Richard Lemarchand, told EuroGamer “I’ve said for a long time it’s rather a shame we have to give video games scores in reviews… It’s just part of the way things are done, isn’t it?” Perhaps it’s time for developers, publishers and journalists to actually try to take a stand and change how things are done.

About Stephen Crane

Stephen was hooked by the NES at a very young age and never looked back. He games on a daily basis and is currently trying to climb his way up the ranked ladder on League of Legends! Outside of the video game world he actually likes running and owns a rapidly growing collection of toed shoes. Stephen Crane is the owner of Armed Gamer.

Recommended for you