Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
General: Two Thumbs Way Down
Stradden
Managing EditorMember CommonPosts: 6,696
MMORPG.com columnist Justin Webb writes this article about Metacritic, a great resource for single player games that doesn't necessarily take into account the intricacies of MMORPG reviewing.
As most of you probably know, Metacritic is an aggregate review site. It takes reviews from "reputable" sites, converts those reviews into percentage scores, and then averages them, resulting in a "metacritic score". Metacritic can be a useful tool if you want information regarding a new game. What's the "word on the street"? Is the general consensus good or bad? When a new-release game costs sixty bucks, this kind of consumer information is really useful to have at your fingertips. Who wants to spend $60 on a crap game?
Given that a Metacritic score is an average of review scores from multiple popular gaming sites, it sounds like it should be pretty accurate, right? Well, here are the Metacritic scores (as of 2/16/2010, in descending order) for some "recent" and "big-name" MMORPGs:
Read Two Thumbs Way Down.
Cheers,
Jon Wood
Managing Editor
MMORPG.com
Comments
STO higher than UO??
I need to leave this planet ASAP.
There is no knowledge that is no power.
Hi Justin
Quote: "So, how come these scores are so messed up?"
Maybe you could elaborate whats so wrong with those scores? Looks good to me. Hard to understand your article if I can't see the your point of view.
Edit: Newer Games (see STO) might move up a bit more in Metacritic as more Reviews are released but thats about the only one that seems a bit ona low the rest looks very much like mainstream consensus.
I dont really buy alot of games tbh, I am very well acquainted with my tastes, and usually know before hand if Im going to like a game or not. And Im usually right. So reviews are useless to me. I read them after I play games though, to see what people thought about the things I experienced in-game.
If you stand VERY still, and close your eyes, after a minute you can actually FEEL the universe revolving around PvP.
Hello, just something worth taking a look at is the way as Metacritic uses the reviews here
www.metacritic.com/about/scoring.shtml
They are weighted averages and we definitely do not now how weighted they are.
Moreover lots of the linked reviews are not accessible anymore.
I agree with the final opinion that for some games (and especially mmos) the resulting score is not appropriate, but for 90% of them I usually agree with the score they give.
I suppose it also depend on the state when mmos are delivered which is usually a beta, and the reviewers gives out their score for that as well.
This last is definitely the reason behind the whole thing.
Cheers.
I agree. Those scores don't seem outrageously inaccurate. Maybe WoW isn't that great of a game, but a lot of people seem to like it and play it. And maybe STO is a good game, but it's getting a lot of hate in forums and reviews because it's not MMO enough, you can never satisfy the hard-core fans of an IP, and it's not cheap enough for some people.
With youtube vids, beta breaches, copies being released before "official" release dates, and general word of mouth, who the hell needs Metacritic or any other review site nowadays?
Well first, before saying the scoring system is flawed and innaccurate you should look at yourself first cause your using the same system on this site. Game score is an avrage of all the scores, maybe that's why you have a game like Atlantica Online being rated #1.
However i dont think a "serious" review site would give an MMO all 10's or all 1's just to try to cheat like we have here from users making multiples accounts just to bump or lower a game rating.
In ALL a system that makes an average of all the scores seems accurate to me, as long as the reviewers are somehow serious in their jobs. As far as the scores in this article, they seems FINE to me. Of course WOW is number #1, but WOW is like McDonalds.
Anyway, that shows a little how weighted the score are. We can imagine how one score on IGN can bump up or down a game'S score. Yes, the more reviews Metacritic aggregates (in my example there are only four), the closer to "reality" the score should be, as excessive enthusiasm should be pondered by excessive negativism, and vice versa.
And yet, we still got skewed scores.
Honestly I'd never heard of this site before and now that I know about it I doubt I'll bother with it for the reasons you described in this article.
I'd like to say that I'm disappointed that developers would take a number from one site and base all their praise (or lack there of) on it. I'd like to think that developers have people who go out an research what people actually say about their products and look at things beyond just a made up score. But unfortunately I know better.
Nice read.
When you say MMORPG's are not easily reviewed, I think you really hit the nail on the head. One of the X factors that is (almost) never mentioned when reviewing an MMORPG, is the community. Community is generally something that's beyond the marketing departments mental grasp, and therefore beyond their control, yet is crucial to a games success or failure.
Bit of a tangent, but it's something I've been thinking about, and I think it's somewhat related to your article.
In my opinion MMOs should be reviewed yearly, and the older reviews and average still displayed, but remain for year X.
So MMOs should have:
1- The aggregate score from all years and that is what is displayed for users on the search, the main score.
2- The aggregate score of each year and their review listings.
3- And a system in place to automatically make these changes as each MMO completes an year of existence so that reviewers can notice the missing reviews for the year.
Now, good luck convincing Metacritic and the industry that MMOs are software that isn't static and get content patches for free frequently (it's a damn evolving world) including DRASTIC gameplay changes... completely opposite to single-players where content patches will be usually either DLC or Expansions. Shouldn't really be hard to apply when they understand this aspect of MMOs, this would be already a great step into giving MMOs some justice in reviews.
Anyway, I do use Metacritic for weighting some game purchases, especially console ones.
Well i say mmorpg.com over years ain't good reviewers also so you can hold hand with metacritics hehe.
But i agree on mmorpg been reviewed get scores when launched and it dont chance anymore harm done, there are plenty mmo's out there that are so much better after 6 months but never got a chance to recover from those bad reviews even some died while there very good after 1 year or more.
I dont buy games becouse of reviews i investigate myself follow i game i like and with info developer, forums and players who play beta i deside if its worth or not if i think is risk buying at launch i just wait for while and see if its more polished.
So no mis buy on my part.
Games played:AC1-Darktide'99-2000-AC2-Darktide/dawnsong2003-2005,Lineage2-2005-2006 and now Darkfall-2009.....
In between WoW few months AoC few months and some f2p also all very short few weeks.
I heartily agree with this, as MMO development is generally an evolutionary process.
Metacritic gives you a number and points you to all of the reviews that go into said number. It's up to you to determine how useful that number is.
Until you cancel your subscription, you are only helping to continue the cycle of mediocrity.
Good post!
You can definitely see a change of moods and standards in those scores. I think the low UO score comes largely from having little to compare it to, for example. I think reviewers likely compared it to the Ultima series and thought, "wow, it's like other Ultima games except without the purpose. Oh, and getting PK'ed all the time."
2003-2004 roll along, and it seems that no one could do any wrong. AoC would come along a few years later with the bait and switch that is Tortage, which totally skews the game perception to be something that it isn't. This would give it undeservedly high reviews for all the wrong reasons.
The most interesting to me is WAR. I recall the consensus being that it was an attempt to be WoW 2.0 with a deeper pvp focus and that they had pulled that off to some degree. But the devil was in the details, or the lack thereof, and now it seems more like WoW .3 with a few interesting innovations.
It is tough to review MMO's, I agree with Justin. To make a fair assessment, you'd have to play for months. The only REASON you'd do this as a reviewer is if you enjoy the game. So then, only "fair" assessments would be done by people that enjoy the game, thus skewing the chart even worse. And that's not even talking about the logistics of being a paid reviewer getting a few hundred bucks for 100's of hours of your time.
This hits the nail on the head for me and pretty much sums up my feelings and input.
Your fail comment, failed.
Yes but that's what he's saying. That number can be excessive or correct; but there's no way for us as consumers to know how many numbers went into creating that number (the number of reviewers) and there is no way to verify how accurate the score is versus real world play, and finally that this number is totally static and won't be dynamically altered to reflect patches and new content.
Plus to add insult to injury the CEO's of all these MMOG's treat the damn number as if it's from God himself; and also it's a statistic of a statistic which anyone who knows statistics knows is a BS number!
This is the Harry Mudd dilemma. "Everything Harry tells you is a lie! Listen carefully, Norman, I am lying!" You can't tell which scores are accurate or not. The only thing to do is to ignore the whole lot of them!
"In individuals, insanity is rare; but in groups, parties, nations, and epochs it is the rule." Neitzsche
This hits the nail on the head for me and pretty much sums up my feelings and input.
I'll never buy a MMO without a solid 20 hours to play it beforehand. That's usually enough time for me to know if it will at least amuse me for a short time.
General scores mean little to me, I agree there. But specific details about game mechanics are important to me in a review. When reviewers bring up how the "crafting" works in STO, for example, I think that's an extremely important and invaluable piece of information.
But yeah, general, average percentile scores mean very little to me. A game can get review scores based largely on a mode of gameplay I may have 0 interest in in the first place.
Sites like Metacritic and Gamerankings.com suck.
If you follow those scores like a sheep you're part of the problem.
Playing: Rift, LotRO
Waiting on: GW2, BP
Yes but that's what he's saying. That number can be excessive or correct; but there's no way for us as consumers to know how many numbers went into creating that number (the number of reviewers) and there is no way to verify how accurate the score is versus real world play, and finally that this number is totally static and won't be dynamically altered to reflect patches and new content.
Plus to add insult to injury the CEO's of all these MMOG's treat the damn number as if it's from God himself; and also it's a statistic of a statistic which anyone who knows statistics knows is a BS number!
This is the Harry Mudd dilemma. "Everything Harry tells you is a lie! Listen carefully, Norman, I am lying!" You can't tell which scores are accurate or not. The only thing to do is to ignore the whole lot of them!
I realize that. My point is that it's up to you to decide how useful the metacritic score is just as you would a single review from ign or whoever.
Until you cancel your subscription, you are only helping to continue the cycle of mediocrity.
Webb,
I understand your frustration with metacritic and it I am a user of the service. Although, I don't follow it to the letter of the law. I do use it as a guideline useful to gather information from various sources. One thing I did look at is the user reviews that seem to vary wildly. If you take a look at champions online it swings between 8's and 3's, there is even a 0 in there. Out of 176 user votes it scored a 6.3.
Granted, not every game is for everyone. But if you put your ear to the ground you can usually get a vibe from the community about mmo's. Sometimes just good ole viral marketing is the best way to get a feel to what the community thinks. I read forums, nurmerous blogs and listen to some podcasts to get a feel for the overall vibe. And from what I gather about champs online is true. It's really not that great of a game. It might be the end all, be all for some players but not the overall.
I also believe in the increasing law of averages, if someone likes something they will tell one. If they hate it they tell 10, that 10 try it and tell 100. It works against most MMO's except for Darkfall. It has had so much negative press that it makes it interesting.
INB4: any talk of sheep or lemmings, let me tell you. I usually will try it out for myself, even if it gets bad reviews. If I want to play it, I play it. If it stinks I tell someone and keep it moving.
The problem in comparing that to lets say World of Warcraft: wrath expansion. They are rating an endgame focused expansion to a brand new game at level 1. This runs into the starter area problem as you said in your post. Could they rate Champions endgame content. No, because Champions was notoriously lacking in endgame content. So you really can't compare.
At the end of the day, I don't think metacritic is the law. But it does keep many game publshers/designers honest. For how many years have we put down cold hard cash hoping for the daimond, only to find coal in our stockings. We just don't want to get duped anymore and if this means that game companies have to be wary of the almight meta, then that might make them more apt to make a better game.
In that case everyone wins.
You should have also mentioned not listening to players on the MMORPG.com forums, because a great majority of the threads are troll-posts.
fail...
if a MMO does not grab me in the first 10 minutes of play, nothing but money wasted
if your game cannot pull that *simple* feat off, you fail
Excellent article - I had always ignored the Metacritics score on Steam because it never seemed to apply to anything. And now that I've seen how it's created, I have a more logical reason for ignoring it than just my gut feelings.
I also like the idea that you should wait 1 month after launch to do a review on a MMO -- it just makes a lot of sense. Er, didn't you guys just review Global Agenda though?
I agree. Those scores don't seem outrageously inaccurate. Maybe WoW isn't that great of a game, but a lot of people seem to like it and play it. And maybe STO is a good game, but it's getting a lot of hate in forums and reviews because it's not MMO enough, you can never satisfy the hard-core fans of an IP, and it's not cheap enough for some people.
STO is full of problems but the OP makes a good point most reviews are one time only, hell most links are broken, there is no re evaluation or big picture.
Consoles are "release once" while mmos are "growing boys" and need more attention....or in the case of STO a big slap across the face.