You preview just about every single game, you interview developers, publishers, marketers, sound engineers, etc etc etc. Take all of that information they gave you. Then take all the information from the back of the box. List it all. Install the game and begin playing. Did they bull**** you or were they honest? Is everything that they, through you, said would be there, there and fully functional? Is the combat as revolutionary as they claimed? Are the graphics really as awe inspring as the screenshots you have been pushing like crack in actual play, or do you have to have a 50 grand super computer to run it? Are the different systems everything that they portrayed them to be?
If the answer is unequivocally yes then tell us. But, when it isn't you have to tell us. You also cannot possibly expect us to accept some nonsense like, well while this feature hasn't made it in, I talked to so and so developer and they assured me that it would be sometime in the next few weeks but wouldn't tell me exactly when. Look at it like you are a teacher. Little devvie has a project due on Tuesday. He hands in his project but tells you that he was not able to do the whole thing and asks for an extension. Well, in a school where acheivement and education is important little devvie gets an F and devvie's mommy and daddy get to come down and talk to the teacher and the principal. If a developer promises the sun, the moon, and all the stars but only delivers on the sun they should be held accountable.
Keep all the personal opinions seperate from the important stuff, because opinions are just that. Opinions. Some more educated than others, but it doesn't matter. There are some 7 billion people on the planet and every last one of us is a wholly unique individual and every last one of us have our own opinions lol. In the response to me Stradden uses the term analyst. Analysis and opinion are separate. Analysis uses measurable criteria. Opinion does not an analyst make.
I can't speak for anyone else but I never pay attention to the score anyway. the way I research my games (in general, not just MMO's) is to read multiple reviews of the game, take all the points that are common to all the reviews, then decide if those things are positive or negative IN MY OPINION.
there are a lot of reasons for a low score but not all of those reasons are necessarily a deterrent. I've played games in the past that were universally panned in reviews but I enjoyed all the same.
with MMO's its even easier cause one of the main put offs are broken games but with MMo's that cam be just a temporary condition. If I really like the concept behind a broken game, I'll just wait till they fix it then I'll pick it up. I'm in a minority like that tho, once you've given a poor score to a game its over. no matter how good a game may become the average player has already put it behind them.
so my suggestion is to not make a scoring system at all, just state the pros & cons then let the readers decide what it means to them
Good luck with that lol. Remember you can't please all of the people all of the time. Heck its hard enough trying to please some of them some of the time.
Personally i read the review mainly to see if the game is buggy and how polished it is. I ignore scores because everyone's tastes are different and i like to make my own mind up.
I think categories could be useful as long as they are relatve to the genre.
Content (is there more stuff to do than is required to level up or does it run out and i need to grind some activity to progress)
Replayability (will i want to go back and start again with a different toon, is there more than 1 starting area)
Polish (how many bugs? are they game breaking or just annoying? how does the game run on minimum specs...do you guys even run mmo's on different computers to see how it goes?)
Socialization (does the game promote soloing over grouping while leveling or are they both supported, is the world open or instanced? are there good guild tools like guild banks etc, is there a viably economy, decent non-combat activities such as crafting, how is the chat system, is it easy to use)
I can't speak for anyone else but I never pay attention to the score anyway. the way I research my games (in general, not just MMO's) is to read multiple reviews of the game, take all the points that are common to all the reviews, then decide if those things are positive or negative IN MY OPINION.
there are a lot of reasons for a low score but not all of those reasons are necessarily a deterrent. I've played games in the past that were universally panned in reviews but I enjoyed all the same.
with MMO's its even easier cause one of the main put offs are broken games but with MMo's that cam be just a temporary condition. If I really like the concept behind a broken game, I'll just wait till they fix it then I'll pick it up. I'm in a minority like that tho, once you've given a poor score to a game its over. no matter how good a game may become the average player has already put it behind them.
so my suggestion is to not make a scoring system at all, just state the pros & cons then let the readers decide what it means to them
Have "game impressions" for the initial review and follow them up soon after with "longevity reviews" with reviewers going to max level and then some. Maybe find a way to combine them in one final review article when they are both in. (Make visible that the final score is still in the works and that important information is still lacking at this time).
Scoring:
Only use the mmorpg.com average member score as a numbered score: it is most often a very fair score of a released game after some time. And if the reviewer's own score is way off from the average one, it always looks kind of funny. (Also this empowers us to a limited degree and makes us feel more involved
But do describe key-factors with terms like great, good, average, bad, terrible. Etc.
Key factors to take into account:
- Graphics, sound and animations
- Polish and stability
- Gameplay and game mechanics (perhaps specify for group, solo, and combat gameplay)
- PVE content
- PvP content
- Immersion value / RP content
- Longevity
Alternatively, if you REALLY NEED to give your own rating, make a nice formula and give an average score of the above key factors. And while waiting on a longevity review you could add 'preliminary score' to your initial 'game impression' score.
Actually I think breaking down a game into sections of say Aesthetics (yep ok I do check out the escapist website and enjoy Extra Credits), sound, gamplay and such would be a good idea and for sure should be given equal weight. People like different things and as such some prefer aestheticly pleasing over gameplay and some prefer the other way round and then look to say sound to be the tiebreaker. Not sure if there needs to be a average of the grades or not at the end maybe a table at the end with the catagories and grades. Oh and potential needs a grade of it's own with qualifiers from the reviewer on weather this is potential derived from if they fix this stuff it would work better or potential derived from if they can add this feature it would work better.
As to doing the beta/release day preview followed up by a review that seems feasable and makes alot of sence. The preview should probably focus on is this worth the box price rather than long term things. Stuff like how easy was it to install, how long did it take to patch, was it easy to get into the game, did the previewer have any troubles getting stuff to work and if they did how did customer service react. Maybe a quick rundown on the systems and if they are working as intended or need fixing. Basicly the goal should be to inform the reader if they are going to at least get a months worth of entertainment out of the product because thats all they have invested at this point. If possible maybe highlight any improvements or breakdowns from the switch from Beta to Launch. Another thing could be how long did the reviewer get to play in this limited time. Was it just a few hours? What kind of stuff did the preveiwer do in that time?
The review I think should focus more on long term stuff and could probably wait till close to the end of the first month. Things to include would be how did the game improve from launch to the end of the first month or was it a case of 1 step forward 2 steps back? Were any new systems/content added in this first month or was it all spent on bug fixing? Did anything start to grate on the reviewers ie annoying sounds, disconnects with skills and animations those kind of things that you can usually ignore or miss initially but build up over time to become immersion breaking.
I also agree with the idea put forward by several posters that more than one person should give an opinion incorperating both veiws into 1 article probably the way to go though that would deffinatly make the editors job harder.
The lesser of two evils is still evil.
There is nothing more dangerous than a true believer.
You preview just about every single game, you interview developers, publishers, marketers, sound engineers, etc etc etc. Take all of that information they gave you. Then take all the information from the back of the box. List it all. Install the game and begin playing. Did they bull**** you or were they honest? Is everything that they, through you, said would be there, there and fully functional? Is the combat as revolutionary as they claimed? Are the graphics really as awe inspring as the screenshots you have been pushing like crack in actual play, or do you have to have a 50 grand super computer to run it? Are the different systems everything that they portrayed them to be?
Here's the issue. A lot of those games will advertise the epic features that are often found only in end-game when you've reached Lv.50-70 or whatever the level is or at least spent a considerable amount of hours into the game. It's very hard to review an MMORPG in such case because it can takes weeks to reach that level or threshold and as mentionned by Jon, they need to post reviews as close to launch as possible to inform the readers on the product. So in that sense, they can only cover the features they have access to within the few hours they play. For exemple I believe EuroGamer reviewers are tasked to play the game for a total of 20 hours before making a review... even if that's not always the case *points at Ed Zitron*. But nonetheless, it's very hard to critique everything that's been said when so little time is available.
In any case, this isn't the thread to talk about whether or not you think our reviews are biased, etc. it's a place to talk about the direction that the format of our reviews should take in the future. We could continue arguing, but it's just going to boil down to the same stuff it always does. People are going to make accusations and I'm going to respond to them and no one will be more convinced of anything either way.
If people want to think we're corrupt, then that's their perogative, I suppose.
So, if I overreacted,then I apologize. I'm tired of having to defend myself from accusations against my integrity on a daily basis and I lashed out.
In any case, let's get this back on topic please.
I feel no apologies are warranted. The poster outright accused you and other journalists of having their opinions bought, and then went on to explain his dream fantasy land where gaming journalists are treated like kings and showered with phat loot on the scale of Will Smiths ultra mega trailor. So you stomped on him hard with both feet. Great job in my book. The world of video game creation is not a glamorous one, and neither is the world of the video game critic.
That said back on track with what the article is about. I've always found the 1-10 scale to be laughably unreliable. When people like something they tend to treat it quite favorably, hence the astonishing number of 8 -10 ratings. When people do not like something they are less inclined to treat it favorably hence the disporpotionate number or 1-2 ratings and an quite a deficite in the 4-7 area.
So here is my idea. STOP RATING THESE GAMES AT ALL. At least numerically.
Instead just tell us what you liked and what you didn't like. And make sure your carefully explain why or why not. Telling me that the game UI feels clunky and difficult will help me out a hell of a lot more than you just saying on "I give it 7 out of 10".
As I explained earlier the 1 - 10 scale is badly skewed twards strong feelings one way or the other, so kick it aside. I recommend you simplify the whole thing by just making it a yes/no or hot/cold vote, because frankly you either liked it or didn't like it. Trying to split hairs by saying I liked it "this" much, is just self denial. End of problem.
Oh and before someone brings up the spector of "what about undecided", if you are a critic getting a hands on look at something and cannot make a decison as to whether you like it or not, you are in the wrong business.(I do suppose that an undecided category would work for an unreleased game, but honeslty I feel it is still a copout.)
Of course the new system won't stop we forum gladiators from yelling our heads off if we disagree with your yes/no vote, but you know what? I doubt even a prolonged global thermo nuclear war would be able to do that.
Oh and another idea would be for you to have more than one set of eyes on a product. Having more than one person expressing their thoughts in a review gives us a better idea of how they were impacted by the whole event. Of course in the case of previews such a thing may not be possible as it is hard enough to get one journalist into these little meetings. And as we both know Will Smith mega ultra trailors don't grow on trees.
"Gypsies, tramps, and thieves, we were called by the Admin of the site . . . "
I recommend that if you continue to use a rubric of any sort, please create a ridiculously detailed one for yourselves (not that we see) and have a series of workshops/classes for all of your reviewers to both teach the use of the rubric and get on the same wavelength. I say this because historically on this site, a 7 will signify a total mess from one guy, and the best thing on the market from another guy. You have reviewers that give pity points, and reviewers that don't. The numbers have become almost meaningless.
Not only that, but everything comes out as a B- to B+. If everyone's going to 'pass' anyway, and all the marks fall within a target area, what's the point in continuing to generate marks?
I actually suggest that you leave behind the current system. Discard the rubric, and discard the single reviewer. Have several of your employees do simultaneous reviews highlighting the merits and flaws that they perceived during a play period (making certain that they do not play with or talk to each other about their reviews until after they've been submitted) so that we get a more well-rounded impression. Let's hear from the bitches AND the pushovers at the same time, instead of one or the other as we have been.
Alternatively, have one employee round up a bunch of non-employee volunteers on the site to do the same thing: play the game for a period and report their fun and frustrations. Then the employee compiles a review from the myriad sources, citing them, and giving a more community-generated impression. A review from one point of view about an MMO really seems to miss the point anyway. I want to hear from the guy that loved it, the guy that sees potential, the guy that broke his keyboard from how much he hated it, and the guy that went 'meh' and couldn't bring himself to do much with it. Multi-player review for a multi-player game. As it is right now, I think most of us are rejecting the reviews you post, and reading the litany of comments afterwards to try and get a more realistic impression than the review managed to give in the first place.
Basically, fix your rubric system and get everyone implementing it onto the same wavelength, or ditch the rubric and use the conflicting wavelengths to create a better review. I vote the second option, but that one takes more balls on your part. *^_^*
Multiple reviewers is a great idea. Two people discussing the game can present more evolved ideas and discussion points than just one reveiwer, and differing opinions bring perspective to the various facets of the games.
Come on. Siskell and Ebert, guys. It's a no brainer! :-)
Multiple reviewers is a great idea. Two people discussing the game can present more evolved ideas and discussion points than just one reveiwer, and differing opinions bring perspective to the various facets of the games.
I agree, sounds like a plan, if you can find more reviewers willing to spend time on a new MMORPG (which for the AAA titles shouldn't be hard to achieve).
And make it 2 people with a very different opinion and taste in MMO games, that'd make it more interesting to read.
Also, if you really want to get rid of the rating system and get radical, do it like Rottentomatoes.com, like Tardcore said, a review with a simple 'yes' or 'no' as end conclusion.
I'd expand on it though and make it 'yes', 'no' or 'maybe'.
And do it for different types of gamers, like 'sandbox gamer', 'themepark gamer' and the Bartle archetypes like explorer, achiever or socializer, so a game could be a 'yes' for a themepark gamer but a no for a sandbox gamer and a maybe for an explorer.
The ease with which predictions are made on these forums: Fratman: "I'm saying Spring 2012 at the earliest [for TOR release]. Anyone still clinging to 2011 is deluding themself at this point."
I literally read every post/comment to this thread and while some great ideas were posted I still don't think the real problem is being addressed. As a personal rule, I don't like or trust video game reviews (MMOs or otherwise) done by the gaming media as it stands in this day and age. For one thing, they're not "reviews". They're exaggerated opinions labeled as a "review". In order for journalistic integrity to be upheld, a real REVIEW needs to be objective. Yes REVIEWS can be objective. Just because 90% of the gaming media aren't objective doesn't mean that they shouldn't be. The gaming media likes to use the excuse that because they themselves are fans of the genre most if not all professional journalistic integrity has to be thrown out of the window and so then it's impossible to deliver an objective REVIEW. To that notion I say Bull%$^&! Until that attitude and notion is addressed and changed, you're going to always have the disconnect that exists between your so called "reviews" and whatever rating scale is being used. It doesn't matter what kind of scale you use (this very site is attempting to change their scale once again). Until you all are able to put the "gamer" part of yourself aside long enough for a "review" to stay journalistically objective, nothing's ever going to change and every three or so years you'll attempt to do again what you're doing now. You ever notice that the preview to any given game is overtly positive and the person previewing the game swears up and down that it's the best thing since sliced bread and then when the review hits (unless its a tripple A title and even those suffer), its all "bring out your pitchforks", and "crucify him". You guys (the gaming media) are flattering yourselves calling this crap that you write a "review". They're mostly filled with "what I didn't like", "what I wished they would have done" and I'm sorry but thats monday morning water cooler talk. IE - An opinion. I have absolutely nothing against personal opinions. I love and welcome them. My problem is reading up to sometimes 3 web pages worth of nothing but personal opinions and it being called a "review". The other part of that problem is when people who can't think for themselves read these negative opinions or better yet, just look at the box score (because seldom do the "review" and the box score match up) and say to themselves - "man that game must really suck" and in turn don't buy it . I've lost count of how many games got a safe score/review or negative score/review that I absolutely loved. As it stands now, I'd trust another developer "reviewing" a game in a heartbeat as opposed to the gaming media because at least with another developer, they understand what goes into crafting a game and why they may have chose to do "A" instead of "B". It's just not coming from a pure place where the gaming media is concerned. As to MMORPG reviews, I like the 3 prong approach that some have suggested here. It makes the most sense in terms of the way an MMO is played.
I believe that the reviewers of this site go too easy on the features of a game that should be downright unacceptable in the MMORPG industry and they cushion the impact of those features to save face with developers so they can be invited to private play sessions and interviews.
Although, if they constantly played the role of "mmorpg watchdog", they would never be invited to anything and would quickly fall out of business. In a paradox, their need to survive directly conflicts with their their desire to be unbiased. They can't help it, that is just the nature of not only the industry but also the entire world's capitalist society. It doesn't pay the bills to be honest and although they try, they cannot ever be 100% brutally honest about anything or they will never make any friends. I believe they try to stay as unbiased as possible but are restricted by too many factors.
I blame society, not the MMORPG.com staff.
At least they didn't sell out like MMOHUT and call Allods "The biggest MMO of the year".
I always kinda liked the multi reviewer style. Have three reveiwers, (with their backgrounds and play preferences listed) and then each have a short blurb dissecting their likes and dislikes of the game. Sometimes you get some really great opposing points about the game, and the reader can value these points based on which playstyle they identify with.
As for the grading scale, it does help to have that sense of standardization there, but I think few people actually buy games based on ratings score. It might be a better indicator of interest in the game and maybe corollate it with the number of page hits or article hits for that game, but overall I'm more interested in the reviewers thoughts than than the numbers.
Thanks for posting an article like this BTW, always like to see it when sites are trying to improve their quality through consumer participation.
I believe that the reviewers of this site go too easy on the features of a game that should be downright unacceptable in the MMORPG industry and they cushion the impact of those features to save face with developers so they can be invited to private play sessions and interviews.
Although, if they constantly played the role of "mmorpg watchdog", they would never be invited to anything and would quickly fall out of business. In a paradox, their need to survive directly conflicts with their their desire to be unbiased. They can't help it, that is just the nature of not only the industry but also the entire world's capitalist society. It doesn't pay the bills to be honest and although they try, they cannot ever be 100% brutally honest about anything or they will never make any friends. I believe they try to stay as unbiased as possible but are restricted by too many factors.
I blame society, not the MMORPG.com staff.
But it's quite hard to do a review without being biased, otherwise you just end up with a list of features accompanied with a description and you can pretty much find the same thing on said MMO's website. It's quite normal for a reviewer to view a feature as perfectly acceptable even if someone else disagrees. Just like you can't make the most "carebear" reviewer go review the most hardcore FFA PvP MMORPG with full loot and Perma Death, or the most hardcore reviewer to review Hello Kitty Online. So I really have to disagree with you that MMORPG.com is simply trying to avoid being too much of a "mmorpg watchdog", and rather that reviewers simply have different opinions on common features.
In fact, I'll go ahead and say that it's simply impossible to make a suitable review without being biased.
I haven't read a game review since YouTube was invented. I watch gameplay videos on YouTube and decide if I'll try the game or not. No review I've ever read has given me more insight about a game than watching a 5 minute gameplay video has given me.
We live in the Web 2.0 era, you're behind the times already only having written reviews. If you really want to distingush yourself from the rest of the gaming sites on the web than video reviews is the way to go. Sure, some sites are already doing this, like Game Trailers, but they aren't MMO specific, and honestly, the reviews I've seen on there aren't great.
As far as the actual grading/reviewing process I would recommend a combination of a lot of what I've seen here.
First thing to do would be to get rid of any grading scale at all. You can still have ratings (because the majority of people need that), but make it very simple. I would suggest a "Buy", "Try" or "Skip" rating scale. Self explanitory, no room for confusion.
Second the review needs to be 3 parts, as many have suggested. Start with a first impressions on day of release, or a week or so later.
Two weeks later, more indepth information on how the game plays, technical information, etc.
Then after 30 days give the final review with the Buy, Try, or Skip rating attached at the end. This would be a review of parts 1 and 2 + additional information like progression info, griding scale, etc.
And of course, ALL IN VIDEO FORMAT!
Edit: The reason I suggest a 3 part review is because if you do one on day 1, and then one on day 30, people will have lost interest. Having a shorter time span between each part keeps people's attention.
I would actually watch the reviews on here if they were done like this!
"There is as yet insufficient data for a meaningful answer."
Why should scores be inflated and merely not graded on their own merits? Why does CrappyCo's game's score get inflated just because Stellar Inc made something so awesome?
Have you seen the overwhelming opinion about the current trend of games? Many people aren't happy and people reply with "This site gave it an 8.[x], it's probably just that you're wrong.".
Use your scoring system because as to the weight of categories because yes, "Sound" isn't that important when compared to "Community".
Maybe me not paying $15 to Blizzard, Trion, or Bioware won't make a difference but maybe when they see their product roll out with a 5.8 (exxagerated), they'll get the point. I might not have any influence, but you do.
Spec'ing properly is a gateway drug. 12 Million People have been meter spammed in heroics.
I believe that the reviewers of this site go too easy on the features of a game that should be downright unacceptable in the MMORPG industry and they cushion the impact of those features to save face with developers so they can be invited to private play sessions and interviews.
Although, if they constantly played the role of "mmorpg watchdog", they would never be invited to anything and would quickly fall out of business. In a paradox, their need to survive directly conflicts with their their desire to be unbiased. They can't help it, that is just the nature of not only the industry but also the entire world's capitalist society. It doesn't pay the bills to be honest and although they try, they cannot ever be 100% brutally honest about anything or they will never make any friends. I believe they try to stay as unbiased as possible but are restricted by too many factors.
I blame society, not the MMORPG.com staff.
In fact, I'll go ahead and say that it's simply impossible to make a suitable review without being biased.
This.
It boggles my mind how people think a review can be churned out of an algorithm. A (re)VIEW is just some's perspective. Reviews can be formatted and reformatted but they are still just opinions. Having said that, I'm liking some ideas like the multiple reviewers and standardizing which elements MUST be in the review (but certainly others should get to be in if the author chooses). I read some of the authors for their writing style and the others I just skip. Reviews are ultimately like 5% of my decision making process. Gameplay videos and free trials make up the rest for me.
There's so much emotion hiding behind some of these responses it just blows me away.
These sites are supposed to be as unbiased as can be. Sometimes you can recognize your own bias and sometimes you cannot.
Nearlly every game gets a good review. I'm not hating on the review sites but it seems to me and many players that they are bound and determined to give a game a positive review if at all possible. Perhaps, for instance, they might be a little easier on the smaller companies, than say DCUO, but is that fair? Is it fair for a reviewer to ignore certain aspects? Some of these games, are not worthy of 7.5s.
Spec'ing properly is a gateway drug. 12 Million People have been meter spammed in heroics.
I mean, does an indie game get leeway since it had so much less money? How much leeway? Should game that spent 50 million get more leeway then one that spent 100 million?
I think basic things should be rated in some way. Like UI, art, graphics.
But when it comes down to the gameplay and functions, you should just point out facts. Does it work well. Does combat feel clunky. Are animations choppy. Just give information. If you wish to give your opinion, save it for the end.
One reveiwer can talk about the game, and you could have, let's say five, that "rate" it. But this is not an A, B, C, D or 1-10 system. But more of a system of how much you want to keep playing this game and would recommend it to others.
So the left side of the bar is red, leaning more towards saying , no rush to try it out. 1/4 maybe means, give it some time. The middle point could be mean the game will be fun for some, but might not be for everyone. As far to the right as you could go means, even if you don't enjoy "sandbox" or "F2P", you should try the game out.
Of course, these 5 reviewers are the same five every review who rate each game. None of them HAVE to be the one to write the main review. You could also have a link before their rating to show what sorts of games/mmos they enjoy. After their "bar" rating, they give details, ie pros-cons.
Hell, you could have five forum posters who would work on this with you to also post their feelings. Five staff reviewers and five posters (who meet certain standards set by you guys, but these people could change for each game, who also mention the games/mmos they enjoy/dislike).
What it really comes down to, is was the game fun enough to keep playing. It just becomes subjective sooner or later. But laying out as much information with as little opinion as possible for the REVIEW is the best way to go. Save the opinions for the end, and show what the rater likes, in regards to gaming. If you hated GW1, and rate GW2, you should mention you did not like the first (as that COULD affect the score). I do think a review a few weeks later as people progress through the game more, a great idea. They sort of "update" their pre/review.
I will admit, some games do seem to be rated rather high compared to how they are doing in the industry and how much fun they seem to be to most people. Games should not be rated JUST for the niche audience. You are more then welcome to mention "It's great if you enjoy FFA PvP with looting", but shouldn't give it a 7.5 just for that, let's say.
I liked the original system better. But I think there should also be a less structured opinion section where the "feel" of the writer should be talked about. Sometimes that doesn't equate to a number. Also, whenever possible, more than one writer should weigh in on the review. Have two or three other writers throw in a paragraph or two. I think the reviews I have seen like that are more effective and even if I don't agree with the primary reviewer's opinion (*cough* DCUO) I can accept that his/her peers have similar opinions or even differing.
As long as they are asking for money, they should be reviewed the minute they get it, they being the publishers, developers, whomever. There is no such thing as too early once the door is open.
parrotpholk-Because we all know the miracle patch fairy shows up the night before release and sprinkles magic dust on the server to make it allllll better.
If you're using a scale, use the whole scale. And if you need to make exceptions for games that won't even run properly or function as intended...don't try to apply it to the scale. I am rather tired of reviewers in general bending over backwards to make games work or giving special exceptions to the game and they still end up with decent scores.
For instance, let's just say for instance that an MMO came out tomorrow and it crashed so often that it became a nuisance to even get it to run. This would alter any paying customers perception of the game enough so that even if the game was fun as could be, they will tire of the constant issues. So, it's unfair to judge this game on every other aspect of it, when "Stability" is an obvious hinderance to everything to do with the game. Most places will break down the scores into categories, but if one is so bad that it overpowers the others entirely...the game would still score decently if not better than most due to other portions being strong.
So......how can you score a game and be fair? I don't think you can make everyone happy, but I think giving a game with obvious game breaking bugs and stability issues anything over a 5 is a disservice to gaming in general. It just isn't acceptable to release a barely working game and collect money from people who were "deceived" by non-biased scoring systems.
A couple ideas for scoring then:
If you do "grade" scores. When you give a game a C it should be 90-95% stable with average gameplay. Where the basics of the game is there and it could be fun for some, but overall it's nothing special.
A and B scores should differentiate between interesting mechanics, new things you think really bring something to the genre. Exceptional animations, sounds, graphics.....stuff like that.
D and F scores. In my opinion an F should be given to unstable games, or games where DRM (never seems to apply to MMOs but who knows where we'll end up eventually) interfere with playing them at all. D should be for games with medium to severe stability issues. IE, if zones go down every couple hours...that's a medium. If they go down every 30 minutes, that's severe. If they crash once a day and come back up in 15 minutes and stick around all day, that's a light stability issue. If the game is just no good, stable or not..then we get back into these letter grades, but these games seem to be fairly few and far between. So..I think the majority of your games are only going to end up in F and D if they are unstable.
Most F2P mmos would be C and maybe B...I haven't seen any I would give an A to. They especially shouldn't be given high scores when they are still in beta stages, because they seem to get more draconian when they go "retail" and more stuff moves into cash shop. Which if you can't enjoy a Free 2 Play game without paying money, I question it's categorization as "F2P".
I also think beyond what normal games are judged on...MMOs have more factors that impact enjoyment.
Gold Farmers, Gold Sellers, Spammers, Griefing, Cheating like Aimbotting on FPSMMO (For instance APB atm), Botting (WoW bots, it's no fun to play with computer controlled "Player characters".
If the above list is noticeable during a review, it's probably 20x worse for players. If they don't improve on the problems during your review....you should knock their score. Because stuff like that absolutely ruins MMOs overtime.
And, I really wouldn't mind seeing biased reviews. If someone is biased against a MMO genre....and they end up liking it after the review. I would be more impressed by that than if someone who likes the genre gives it a positive review.
In general, just don't be afraid to tank a game on the score board if it has obvious failing points during the review. Spell out the failing points, make the score reflect how bad they bugged you. And hopefully it will kick the devs in the pants to work on the problems, because overall there is almost no outlet for the normal player to complain about issues in games. Everyone is too busy waiting for devs to fix problems......it seems they are willing to overlook everything...and very little ends up getting addressed.
Comments
Not even half of the games on the website's list has as review. So, well. MAKE MORE REVIEWS!
The primary thing is this for me:
You preview just about every single game, you interview developers, publishers, marketers, sound engineers, etc etc etc. Take all of that information they gave you. Then take all the information from the back of the box. List it all. Install the game and begin playing. Did they bull**** you or were they honest? Is everything that they, through you, said would be there, there and fully functional? Is the combat as revolutionary as they claimed? Are the graphics really as awe inspring as the screenshots you have been pushing like crack in actual play, or do you have to have a 50 grand super computer to run it? Are the different systems everything that they portrayed them to be?
If the answer is unequivocally yes then tell us. But, when it isn't you have to tell us. You also cannot possibly expect us to accept some nonsense like, well while this feature hasn't made it in, I talked to so and so developer and they assured me that it would be sometime in the next few weeks but wouldn't tell me exactly when. Look at it like you are a teacher. Little devvie has a project due on Tuesday. He hands in his project but tells you that he was not able to do the whole thing and asks for an extension. Well, in a school where acheivement and education is important little devvie gets an F and devvie's mommy and daddy get to come down and talk to the teacher and the principal. If a developer promises the sun, the moon, and all the stars but only delivers on the sun they should be held accountable.
Keep all the personal opinions seperate from the important stuff, because opinions are just that. Opinions. Some more educated than others, but it doesn't matter. There are some 7 billion people on the planet and every last one of us is a wholly unique individual and every last one of us have our own opinions lol. In the response to me Stradden uses the term analyst. Analysis and opinion are separate. Analysis uses measurable criteria. Opinion does not an analyst make.
http://www.speedtest.net/result/7300033012
I can't speak for anyone else but I never pay attention to the score anyway. the way I research my games (in general, not just MMO's) is to read multiple reviews of the game, take all the points that are common to all the reviews, then decide if those things are positive or negative IN MY OPINION.
there are a lot of reasons for a low score but not all of those reasons are necessarily a deterrent. I've played games in the past that were universally panned in reviews but I enjoyed all the same.
with MMO's its even easier cause one of the main put offs are broken games but with MMo's that cam be just a temporary condition. If I really like the concept behind a broken game, I'll just wait till they fix it then I'll pick it up. I'm in a minority like that tho, once you've given a poor score to a game its over. no matter how good a game may become the average player has already put it behind them.
so my suggestion is to not make a scoring system at all, just state the pros & cons then let the readers decide what it means to them
Good luck with that lol. Remember you can't please all of the people all of the time. Heck its hard enough trying to please some of them some of the time.
Personally i read the review mainly to see if the game is buggy and how polished it is. I ignore scores because everyone's tastes are different and i like to make my own mind up.
I think categories could be useful as long as they are relatve to the genre.
Content (is there more stuff to do than is required to level up or does it run out and i need to grind some activity to progress)
Replayability (will i want to go back and start again with a different toon, is there more than 1 starting area)
Polish (how many bugs? are they game breaking or just annoying? how does the game run on minimum specs...do you guys even run mmo's on different computers to see how it goes?)
Socialization (does the game promote soloing over grouping while leveling or are they both supported, is the world open or instanced? are there good guild tools like guild banks etc, is there a viably economy, decent non-combat activities such as crafting, how is the chat system, is it easy to use)
I'm
Sorry. ^^^^ Great sig dude.
This is what I would like.
Review system:
Have "game impressions" for the initial review and follow them up soon after with "longevity reviews" with reviewers going to max level and then some. Maybe find a way to combine them in one final review article when they are both in. (Make visible that the final score is still in the works and that important information is still lacking at this time).
Scoring:
Only use the mmorpg.com average member score as a numbered score: it is most often a very fair score of a released game after some time. And if the reviewer's own score is way off from the average one, it always looks kind of funny. (Also this empowers us to a limited degree and makes us feel more involved
But do describe key-factors with terms like great, good, average, bad, terrible. Etc.
Key factors to take into account:
- Graphics, sound and animations
- Polish and stability
- Gameplay and game mechanics (perhaps specify for group, solo, and combat gameplay)
- PVE content
- PvP content
- Immersion value / RP content
- Longevity
Alternatively, if you REALLY NEED to give your own rating, make a nice formula and give an average score of the above key factors. And while waiting on a longevity review you could add 'preliminary score' to your initial 'game impression' score.
My brand new bloggity blog.
Actually I think breaking down a game into sections of say Aesthetics (yep ok I do check out the escapist website and enjoy Extra Credits), sound, gamplay and such would be a good idea and for sure should be given equal weight. People like different things and as such some prefer aestheticly pleasing over gameplay and some prefer the other way round and then look to say sound to be the tiebreaker. Not sure if there needs to be a average of the grades or not at the end maybe a table at the end with the catagories and grades. Oh and potential needs a grade of it's own with qualifiers from the reviewer on weather this is potential derived from if they fix this stuff it would work better or potential derived from if they can add this feature it would work better.
As to doing the beta/release day preview followed up by a review that seems feasable and makes alot of sence. The preview should probably focus on is this worth the box price rather than long term things. Stuff like how easy was it to install, how long did it take to patch, was it easy to get into the game, did the previewer have any troubles getting stuff to work and if they did how did customer service react. Maybe a quick rundown on the systems and if they are working as intended or need fixing. Basicly the goal should be to inform the reader if they are going to at least get a months worth of entertainment out of the product because thats all they have invested at this point. If possible maybe highlight any improvements or breakdowns from the switch from Beta to Launch. Another thing could be how long did the reviewer get to play in this limited time. Was it just a few hours? What kind of stuff did the preveiwer do in that time?
The review I think should focus more on long term stuff and could probably wait till close to the end of the first month. Things to include would be how did the game improve from launch to the end of the first month or was it a case of 1 step forward 2 steps back? Were any new systems/content added in this first month or was it all spent on bug fixing? Did anything start to grate on the reviewers ie annoying sounds, disconnects with skills and animations those kind of things that you can usually ignore or miss initially but build up over time to become immersion breaking.
I also agree with the idea put forward by several posters that more than one person should give an opinion incorperating both veiws into 1 article probably the way to go though that would deffinatly make the editors job harder.
The lesser of two evils is still evil.
There is nothing more dangerous than a true believer.
Here's the issue. A lot of those games will advertise the epic features that are often found only in end-game when you've reached Lv.50-70 or whatever the level is or at least spent a considerable amount of hours into the game. It's very hard to review an MMORPG in such case because it can takes weeks to reach that level or threshold and as mentionned by Jon, they need to post reviews as close to launch as possible to inform the readers on the product. So in that sense, they can only cover the features they have access to within the few hours they play. For exemple I believe EuroGamer reviewers are tasked to play the game for a total of 20 hours before making a review... even if that's not always the case *points at Ed Zitron*. But nonetheless, it's very hard to critique everything that's been said when so little time is available.
Hi everyone,
Just wanted to step in here and remind everyone to please stay on topic. Off-topic posts will be removed.
This is your chance to help improve the review process here at MMORPG.com with your feedback, but we ask you to please be constructive when giving it.
Non-constructive posts (such as basing the premise of your feedback on the fact we need to sort out our journalistic integrity) will not be tolerated.
There are many excellent examples of good feedback in this thread and we'd like to thank you all for the feedback coming in so far.
I feel no apologies are warranted. The poster outright accused you and other journalists of having their opinions bought, and then went on to explain his dream fantasy land where gaming journalists are treated like kings and showered with phat loot on the scale of Will Smiths ultra mega trailor. So you stomped on him hard with both feet. Great job in my book. The world of video game creation is not a glamorous one, and neither is the world of the video game critic.
That said back on track with what the article is about. I've always found the 1-10 scale to be laughably unreliable. When people like something they tend to treat it quite favorably, hence the astonishing number of 8 -10 ratings. When people do not like something they are less inclined to treat it favorably hence the disporpotionate number or 1-2 ratings and an quite a deficite in the 4-7 area.
So here is my idea. STOP RATING THESE GAMES AT ALL. At least numerically.
Instead just tell us what you liked and what you didn't like. And make sure your carefully explain why or why not. Telling me that the game UI feels clunky and difficult will help me out a hell of a lot more than you just saying on "I give it 7 out of 10".
As I explained earlier the 1 - 10 scale is badly skewed twards strong feelings one way or the other, so kick it aside. I recommend you simplify the whole thing by just making it a yes/no or hot/cold vote, because frankly you either liked it or didn't like it. Trying to split hairs by saying I liked it "this" much, is just self denial. End of problem.
Oh and before someone brings up the spector of "what about undecided", if you are a critic getting a hands on look at something and cannot make a decison as to whether you like it or not, you are in the wrong business. (I do suppose that an undecided category would work for an unreleased game, but honeslty I feel it is still a copout.)
Of course the new system won't stop we forum gladiators from yelling our heads off if we disagree with your yes/no vote, but you know what? I doubt even a prolonged global thermo nuclear war would be able to do that.
Oh and another idea would be for you to have more than one set of eyes on a product. Having more than one person expressing their thoughts in a review gives us a better idea of how they were impacted by the whole event. Of course in the case of previews such a thing may not be possible as it is hard enough to get one journalist into these little meetings. And as we both know Will Smith mega ultra trailors don't grow on trees.
"Gypsies, tramps, and thieves, we were called by the Admin of the site . . . "
I recommend that if you continue to use a rubric of any sort, please create a ridiculously detailed one for yourselves (not that we see) and have a series of workshops/classes for all of your reviewers to both teach the use of the rubric and get on the same wavelength. I say this because historically on this site, a 7 will signify a total mess from one guy, and the best thing on the market from another guy. You have reviewers that give pity points, and reviewers that don't. The numbers have become almost meaningless.
Not only that, but everything comes out as a B- to B+. If everyone's going to 'pass' anyway, and all the marks fall within a target area, what's the point in continuing to generate marks?
I actually suggest that you leave behind the current system. Discard the rubric, and discard the single reviewer. Have several of your employees do simultaneous reviews highlighting the merits and flaws that they perceived during a play period (making certain that they do not play with or talk to each other about their reviews until after they've been submitted) so that we get a more well-rounded impression. Let's hear from the bitches AND the pushovers at the same time, instead of one or the other as we have been.
Alternatively, have one employee round up a bunch of non-employee volunteers on the site to do the same thing: play the game for a period and report their fun and frustrations. Then the employee compiles a review from the myriad sources, citing them, and giving a more community-generated impression. A review from one point of view about an MMO really seems to miss the point anyway. I want to hear from the guy that loved it, the guy that sees potential, the guy that broke his keyboard from how much he hated it, and the guy that went 'meh' and couldn't bring himself to do much with it. Multi-player review for a multi-player game. As it is right now, I think most of us are rejecting the reviews you post, and reading the litany of comments afterwards to try and get a more realistic impression than the review managed to give in the first place.
Basically, fix your rubric system and get everyone implementing it onto the same wavelength, or ditch the rubric and use the conflicting wavelengths to create a better review. I vote the second option, but that one takes more balls on your part. *^_^*
Multiple reviewers is a great idea. Two people discussing the game can present more evolved ideas and discussion points than just one reveiwer, and differing opinions bring perspective to the various facets of the games.
Come on. Siskell and Ebert, guys. It's a no brainer! :-)
I agree, sounds like a plan, if you can find more reviewers willing to spend time on a new MMORPG (which for the AAA titles shouldn't be hard to achieve).
And make it 2 people with a very different opinion and taste in MMO games, that'd make it more interesting to read.
Also, if you really want to get rid of the rating system and get radical, do it like Rottentomatoes.com, like Tardcore said, a review with a simple 'yes' or 'no' as end conclusion.
I'd expand on it though and make it 'yes', 'no' or 'maybe'.
And do it for different types of gamers, like 'sandbox gamer', 'themepark gamer' and the Bartle archetypes like explorer, achiever or socializer, so a game could be a 'yes' for a themepark gamer but a no for a sandbox gamer and a maybe for an explorer.
Just throwing out some ideas
The ACTUAL size of MMORPG worlds: a comparison list between MMO's
The ease with which predictions are made on these forums:
Fratman: "I'm saying Spring 2012 at the earliest [for TOR release]. Anyone still clinging to 2011 is deluding themself at this point."
Here's my 2 cents.
I literally read every post/comment to this thread and while some great ideas were posted I still don't think the real problem is being addressed. As a personal rule, I don't like or trust video game reviews (MMOs or otherwise) done by the gaming media as it stands in this day and age. For one thing, they're not "reviews". They're exaggerated opinions labeled as a "review". In order for journalistic integrity to be upheld, a real REVIEW needs to be objective. Yes REVIEWS can be objective. Just because 90% of the gaming media aren't objective doesn't mean that they shouldn't be. The gaming media likes to use the excuse that because they themselves are fans of the genre most if not all professional journalistic integrity has to be thrown out of the window and so then it's impossible to deliver an objective REVIEW. To that notion I say Bull%$^&! Until that attitude and notion is addressed and changed, you're going to always have the disconnect that exists between your so called "reviews" and whatever rating scale is being used. It doesn't matter what kind of scale you use (this very site is attempting to change their scale once again). Until you all are able to put the "gamer" part of yourself aside long enough for a "review" to stay journalistically objective, nothing's ever going to change and every three or so years you'll attempt to do again what you're doing now. You ever notice that the preview to any given game is overtly positive and the person previewing the game swears up and down that it's the best thing since sliced bread and then when the review hits (unless its a tripple A title and even those suffer), its all "bring out your pitchforks", and "crucify him". You guys (the gaming media) are flattering yourselves calling this crap that you write a "review". They're mostly filled with "what I didn't like", "what I wished they would have done" and I'm sorry but thats monday morning water cooler talk. IE - An opinion. I have absolutely nothing against personal opinions. I love and welcome them. My problem is reading up to sometimes 3 web pages worth of nothing but personal opinions and it being called a "review". The other part of that problem is when people who can't think for themselves read these negative opinions or better yet, just look at the box score (because seldom do the "review" and the box score match up) and say to themselves - "man that game must really suck" and in turn don't buy it . I've lost count of how many games got a safe score/review or negative score/review that I absolutely loved. As it stands now, I'd trust another developer "reviewing" a game in a heartbeat as opposed to the gaming media because at least with another developer, they understand what goes into crafting a game and why they may have chose to do "A" instead of "B". It's just not coming from a pure place where the gaming media is concerned. As to MMORPG reviews, I like the 3 prong approach that some have suggested here. It makes the most sense in terms of the way an MMO is played.
I believe that the reviewers of this site go too easy on the features of a game that should be downright unacceptable in the MMORPG industry and they cushion the impact of those features to save face with developers so they can be invited to private play sessions and interviews.
Although, if they constantly played the role of "mmorpg watchdog", they would never be invited to anything and would quickly fall out of business. In a paradox, their need to survive directly conflicts with their their desire to be unbiased. They can't help it, that is just the nature of not only the industry but also the entire world's capitalist society. It doesn't pay the bills to be honest and although they try, they cannot ever be 100% brutally honest about anything or they will never make any friends. I believe they try to stay as unbiased as possible but are restricted by too many factors.
I blame society, not the MMORPG.com staff.
At least they didn't sell out like MMOHUT and call Allods "The biggest MMO of the year".
I always kinda liked the multi reviewer style. Have three reveiwers, (with their backgrounds and play preferences listed) and then each have a short blurb dissecting their likes and dislikes of the game. Sometimes you get some really great opposing points about the game, and the reader can value these points based on which playstyle they identify with.
As for the grading scale, it does help to have that sense of standardization there, but I think few people actually buy games based on ratings score. It might be a better indicator of interest in the game and maybe corollate it with the number of page hits or article hits for that game, but overall I'm more interested in the reviewers thoughts than than the numbers.
Thanks for posting an article like this BTW, always like to see it when sites are trying to improve their quality through consumer participation.
But it's quite hard to do a review without being biased, otherwise you just end up with a list of features accompanied with a description and you can pretty much find the same thing on said MMO's website. It's quite normal for a reviewer to view a feature as perfectly acceptable even if someone else disagrees. Just like you can't make the most "carebear" reviewer go review the most hardcore FFA PvP MMORPG with full loot and Perma Death, or the most hardcore reviewer to review Hello Kitty Online. So I really have to disagree with you that MMORPG.com is simply trying to avoid being too much of a "mmorpg watchdog", and rather that reviewers simply have different opinions on common features.
In fact, I'll go ahead and say that it's simply impossible to make a suitable review without being biased.
I haven't read a game review since YouTube was invented. I watch gameplay videos on YouTube and decide if I'll try the game or not. No review I've ever read has given me more insight about a game than watching a 5 minute gameplay video has given me.
We live in the Web 2.0 era, you're behind the times already only having written reviews. If you really want to distingush yourself from the rest of the gaming sites on the web than video reviews is the way to go. Sure, some sites are already doing this, like Game Trailers, but they aren't MMO specific, and honestly, the reviews I've seen on there aren't great.
As far as the actual grading/reviewing process I would recommend a combination of a lot of what I've seen here.
First thing to do would be to get rid of any grading scale at all. You can still have ratings (because the majority of people need that), but make it very simple. I would suggest a "Buy", "Try" or "Skip" rating scale. Self explanitory, no room for confusion.
Second the review needs to be 3 parts, as many have suggested. Start with a first impressions on day of release, or a week or so later.
Two weeks later, more indepth information on how the game plays, technical information, etc.
Then after 30 days give the final review with the Buy, Try, or Skip rating attached at the end. This would be a review of parts 1 and 2 + additional information like progression info, griding scale, etc.
And of course, ALL IN VIDEO FORMAT!
Edit: The reason I suggest a 3 part review is because if you do one on day 1, and then one on day 30, people will have lost interest. Having a shorter time span between each part keeps people's attention.
I would actually watch the reviews on here if they were done like this!
"There is as yet insufficient data for a meaningful answer."
Is it time, perhaps, to ditch the curve?
http://www.mmorpg.com/faq.cfm/showFaq/15/The-scores-on-the-game-list-all-just-drastically-changed-what-is-happening.html
Why should scores be inflated and merely not graded on their own merits? Why does CrappyCo's game's score get inflated just because Stellar Inc made something so awesome?
Have you seen the overwhelming opinion about the current trend of games? Many people aren't happy and people reply with "This site gave it an 8.[x], it's probably just that you're wrong.".
Use your scoring system because as to the weight of categories because yes, "Sound" isn't that important when compared to "Community".
Maybe me not paying $15 to Blizzard, Trion, or Bioware won't make a difference but maybe when they see their product roll out with a 5.8 (exxagerated), they'll get the point. I might not have any influence, but you do.
Spec'ing properly is a gateway drug.
12 Million People have been meter spammed in heroics.
This.
It boggles my mind how people think a review can be churned out of an algorithm. A (re)VIEW is just some's perspective. Reviews can be formatted and reformatted but they are still just opinions. Having said that, I'm liking some ideas like the multiple reviewers and standardizing which elements MUST be in the review (but certainly others should get to be in if the author chooses). I read some of the authors for their writing style and the others I just skip. Reviews are ultimately like 5% of my decision making process. Gameplay videos and free trials make up the rest for me.
There's so much emotion hiding behind some of these responses it just blows me away.
There are two problems though
These sites are supposed to be as unbiased as can be. Sometimes you can recognize your own bias and sometimes you cannot.
Nearlly every game gets a good review. I'm not hating on the review sites but it seems to me and many players that they are bound and determined to give a game a positive review if at all possible. Perhaps, for instance, they might be a little easier on the smaller companies, than say DCUO, but is that fair? Is it fair for a reviewer to ignore certain aspects? Some of these games, are not worthy of 7.5s.
Spec'ing properly is a gateway drug.
12 Million People have been meter spammed in heroics.
It's hard to really balance these things.
I mean, does an indie game get leeway since it had so much less money? How much leeway? Should game that spent 50 million get more leeway then one that spent 100 million?
I think basic things should be rated in some way. Like UI, art, graphics.
But when it comes down to the gameplay and functions, you should just point out facts. Does it work well. Does combat feel clunky. Are animations choppy. Just give information. If you wish to give your opinion, save it for the end.
One reveiwer can talk about the game, and you could have, let's say five, that "rate" it. But this is not an A, B, C, D or 1-10 system. But more of a system of how much you want to keep playing this game and would recommend it to others.
So the left side of the bar is red, leaning more towards saying , no rush to try it out. 1/4 maybe means, give it some time. The middle point could be mean the game will be fun for some, but might not be for everyone. As far to the right as you could go means, even if you don't enjoy "sandbox" or "F2P", you should try the game out.
Of course, these 5 reviewers are the same five every review who rate each game. None of them HAVE to be the one to write the main review. You could also have a link before their rating to show what sorts of games/mmos they enjoy. After their "bar" rating, they give details, ie pros-cons.
Hell, you could have five forum posters who would work on this with you to also post their feelings. Five staff reviewers and five posters (who meet certain standards set by you guys, but these people could change for each game, who also mention the games/mmos they enjoy/dislike).
What it really comes down to, is was the game fun enough to keep playing. It just becomes subjective sooner or later. But laying out as much information with as little opinion as possible for the REVIEW is the best way to go. Save the opinions for the end, and show what the rater likes, in regards to gaming. If you hated GW1, and rate GW2, you should mention you did not like the first (as that COULD affect the score). I do think a review a few weeks later as people progress through the game more, a great idea. They sort of "update" their pre/review.
I will admit, some games do seem to be rated rather high compared to how they are doing in the industry and how much fun they seem to be to most people. Games should not be rated JUST for the niche audience. You are more then welcome to mention "It's great if you enjoy FFA PvP with looting", but shouldn't give it a 7.5 just for that, let's say.
I liked the original system better. But I think there should also be a less structured opinion section where the "feel" of the writer should be talked about. Sometimes that doesn't equate to a number. Also, whenever possible, more than one writer should weigh in on the review. Have two or three other writers throw in a paragraph or two. I think the reviews I have seen like that are more effective and even if I don't agree with the primary reviewer's opinion (*cough* DCUO) I can accept that his/her peers have similar opinions or even differing.
As long as they are asking for money, they should be reviewed the minute they get it, they being the publishers, developers, whomever. There is no such thing as too early once the door is open.
parrotpholk-Because we all know the miracle patch fairy shows up the night before release and sprinkles magic dust on the server to make it allllll better.
If you're using a scale, use the whole scale. And if you need to make exceptions for games that won't even run properly or function as intended...don't try to apply it to the scale. I am rather tired of reviewers in general bending over backwards to make games work or giving special exceptions to the game and they still end up with decent scores.
For instance, let's just say for instance that an MMO came out tomorrow and it crashed so often that it became a nuisance to even get it to run. This would alter any paying customers perception of the game enough so that even if the game was fun as could be, they will tire of the constant issues. So, it's unfair to judge this game on every other aspect of it, when "Stability" is an obvious hinderance to everything to do with the game. Most places will break down the scores into categories, but if one is so bad that it overpowers the others entirely...the game would still score decently if not better than most due to other portions being strong.
So......how can you score a game and be fair? I don't think you can make everyone happy, but I think giving a game with obvious game breaking bugs and stability issues anything over a 5 is a disservice to gaming in general. It just isn't acceptable to release a barely working game and collect money from people who were "deceived" by non-biased scoring systems.
A couple ideas for scoring then:
If you do "grade" scores. When you give a game a C it should be 90-95% stable with average gameplay. Where the basics of the game is there and it could be fun for some, but overall it's nothing special.
A and B scores should differentiate between interesting mechanics, new things you think really bring something to the genre. Exceptional animations, sounds, graphics.....stuff like that.
D and F scores. In my opinion an F should be given to unstable games, or games where DRM (never seems to apply to MMOs but who knows where we'll end up eventually) interfere with playing them at all. D should be for games with medium to severe stability issues. IE, if zones go down every couple hours...that's a medium. If they go down every 30 minutes, that's severe. If they crash once a day and come back up in 15 minutes and stick around all day, that's a light stability issue. If the game is just no good, stable or not..then we get back into these letter grades, but these games seem to be fairly few and far between. So..I think the majority of your games are only going to end up in F and D if they are unstable.
Most F2P mmos would be C and maybe B...I haven't seen any I would give an A to. They especially shouldn't be given high scores when they are still in beta stages, because they seem to get more draconian when they go "retail" and more stuff moves into cash shop. Which if you can't enjoy a Free 2 Play game without paying money, I question it's categorization as "F2P".
I also think beyond what normal games are judged on...MMOs have more factors that impact enjoyment.
Gold Farmers, Gold Sellers, Spammers, Griefing, Cheating like Aimbotting on FPSMMO (For instance APB atm), Botting (WoW bots, it's no fun to play with computer controlled "Player characters".
If the above list is noticeable during a review, it's probably 20x worse for players. If they don't improve on the problems during your review....you should knock their score. Because stuff like that absolutely ruins MMOs overtime.
And, I really wouldn't mind seeing biased reviews. If someone is biased against a MMO genre....and they end up liking it after the review. I would be more impressed by that than if someone who likes the genre gives it a positive review.
In general, just don't be afraid to tank a game on the score board if it has obvious failing points during the review. Spell out the failing points, make the score reflect how bad they bugged you. And hopefully it will kick the devs in the pants to work on the problems, because overall there is almost no outlet for the normal player to complain about issues in games. Everyone is too busy waiting for devs to fix problems......it seems they are willing to overlook everything...and very little ends up getting addressed.