Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
General: Grinds My Gears: Critique Our Reviews
SBFord
Former Associate EditorMember LegendaryPosts: 33,129
We here at MMORPG.com are always looking for ways to improve our service to you, our readers. One of the things that constantly crosses our minds is the way in which reviews are "done" here at the site. With that in mind, MMORPG.com Managing Editor Jon Wood lays out some of his concerns with the review process and asks you for your thoughts. Be sure to weigh in!
For a long time, I’ve been dissatisfied with the ways that MMORPGs are reviewed. Most of the time, they’re reviewed way too early, without much or any consideration given to the long term nature of the genre, journalists are playing the beta and calling it a review, the current scoring system is broken and more.
Read more of Jon Wood's Grinds My Gears: Critique our Reviews.
)
Comments
I like a combination of your ideas. Use the letter system, as its easier to understand what constitutes an A over a B and then use the rating of different areas of a game and average those findings for a final letter grade. Sort of like giving each game a GPA score.
You are awesome Jon, good read.
In my opinion, the only true way to review a game should be by entirely removing the score and instead letting the reader decide. I mean quite literally you can sit there reading an article and get a very good impression of a game, just to be brought to the end where the actual score is only a 6. And you think to yourself, some of these features sound pretty cool, and he did say the gameplay was enjoyable but...
The issue there though, in this age of gaming, its unlikely people will read an article to get to the score. I find that it usually happens backwards in most cases. They look at the score, make the snap judgement on what to expect, and move on.
For an MMORPG I feel reviews need more time, but as you stated, thats not possible, as your public demands satisfaction! In that sense, I think there needs to be a 3 tier scoring system that spans the course of an entire month. The first review will be of the preview / launch. The second review will be about mid-game.. how population fares and so on. The last review will be at the end of the 30 days. Do you have a reason to resub? Do you see yourself still enamored after 2 or 3 months? Have you noticed population drops, or reduced community involvement in some areas?
Once all 3 scores have been generated, perhaps an average of that would be a good idea. That way, a good first day review could be weighed down by the end game review when you find out that theres only 1 raid that pops ever 4 hours, or that the NPC sounds are too repetitive and get annoying, etc.
I think most people feel like reviewers DO just spend too little time to a game, and really only get a small part of the picture, and its usually the picture based on a snap judgement. Whether thats true or not, it seems to be the average consensus.
Go back to your roots and rate the game on several categories ... its tells alot more than the current rating system and allows users to leaf scores and glean more information in their brief pass.
I would also do something about the rating system in place on the game list ... its to easily manipulated.
[mod edit- non constructive]
I would stage reviews in 3 stages. First is launch. Is everything that the developer promised there and fully functional? If not it is your responsibility to call them on it and not equivocate and forgive because maybe tomorrow, maybe 6 months from now it will be true.
You should also have at least 2 people, preferably 3 or 4, to do each review. Again, need to counter the fact that too many of you are way too cozy with publishers and have had your opinions bought with tchochkies and fluff. Seriously, is a freaking remade lunchpail with a vidiot game toon worth your integrity? Wait, don't answer that
Then those same people should come back and review it again at the 3 month mark, then at the year mark. Each time holding the developer accountable for what they said they would do and what they actually did. On more than one occasion have read a review here where your "writers" give positive marks for future promises. Hey I can promise you that in 6 weeks there will be no more hunger, poverty or war. Doesn't mean that it will happen lol.
http://www.speedtest.net/result/7300033012
I think too that reviews needs more time in game.. Maybe something like IGN is doing with MMOs with their "review in progress" where you get small articles about what the journalist has been testing lately. Maybe a week between each article could be nice.. We'd still get the impressions of the mmorpg staff and you could have more time to settle a more complete and concrete review of the game.
As for the rating system, I like the pros and cons currently in place. The score itself could be a bit more decrypted though, all we have so far is a number with general idea of what that score means with a "Mediocre" or "Great". The 2 last reviews in are DAoC which got a 7.5 and an "Average" quote.. I don't think a 7.5 is average, think it's pretty good.. Xsyon got a "Mediocre" with a 6.8, pretty harsh as well.. I think the comments should go more something like that :
1-3 : Garbage
4-5 : Mediocre
6 : Average
7 : Good
8 : Great
9 : Amazing
10 : Perfect (like if it was even possible from an MMO at release...)
I always wondered how much you take into consideration that the game that is being reviewed is an MMORPG for MMORPG.com for MMORPG players. Because you know a game may be 'fun' but not necessarily a good mmo. There was one game recently where the reviewer admitted that the community functions of the game were near broken and that the game had no long play value beyond a month, yet gave the game a good overall review. To me an mmo that will bore you in a month is a failure as an MMO even if fun to play.
I don't mind early reviews, as long as they are labelled as such, and we are given some idea of how far the reviewer has progressed in the game. With most games, early impressions are usually fairly accurate, particularly when games are released unfinished or buggy. An early review can at least guide people as to whether it is worth the box price and a months sub, even if there is little longevity beyond that.
Chins
I'm not really a fan of how IGN rates their MMOs. They draw it out throughout a 1 - 3 week period which is nice, and update every few days, but the problem arises when they only do PART of the review through each stage. Like, the entire first week is about gameplay, but then the 3rd article they start getting into sound and graphics. When you play a game it encompasses all of these things from the start, so piecing it together as you go along like that doesn't always work.
The ONE THING IGN does right, is that sometimes they have second opinions by other writers. Its one of the best things I think you can have. Someone other than the original writer weigh in on their take. MMORPG did that with some of their articles... its a good take on it, but a more detailed review accompanying their main review would be good as well.
I never really was a fan of the scoring system. Most MMOs, tend to get anywhere between 7.5 to 8.5, as if the reviewer (or the website) did not want to damage the reputation of "x" or "y" company who just bought ads on the site. This goes for a large majority of the websites. So it's very hard to make a clear decision as if wether or not the game is worth it, especially when the score is litteraly inconsistent with the review itself.
Personally, I'd do away with the whole scoring system, and let the readers judge by themselves based on the review. Perhaps an additional option for readers to say if this review was helpful or not (with a comment, basically like the hype meter) would be a nice addition.
As for how to review MMOs, I like the "previews" on Launch Date and the review in the following weeks. It's alright. But there's been a rather significant rise of "video reviews" lately where you can follow the reviewer from the character creation to "x" and "y" level while he discusses and reviews the game verbally. While it doesn't have to be for all reviews, I think it would be a great addition every now and then. It gives a more "hands-on" perception to the readers/viewers. The video could also be accompanied by additional written review by the reviewer if he wants to (sometimes to cover issues that have not appeared in the video or not enough for exemple).
I don' t believe in fact there's nearly as much "payola" going on to influence review scores on MMORPG.com
What I believe is a much bigger factor is rooted in the background of the reviewer.
I know most of your reviewers come from a long history of MMO's, but the real question sometimes is what sort of playstyles do they prefer?
If one prefer's games like EVE, they probably hold a bit of a bias against a title like Rift. Same is true if they love WOW but can't stand Darkfall's style of FFA full loot combat model.
Ideally you'd want an impartial reviewer, but I think that's impossible. Everyone has gaming preferences and I think it goes a long way to explaining sometimes the very high scores some games receive (and some of the low ones as well of course)
First, I'd go with an A, B C system, because I agree, having 10 levels, when in fact only the worst of titles ever dip below 5 makes no sense. As the real range is more like 5-10, just make it A-F and it will make things easier. (screw lining up with meta critic. (edit Also, I'd prefer you don't slip into the +, - syndrome, then you're right back to where you started at)
Next provide a good biograhy of every reviewer, include their previous gaming/review experience and most importantly, have them list their favorite and not so favorite gaming titles. That way we could see where their frame of reference is in writing a review.
Finally, as suggested by Severius and others, consider having a short "counter point" section to a review. If your primary reviewer does favor sandboxes while reviewing SWTOR, consider having a counter point by someone who really enjoyed a title like LotRO.
Also, I like the current system of a preview followed by a full review, at least a month or two down the line so the reviewer has some time to see how the long term playability /end game is shaping up.
"True friends stab you in the front." | Oscar Wilde
"I need to finish" - Christian Wolff: The Accountant
Just trying to live long enough to play a new, released MMORPG, playing New Worlds atm
Fools find no pleasure in understanding but delight in airing their own opinions. Pvbs 18:2, NIV
Don't just play games, inhabit virtual worlds™
"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon
First off, the problem with grading as we see it today is that games are meant to be fun. You can't quantify fun. What I think is fun, what you think is fun, what Bill thinks is fun, what anyone else here thinks is fun will never totally line up. Thus you get conflict, hate, vitriol, and the tin-foil-hat brigade marching on your doorstep.
Personally, I like the idea of breaking down the systems in the game and giving them each a letter grade F-A. Then take the whole of the grades and average them to get the final grade.
Obviously, you then have to decide what categories will get graded (another lose-lose situation for you guys). Some of the basics that I look at that cover all forms of MMOs are: graphics, sound, UI, stability and diversity (races, classes, ect). However, each one of these is subject to personal taste as well. I like the way WoW looks; some don't. I love a sweeping score; alot play their own music. Some people put a huge emphasis on longevity. They considers an MMO to be a lifestyle and the game should never end. I don't really care how many months I play a game. At some point in time, I'll get bored and move on just like any other game. However, value could and likely should be a category for reviewing.
In the end, reviews are opinion and can't be taken as fact as alot of people think they should be. There is no benchmark to grade against and because of that, you guys as proffesional reviewers are screwed. Just do your best and don't worry about us, we will go away at some point.
I want a mmorpg where people have gone through misery, have gone through school stuff and actually have had sex even. -sagil
Just a quick swag. With all the comments and hurt about integrity, relationships, etc. aside. What meaning is there to a score any more? really.
Is the score meaning include 'eveolutionary' changes to the fun,depth, immersion or interest potential factor of game-play?
I'm not goign to dwell on this, but a score should be reflective of the times, the products advances, evolution, the deliverables compared to its competition, compared to its' predecessors. A score of 7 or 8, for example to a game that was launched recently, shouldnt share the same score of a game that was launched 2-years ago, but is essentially similar.
That is unless the score is something reflective of being able to get out of the bed in the morning today, just as we were able to get out of the bed in the morning 2 years ago. In that case, the refelction on a current-day game versus a past game is nothing more than a score of 'viability'.
A breakdown by category would be my prefered answer, with 2 planned reviews.
A technical biased "early look"- heavily concerned with stability and performance matters. Really it's a what is in the game and does it work style of review. It would be nice if you could also do installs on high and low end rigs, playing around enough to judge the differences.
The second review- the resub review a month (roughly) later. This would be the more "traditional" review, grading how well things work together. By this time the community should have settled a bit (and its importance gauged), you are into the typically boring middle game, and you'd have had time to play around with the systems. This is also a good chance to "update" on any technical improvements.
Such a system would be a huge improvement. Placing the review focus on if a working product was delivered early on, then assessing its lasting power with the review a month in. It probably won't make many friends though. Devs will dislike the requirements on content/gameplay, since they won't be able to hide the reality of their game behind the initial "hook" hours.
I always laugh at the people who think that somehow ad revenue for a certain game will get them a better review on this site. Anyone remember when Earthrise came out? There was a giant Earthrise advertisement on the front page, and the news banner was the review basically calling it out for being the piece of shit it was. People seem to have selective memories when it comes to things like that. Attacking the reviewers and the staff is silly in my opinion.
I also agree with Jon in the fact that I don't really care for a numerical scoring system when reviewing games because MMO tastes are wildly different. Some games cater to casuals, other to PvPers, others to sandbox fans etc. PvP is a pro to me, but a con in someone else's book as an example. This is why I don't use numbers when I review games in my blog, but at the same time people who read my work understand the type of MMO player I am and can use that as a benchmark for my opinion.
With this in mind, there's a lot of ways I think the MMORPG.com reviews could be improved upon.
1) One possible idea would be to have a game get reviewed by more than one person to get multiple view points represeented by gamers of various types. A casual, a hardcore, a PvP person all weighing in would ensure the reader has 1 viewpoint he can related to. As someone mentioned before, it's silly for someone whose favorite MMO is LOTRO review a game like Mortal or Darkfall because they are so wildly different and catered towards completely different people. Of course the downside of this is that it would put a lot more pressure on an already busy staff to have multiple people doing the same job.
2) Another idea in term of the timeline for a review would be to do 3 reviews as an example which could be numerical, but space them out over a time period, then use the average score of the 3 reviews to determine the final word. Review the game on day 1, then on day 30, then on day 60 or 90 since this is usually the time frame where a lot of people may lose interest historically.
3) A final idea would be to have a section of player reviews right next to the staff review much like you see on other gaming sites like gamespot so people can not only compare scores but read the testimonials of people who are playing. Player reviews could then be given a thumbs up or thumbs down, or agree or disagree option where people can rate the player reviews.
4) A final option might be for video reviews much like you see on IGN or Gamespot where the actual staff member is talking to a video of him playing and pointing out the things he is talking about. This wouldn't make for a more accurate review, but videos really help give people an idea of what a game looks like when its running, or not running well because its a POS. This would of course require someone who is pretty good with video and editing.
5) Livestreaming. How about having a spot on te site for livestream channels so players can watch other players and staff playing the game with their commentary in real time. I livestream all the time for league of legends as an example, but livestreaming is growing more and more popular for all kinds of games, and it is not overly difficut to do if you have a haklfway decent system.
This is just brainstorming mind you, but maybe this will help the discussion.
TwitchTV Partnered Streamer
MMORPG.com Spotlight Blog Writer
Co-Leader of Inquisition
Youtube Channel
Whilst I am in no way advocating the ideas of the poster you are responding to or any claims of what are tantamount to 'bribes'. But do you really feel the underlined demonstrates the professionalism you are talking about in the centre of your post, the crux of your argument?
I'm all for defending yourself and your staff, but as someone looking at the discussion from a neutral perspective to me it hardly smacks of a professional rebuttal to a customer/service users complaints.
Back on topic -
For me there is little need for any kind of overall score/mark. At the end of the day having a score or grade is only worth while for point scoring against other games. Something which has little to no baring as to whether I personally will or will not try/stick with a game.
A detailed and thorough review should be made of all aspects of the game, taking into consideration how different types of mmo players will respond to them. A final summing up of the game with the reviews opinion of it is good enough for me. If you say it's great and I can see in your detailed analysis why you have come to that conclusion I don't need a number or letter score to help make up my mind.
One thing I would say though is the sites reviewers should keep in mind they are reviewing mmos (often with subscriptions) and not stand alone, offline games. Whilst that may seem blindingly obvious, I have noticed mmos with little content and little real potential (i.e. themeparks with no scope and not enough content) still getting good scores. An mmo needds to have content, or it needs to have masses of potential for player made content (sandbox), if this isn't the case it needs to be hammered home in your reviews.
"Come and have a look at what you could have won."
I'm going to be completely honest here and say that the problem isn't scoring or early reviews. Its honesty and lack of MMO experience. If a person has only actually played one or two MMO's for more than a beta or a trial, (or even just six months) then what business do they have in reviewing anything at all much less releasing that review on the internet for a major gaming website? They can't objectively review anything on the site other than the specs.
Just as an example, STO. Most of the people with more than a years experience in more than one MMO would call that game garbage, even now after its been out for years. Yet the reviewer on here was talking about how truly wonderful the game had become. I mean seriously... that review should never have been released.
Every game in the last 5 years has just been a rehash of other games that came before them with a few changes to make them seem novel, but then they're released and you see how bad they are.
The gaming companies still aren't listening to the players. And no... I don't mean the "nerf this" players. I mean listening to our ideas to actually fix the problems inherent in the genre.
I mean, for gods sakes, most of these company even leave certain bugs in the game for years after they're first reported by players, never mind actually listening to possible ways to actually improve the game systems.
Its time for honest and hard hitting reviews. Not ones that kiss the companies butt in hopes the game company will help pay for the sites existence.
We need honesty, not catering to the money.
Ratings are far too subjective for a simple 1-10 ... everyone wants different things out of their games. What I prefer are attempts to describe who the reviewer thinks the perfect customer is for the game, what sort of people I am likely to meet while playing there.
http://features.metacritic.com/features/2011/game-critic-scores-vs-user-reviews/
I posted the above link as I think it shows that this is a problem that goes beyond just MMORPG.com. Even metacritic realizes that there can be a large disconnect between gamers and the reviewers lest anyone think that this is a problem that only exists here.
Second, I wanted to focus on 1 section of your article here: "MMORPG reviews are beholden to two masters: First, whether or not the game is worth the initial box price and 30 days that come with it. Second, whether or not the game is worth the long term investment of an ongoing subscription fee." I don't mean to pick on this but I feel this is why reviews on this site and many others are not that good. In a journalistic setting you are only beholden to 1 master, and that's objectivity. If you lose that then integrity is soon to follow. I definitely agree with the others who've stated that there needs to be multiple reviewers for each game. What I would suggest is something like this:
Initial Review: No later than 1 week after the launch. This review is like the front page of the newspaper. No opinions about anything regarding the game. (Let the gamer decide if a feature sounds appealing etc.) As the old show Dragnet used to say, "just the facts ma'am." This review needs to list the features of the game and what it offers. It needs to be as objective as possible.
Next Review: Possibly 2 weeks or so after game launch, enough time to hopefully get past the first weeks server issues or whatever. At this point the reviews need to turn into the editorial section of your local paper. Plenty of opinions about the features offered by said game, fun factor and the like. However, I would suggest that you wait to release these until you have both a positive and a negative review in hand. (Lets be honest, no game does everything perfect) That way a reader can look at both side by side rather than getting the good first and then a few weeks later getting the bad.
Lastly, kill the scoring system all together. I think trying to assign a letter or number 'grade' to something makes it more difficult on the reviewer and the reader to interpret what they mean. Let the reader decide based on the information thats given, don't spoon feed them. ;p
And here is where you fell of the wagon. Regardless of a rehash of mechanics thats not to say that games can't still be well made, and enjoyable. If you really look at gaming as a whole, all genres are mostly basic copies of the games before it.
Sure a platformer may have changed from 2D to 3D but does that really change the focus of them entirely? As the MMO genre unfolds we'll start to see more hybridization of styles, its inevitable. Just because one game is using a common system doesn't mean that the game is suddenly garbage.
Too many people are stuck in their MMO hovels scoffing at games that can't meet their standards and thinking everything that isn't appealing to them is beneath them. While I didn't really get into STO, nor did I play Champions for an extended period of time, I would be an idiot if I though that either game was complete garbage. Both games are quite playable, they each have very subtle nuances and changes in what makes them interesting, and while they aren't worth a 9.5 on the gaming scale, they definitely aren't worth a 6 either. (difference between an A and a D for those who love letter grades)
For MMORPGS I feel that you should still post two reviews. At first you should factor in that we do/do not believe this game is worth the initial purchase. Then after you have had time to play the game, you give your opinion on whether this game is worth a longer-term investment.
As for the reviewing system. I believe that you should review the several different parts of the game, splitting the review up. As this is all opinionated, I believe that whoever creates the review should give ratings to all the seperate part. Then the total grade that they believe the game deserves. It should not neccesarily be an average. As you pointed out, while everything that is in the game matters. Some parts just matter more.
Therefore, the major factors of a game should all get graded seperatly, then as a whole you should decide what the game deserves as a rating. WITHOUT looking at the average grade.
"Playable" is fine, but that doesn't mean they're good. Pretending they are is doing them and the industry a disservice. Accepting "mediocre" as good is a real problem. Its like American Idol. They keep letting the mediocre artists win when the good artists are the ones they vote off.
The simple solution to the rating dilemma, in my mind anyway, is to clearly state what the points are for. Allot points based the factors of the game.
A set amount for sound quality and diversity.
A set amount for the graphics without giving a handicap because the graphics are, "stylized", which is code for "I didn't want to do a lot of work so I made the characters look like steroid abusers".
A set amount for control fluidity, and so on.
Letting people reading your reviews see exactly what these points are being based on instead of, "I feel like.. 6.8 today. 6.8 it is." would probably go a long way. If something like that is already in place, maybe have it linked next to review scores as a kind of, "HEY THIS IS WHAT THESE NUMBERS MEAN" type of thing.
A rating system depends on what criteria you are attempting to quantify. What metrics are you attempting to use for a final grade will depend largely on the questions you are trying to answer, what are you trying to provide a reader, what do you think we want to know, or need to know? Once those questions are answered, the scoring can be any scale, however i usually find that if people are not rating a game low very often then that is because they are not being as impartial as they should be and there must be a lack in their ability to judge something without using only their own perspective. Keep in mind when you review something YOU are not the one the review is for. I believe a good review and scoring system should be based on asking a lot of questions.
Well, "good" is subjective. While I didn't like Cryptics newest games enough to subscribe to them, I do think they are worth a play through. Mediocre games are just that, some gamers like games that are more of a diversion while they wait for something else.
On the other hand some people may just really dig being in the Star Trek Universe, and while some people will likely complain its not Star Trekky enough, the game is completely playable, has some mechanics that are interesting. Every game doesn't have to knock everybody out of the park. If some games can just hit their core demographic well enough then I think they'll do fine. More people are expecting every game to stand up to World of Warcraft. What it comes down to though, it really doesn't look like Star Trek Online was trying to do anything but cater to Trekkies in the usual Cryptic Fashion. They aren't bad developers, they just don't shoot for the moon either.
Look at the "best in class" publications and critics. The common way of doing things is to have a preview (a.k.a. "fluff article") where someone that's anticipating the game writes up what it's all about. This part isn't about criticism. It's about passing on whatever information you deem important to the reader.
It is quite often that I will read on a thursday a detailed story about how a "brilliant" director with a "stellar" cast are working "on an original vision" from an "amazing" screenplay. Then on friday, when then film actually comes out, I'll read a one-star review telling me about the totally flat direction, terrible acting and laughable dialogue of this same picture.
There are two different jobs. Previewing and creating interest and excitement are one thing. Criticism is a different thing. A "Review" is about criticism.
As far as rating systems...honestly, it's all the eye of the beholder. The best film reviews I read are in the New Yorker. There is no rating of any kind assigned to them but it's abuntantly clear what the reviewer thinks and whether something's worth it. So the score doesn't matter, it's what the reviewer has to say.
That being said, if you're going to introduce a new scoring system, I think it's important to communicate what every rating means. Whether that means having a permanent legend or having the reviewer say: "i'm giving this a 4/5 because it's worth playing for a long time, but is still not 100% stable".
Given that there are so many aspects to MMORPGs and that every fan values them differently, it may be wise to use a set of ratings rather than just one. Complex things are often ranked on a set of attributes.
What makes up a MMORPG?
Setting & Lore
Solo Content
Group Content
Endgame Content
Crarfting Content
PvP Content
Immersion
World Interaction
Graphics (Technical)
Graphics (Artistic)
Stability / Bugginess
Customer Service
Promise (what's coming)
Value (Price/etc)
That's just stuff off the top of my head. But that's stuff that can be defined and outlined somewhere for your reviewers to use. What is the gold standard in all of these? Decide and state.
No matter what you do, reviews are going to be subjective. The key for a fan is to find a reviewer whose opinion they respect. There are 4 daily newspapers in my city that all have film reviews. I know exactly what i think of all the reviewers in those papers, so I know whose review I'm going to pay attention to. Your people don't have to be 100% consistent with each other. They are individuals after all. At the same time, there aren't that many MMOs out there. Get multiple people to review major releases.
Those are my thoughts, I have one last thing to add, but it's kinda separate from this.
~~~~~~~~~~~~~~~~
Why is it that MMORPG journalists NEVER ask any tough questions of developers or press developers for real responses when they're being brushed off? Getting to the real story is part of what a journalist does.
If you ask a SWTOR developer "How can you possibly design a major part of an MMO to be single-player only?" and the response they give you is "We love star wars as much as you" - that is complete and utter bullshit and any self-respecting journalist would be insulted by the answer.
For the love of god, PUSH! And if you don't get an answer assume the worst and write it, so that next time they know not to feed you bs. You are not idiots. People are not idiots. It often times seems that the writers of reviews and previews are the only people that are incapable of reading between the lines of what's being said.
"Id rather work on something with great potential than on fulfilling a promise of mediocrity."
- Raph Koster
Tried: AO,EQ,EQ2,DAoC,SWG,AA,SB,HZ,CoX,PS,GA,TR,IV,GnH,EVE, PP,DnL,WAR,MxO,SWG,FE,VG,AoC,DDO,LoTRO,Rift,TOR,Aion,Tera,TSW,GW2,DCUO,CO,STO
Favourites: AO,SWG,EVE,TR,LoTRO,TSW,EQ2, Firefall
Currently Playing: ESO