I think the reviews would benefit from being scored relating to different gamer types. For example:
* Achievement hunter
* Trader/hoarder of stuff
* Social gamer/Guildie
* PvP/PvE gamer
* Explorer
It would be nice to read your opinion about how the game fits different gamer types either during the course of the review or in a review summary. I think this would help relate to the game being reviewed.
For me, I normally glance at the pro and cons reviewers give to see if any (personally) game-breaking cons are listed. Then I look to numbers only for technical or semi-technical ratings like performance. Finally, I don't care what number a reviewer gives to gameplay or fun or things of that sort. I just read what they write. Lastly, I look at the overall grade for a reality check. Bottom line for me is whether or not the pro v con list scares me off.
Good article. On a side note, the whole concept of journalistic integrity I find silly. It's like people want to legislate and regulate deeply personal decisions an individual reviewer makes. Give me a break. It's impossible to know what influences a person really has without really getting ot kow them. Even then it's just conjecture. Take their review for what it is and make your own choice.
Finally, happy Friday everyone! Have a great weekend!
I think when trying to decide on a new format you need to think about how that format would apply to existing games. I'm not sure one format is going to fit all - for example: Darkfall, WoW, and Free Realms all are MMOs. The folks looking for a new game are going to want to know if they are good games for them. So target audience is going to have to be identified and any grades/rating reflect that fact. Ideally a member of the target audience would be the one doing the reviewing.
What would be nice is to know how each of the following grade a game: raider, crafter, pvper and soloer. I expect there are some reviewers that can fulfill multiple roles but even then we're talking a lot of reviews because MMOs evolve and what was experienced on day one will be totally different within a few months.
Ultimately if I know the reviewers game background and if they themselves are willing to shell out bucks for the game that is more information than any score could ever give me.
The biggest thing that is needed with your scoring system is some kind of consistency in the ratings. I remember at one point a review for Age of Conan gave it a very low score (like a 6 maybe, I didn't go look it up). That was probably one of the lowest reviews ever. To put it in comparison, Vanguard scored better. So did Matrix. So did SWG for the NGE review!
That really ticked me off. Sure AoC had issues, but worse than the NGE? Worse than Vanguard? Seriously? I do not know if the reviewer was a big PvPer (as so many at MMORPG.com are. Nothing made me chuckle more than early reviews of DCUO that started aout by talking about the issues like bad chat but then got all giddy as a school boy talking about how great the open world PvP was and that made the game AWESOME....anyway) so maybe he was just upset that all of his Sieging dreams did not come true.
Regardless, scores should be meaningful. if you give something a 4, any game that gets a 4 needs to be in that same category. Maybe the final score should be tweaked by the Editor to make sure that there is consistency.
A breakdown by category would be my prefered answer, with 2 planned reviews.
A technical biased "early look"- heavily concerned with stability and performance matters. Really it's a what is in the game and does it work style of review. It would be nice if you could also do installs on high and low end rigs, playing around enough to judge the differences.
The second review- the resub review a month (roughly) later. This would be the more "traditional" review, grading how well things work together. By this time the community should have settled a bit (and its importance gauged), you are into the typically boring middle game, and you'd have had time to play around with the systems. This is also a good chance to "update" on any technical improvements.
Such a system would be a huge improvement. Placing the review focus on if a working product was delivered early on, then assessing its lasting power with the review a month in. It probably won't make many friends though. Devs will dislike the requirements on content/gameplay, since they won't be able to hide the reality of their game behind the initial "hook" hours.
I agree with this. Then you can also add a final score so you have something for everyone. More options is always nice imo. Although, personally I don't have major problems with the current system also, so..
I think arieste just became my personal hero. Talk about hitting the nail on the head.
Its like when EQ2 went F2P and we all started asking for direct answers to direct questions in detail on how it would work and what it would do and we kept getting blown off then got upset and were totally ignored.
PUSH. Get the answer. You're reporters, so do your jobs, be fearless, and get it.
Why not allow gamers to score the games themselves along side of your scoring method, since this site already kind of does this using the anticipation factor currently in use. Then you could either have both scores stand alone together, or average the scores to provide the overall score. Either way i feel most here really want more of a break down of the metrics your using, they want more depth to the review than just commentary by the authors saying that its the bees knees, or lackluster.
Score a game on several key "MMO" specific features and toss in a bonus (x2) score for "innovation" or "creativity". Total up the 1-10 scores, divide by the number of catagories and be done with it.
The most important thing is establishing specific items you feel are key to the MMO genre you represent (grouping, guilds, chat etc). Any game missing one of the pillars of traditional MMOs will get a 0 in that catagory (housing for instance), but they would be able to make up for it with something "innovative" (Rifts, action combat, voice acting).
Heck with reviews, I would be happy for some SOE outtage coverage here as of late. If armchair analysts have been putting up a slew of serious updates and articles - I am most eager to see what actual analysist without weak wills and with integrity can do. Anxiously awaiting those news pieces. Especially since they do have contacts with developers and can give us honest feedback.
Is this article for finding solutions or as a means to have it out with your own community? Because i see a scorn and contempt here that seems out of place, and it isn't just coming from the community either.
In any case, this isn't the thread to talk about whether or not you think our reviews are biased, etc. it's a place to talk about the direction that the format of our reviews should take in the future. We could continue arguing, but it's just going to boil down to the same stuff it always does. People are going to make accusations and I'm going to respond to them and no one will be more convinced of anything either way.
If people want to think we're corrupt, then that's their perogative, I suppose.
So, if I overreacted,then I apologize. I'm tired of having to defend myself from accusations against my integrity on a daily basis and I lashed out.
I dislike the overall number for a MMO. No matter if its numbers or letters. As long as there is no consistency between different reviewers, then that score is useless. You guys have to remember that the readers only compare scores and not include 'oh but thats reviewer x and he gives relatively high scores compared to reviewer y' etc.
I do like a pros/cons list though. But instead of just naming some random features or whatever the reviewer thinks of, why not make this list consistent? Make a list of points to which you put the MMO against. The reviewers personal view can then be put in his conclusion at the end of the review and maybe with added comments if the pro/con list isnt complete enough. Maybe its an idea to hold a poll for which things should be on the list.
Then there were those previews which then end up with a recommendation to buy or not. I dont see why that is necessary at all. Isnt the idea that the reader makes up his/her own mind after reading the review? The problem with this recommendation is that a reader might have complete different expectations from a MMO then the reviewer has. This just asks for disappointed ppl. So I rather would see a pros/cons list instead with maybe some sections that are more general.
Something like this.
Graphics : A description about player toons maybe, armoursets, textures etc.
Animation : Is it choppy, does using a skill break off animations etc.
Music/Sound : What does it add to the game? Is it just generic music that you will soon turn off?
World : How immersive, guided paths?
PVP : (could break if up in types even if necessary or list it behind this) FFA, arena, RvR etc
Crafting/Gathering: recipedrops, nodehunting etc
Housing : Instanced or available at all
A list thats used with every review, with behind it a short description and then put in Pros if its exceptional, or Cons if its bad. Everything else just in seperate neutral section or so. If a summary would be made like this, then an overall score isnt even necessary. You can leave that up to the readers and their voting system. But anyone who wants to try a new MMO, finds a summary like this maybe more usefull.
Reviews for me serve the purpose of getting a inital overview of things, reading multiple site reviews and general player comments on a game is a good thing sure but the only review I would ever really trust is my own internal review - what i mean is the first month in a game will make a much bigger impact on potential future play than any review will
As far as rating are concenred, like the review themselves they are subjective and personal to the reviewer and I only take them lightly, getting included onto metacritic isnt such a big deal, most of us are wary of those scores now anyway so maybe running a system that doesnt conform to metacritics system would make you look more open/genuine (i am not saying to me you dont now)
The main issue with the review is that it should be descriptive of features but not judging them. Why? because some feature or lack of them are good for some players and bad for others. For instance, if you state that instancing is bad, that is pretty relative even is bad it will be more important for some people than others. Using descriptive terms like quest oriented or mob grinding oriented, open world, someway instanced or fully instanced, are the things that the gamers need to know to take their own decision. I don't need to be told what I should like. I need to know how the game is in order to anticipate if I am going to like it or not.
I would really like it if a review wasn't released after only say 4 hours of playing. Or even 24 hours. Reviewers need to take long than that before they start actually saying its great.
And a game should NEVER be reviewed in beta stages. Games are always fun when they're new and shiny.
I'd like to say what happened when I started Rift. I got in on first beta. For the first few days of playing (about 10 hours), it was alot of fun. Everything was shiny and new, it was pretty to look at, true. But then I started noticing the flaws. The fact that there were too many mobs in one spot, that they spawned too fast, that Rift's started to become an annoyance and a chore. And then I started noticing that actually there's very little about the game thats actually different from any other game. Even the multi souled idea has been done elsewhere. And there are major flaws in even minor functions. Like the map is pretty much totally useless. The functionality of the brokers is very limited even with the most recent update they did to it, and its definitely much more limited than EQ2 which has been out for far longer. I started out with a renewing sub every 6 months. By the time beta was over, I had changed it to 3 months. Within a week after release, it was down to month to month. I barely log in, and if I do, I play for an hour or so and then log. For gods sakes, the game company just released a week long trial period and thats not a good sign when its only been open two months so obviously I'm not the only one to feel this way.
These games are releasing before they're finished and none of the reviewers ever mention it. If you're going to review, then review it completely, and not only after a few hours of play. The company isn't willing to give you more than 4 hours to try it out? Then maybe there's a reason for that. They don't want you noticing what the game is actually lacking. Mention it. Its a big deal. Be honest. Be open. Don't say its good if it isn't.
The scoring system, I think is fine. It could use less numbers. I prefer 1-5. But as far as preview, review, whatever, listen here.
The reason most people get upset about your reviews is the unbelieveable inconsistantcy at which you review. The latest failure was with FF14, where you guys were showing a clear bias to not review that game. Many people thought you were paid, some people thought you guys were just fanboys of square (aren't we all). The question being, why did FF14 get such grace when other MMO's that released in better shape didn't.
Consistancy is what you need. Whatever you do, however you do it, be consistant. As a professional reviewer you have to be over sensitive to showing bias.
While I'm on the subject. The "Buy it now" logo or whatever, is a terrible idea. Just god awful. If you wanted to breed conspiracies, this is definately the route to go. A review should be a review, give insight to the game and how it plays. A simple buy it now button does none of that, and just says that the reader should buy it. This is niether a good way to review, or not show bias.
I think that people who are inclined to read reviews are smart enough to understand that waiting longer for a review is not only warrented, but a smart move as far as MMORPG's go. You can't review MMORPG's without an end game. Perhaps a multi part review for all MMORPG's, review early, mid, and end game for all of them.
But again, whatever you do, be consistant.
Gauge
EDIT:
In response to the last post by the OP. The ranting about bias and things of that nature are coming from long time users of this site who had no problem when you were reviewing when the site opened. They are angry because the reviews have not been getting better, they have been getting worse. The moderators are getting worse, the selling is getting worse. Everything is getting worse.
I'm sure sometime in your life you felt a product that you like sold out and lost what you held dear about it. That is how a lot of us feel about mmorpg.com right now. It's hard to let go of a community you've been a part of so long. Many of us had an unsurmountable respect for the unbiased nature of this sites reviews.
IMO it is impossible to talk about reviews on this site without this subject coming up.
Also before I get discredited, Like many I was a long time reader before I decided to post. I started on this site sometime in the very early 2000's...I don't know the exact year or date.
It's quite clear that your sites integrity is in question. Be it deserved, or not it seems to be the problem. This is because of what ??? Soft reviews? Not asking the tough questions ? Unrealisticly high reviews of games that later turn out to be crap?
The whole Payola/Swag issues is really but a symptom of the larger issue. To attack people that are only doing as you ask them to do is not very smart either. You admitt your system is flawed and you ask for input.
Be honest, If a game is crap say so.
If a game is the same old pig with a different shade of lipstick, Say So.
Players are looking for someone that will hold the dev's feet to the fire and make them introduce something new and exciting. Live up to their own hype and deliver what they promise. If they don't hammer them hard.
Asking if we prefer numbers or letters in our reviews is like putting a bandaid on the sinking Titanic. If you lack the Integrity and your readers trust. No system will mean anything.
The problem with a review is that it is something that someone who has paid for the game gives to someone who hasn't. This means it's one person's individual experience appealing to one person's individual characteristics. The MMORPG experience is going to last about a month before you can give a real review. But after a month it stops being a review and starts being a critique.
It's not possible to review a game in a way that will suggest whether or not it will last with any player. That's something the player needs to decide.
What the reviewer can and should do is to review the game based upon what the game is trying to accomplish, the genre it's trying to fit in (or the ways it crosses genres) and whether it's successful in doing what it sets out to do.
Also the parts that make up the whole.
A game might be so fun your head bleeds but the crafting might suck or be non-existent. To someone who hates crafting this could be a plus. To someone who requires crafting a huge minus.
Let's take "Lord of the Rings online", a game that I have a subscription to but haven't really played in earnest in quite a few months.
It's not a bad game and does some things realy well. But what grade or score does one give it? It does many things well so should it get an A? A B+?
Well, it falls short in the pvp side of things so giving it a high score will immediatley cause those who love/want pvp to scratch their heads because for them a game with bad pvp or no pvp might not be worth playing.
Some games might have everything that a hardcore player might want but are released with a lot of bugs. The first inclination is to give it a failing grade. But if it's playable and offers game play that hasn't been in recent releases then maybe it has worth to a player who is looking to build his own hut and make his own tools and who doesn't care about some bugs.
I suppose in the end what one needs to do is to get in their and just describe the experience.
Is it buggy, does it include all the promised features, do those features work as intended and are they done well? Is the art design good and consistent throughout game world, are the graphics (lighting, shadows, DX "whatever") done well or does it feel dated?
Is the writing good, does the game guide the player or does it expect the player to find his/her own way?
Just telling it "like it is" is worth quite a lot to a player. Not "game play sucks" but "game play incorporates a lot of repitition" or "game play requires the player to have his/her hand in many facets of the game or they wont' be compeitive", etc.
I completely understand the need for a letter or number grade because it sums up the reviewers thoughts. However, I don't think one grade is going to tell the whole story or even make an honest impression. Again, if one hates pvp then LOTRO could be an A. If one wants/requires pvp then it could be a D.
The other thing is that, as the article suggests, 1-10 isn't good because anything below 6 is never used as it's failing. The converse could also be true. Should a game get a 10? what constitues a 10? Nothing is perfect so a 10 shouldn't be a "perfect" score. 10 should be achievalbe. Or 1.
or better put, the highest and lowest scores of whatever system you use should be achievable.
Like Skyrim? Need more content? Try my Skyrim mod "Godfred's Tomb."
the reviews, it should be obvious that there need to be two, one near launch or close afterwards, call it "launch review", "progress report/review" or "initial review" and make it clear that it's not the definitive review but more of an early report.
That review covers like about the first 40-80 hours of gameplay and what can be said about the game as experienced by the reviewer in that time.
Then, after a month or 2 after launch a second review follows that can hit all the areas of gameplay that an MMO offers, especially it'll pay attention to the endgame content that was left out in the initial launch review, how much of a switch it is in gameplay or not, and how the new MMORPG as a whole stands. For some MMO's, the reviewer will have been able to hit level cap and the endgame content sooner just like other players (eg DCUO, CO), for other MMO's it'll take longer to hit that level cap and the endgame content (eg Aion), the point is that this second review can take the endgame content into account, because this is where often the (in some cases 'bait and') switch takes place in gameplay, and it's often this type of content, the amount and variety in it, that has to provide longterm value to an MMORPG, the reasons why people keep being subbed to it.
If deemed fitting, the rating of an MMORPG will then be corrected to reflect this progressive insight better.
Or to put it shortly, the initial launch review will focus on whether it's fun to play the game in the first place as far as can be determined and guessed with a mere 40-80 hours under the belt, why the second definitive review will showcase a wizened outlook upon the new MMORPG, one that will focus on its longterm value and what arguments it offers for players to be subbed to it.
About the rating, I see no problem with it, although as said above, the rating should reflect progressive insight which a follow up review a few months after launch can give.
And maybe different game aspects could be rated, plus a short 3 sentence commentary accompanying the rating, why a rating is given like it is. I don't think that a game rating should just be an average of the rating of the subratings of that game's gameplay aspects, the overall rating should show how these separate game elements combine into the overall gameplay experience.
Note: and if you want to show site visitors that you're willing to listen to their recommendations and viewpoints regarding how they want to see reviews being done, then just collect a number of the best and most well received suggestions, present them in a few concrete options and then let forum visitors vote on them in a poll, with the addition that ideas can slight changes can still be added to the most popular option based upon emerging ideas from the debate about it
The ease with which predictions are made on these forums: Fratman: "I'm saying Spring 2012 at the earliest [for TOR release]. Anyone still clinging to 2011 is deluding themself at this point."
If you really want to improve your reviews it is actually quite a simple path. Whether you use scores of 1-10, 1-00, A-Z is irrelevant. What is important is that YOU as the managing editor set standards which reviewers will use to judge a game
You want game companies to be held accountable for shipping incomplete and buggy games? Well then make that a cornerstone of your reviews. Don't leave it up to various writers to value this. It results in contradictory reviews and scores.
All the criteria should be the same for each review. Come up with a STANDARD scoring system. For instance:
25 Completeness- Does the game feel complete, polished and free of bugs? Does it meet a minimum acceptable level of content?
15 Graphics- Self explanitory
15 Story/Lore/Gameworld - How developed is the gameworld. Does it feel immersive...
15 Innovation - Does the game improve upon what has come before?
10 UI - How intuitive is the interface.. how customizeable?
10 Social Tools - Does the game foster player interaction?
10 MISC - Reviewers choice
That's just a sample.. it doesn't matter if you make it A-Z, 1-100 or anything else. Simply have clearly defined categories which we can use to compare from review to review. Make sure the reviewers are trained to understand what these guidelines mean so we don't have instances where a reviewer says he doesn't judge a game on how incomplete or buggy it is at review time because it has "potential" to be fixed later. Oh yeah.. the word "potential" should be banned from reviews as once you go down that path it becomes a PREVIEW and nor a REVIEW. Grade a game on what exists at the review time. Go back in 6 month and review it again if you liek but please dear God never EVER give points in a review for what might "potentially" be added or fixed in the future...
Lastly, make sure the review categories mirror the standards and values of the site. If you as the managing editor truly feel that games that are buggy and incomplete should NOT be rewarded with great reviews then make sure that category exists in your review AND it is weighted more highly than the other categories!
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
When I first read the thread title I was hoping you were going to actually rate our critiquing. I was thinking, "OK this guy has some serious courage" to go up against the kind of people you find on some forums. The kind of people that would be like,
What? You are critiquing our critique on your critique? Well here is my critique of your critique of us critiquing you on your critique:
But then I can see why one would think better of it. I would love to hear the positive and negative comments the writers have about some of the things you guys read from the posters after a review though. I was hoping to have some light shed on why players react the way they do to your reviews and how you guys deal with it. I am glad that you were able to better explain the way you envision game reviews for the site and sharing some of the complexities of online game reviewing that many readers rarely think about. There are probably a lot of positive things players say about reviews.
Personally, I like the re-reviews and would even like to see 3-month, 6-month, and anniversary reviews for some of the games that interest me the most. I don't have the time or money to check out all the games that look good to me initially and I can appreciate what reviewers have to juggle as these games progress past launch. I pay attention to the forums here because you can get a fairly accurate impression on whether or not developers are able to step up to the plate after launch or if they blow it, in my opinion. Initial reviews can be spotty for online games. I even expect them to be. But the review combined with feedback from players here, trolls and all, paints a pretty clear picture for me, most of the time.
Vault-Tec analysts have concluded that the odds of worldwide nuclear armaggeddon this decade are 17,143,762... to 1.
I don't see many problems with the reviews as they stand. I like the whole pre-release - launch - re-review thing myself. Not a big fan of the scoring sistem but what you could replace it with? That's for the wiser minds than my own. What I would like to see is more than one opinion on one review, after all one persons tripple A is another persons Fly for fun. (No offence to ment to any flyf fans out there.) For instance review as normal then a small section of counter opinions like "Jon thinks this, Mike thinks this" e.t.c e.t.c. Whatever you do there is going to be some unsatisfied customers out there. As much as I love this community, and I do. Sometimes we come accross as some of the most negative, self entitled, and frankly paranoide people I have ever seen on all the internets. I feel the more information we get the less moronic conjecture there will be. But I am probably deluding myself The thing is we are mostly aloud to act like the worst kind of total turdburger and people expect the staff to act with total professionalism at all times. On a platform with (Mostly) this much freedom of expresion I think that's nigh on impossible. Also from some of the things I have been reading in this discusion I get the idear that some posters belive the staff are all taking huge backhanders from Blizzard and flying around in private jets. Let's keep a sense of perspective here shall we? Just my own opinoion, no offence ment. I still enjoy reading every article on this site.
Obviously, a game journalist has to review a game ASAP. But it does create a problem with MMOs, because they are expected to be played for a long, long time. And too many games are fun for the first 5 hours, and then the rest of the game just stinks. Its hard to review what the game is like at level 50 when you're doing a review based off 3 days of play or less. Therefore, I suggest a level range review. As an example, if a game has 50 levels review every 10 levels. 1-10, 11-20, 21-30, etc. Once the reviewer has done all the levels, then he/she can give a synopsis. Of course,a problem here is how will the reviewer find time to do this when he/she was expected to play the game for 5 hours, and tell the world all about the game.
Maybe this can be left up to player correspondents. That's a good way for someone wanting to get into the journalism to gain some experience.
Honestly, I don't even bother to look at scores. I just read what others players say, and I find that I agree with what the majority say.
Comments
I think the reviews would benefit from being scored relating to different gamer types. For example:
* Achievement hunter
* Trader/hoarder of stuff
* Social gamer/Guildie
* PvP/PvE gamer
* Explorer
It would be nice to read your opinion about how the game fits different gamer types either during the course of the review or in a review summary. I think this would help relate to the game being reviewed.
For me, I normally glance at the pro and cons reviewers give to see if any (personally) game-breaking cons are listed. Then I look to numbers only for technical or semi-technical ratings like performance. Finally, I don't care what number a reviewer gives to gameplay or fun or things of that sort. I just read what they write. Lastly, I look at the overall grade for a reality check. Bottom line for me is whether or not the pro v con list scares me off.
Good article. On a side note, the whole concept of journalistic integrity I find silly. It's like people want to legislate and regulate deeply personal decisions an individual reviewer makes. Give me a break. It's impossible to know what influences a person really has without really getting ot kow them. Even then it's just conjecture. Take their review for what it is and make your own choice.
Finally, happy Friday everyone! Have a great weekend!
I think when trying to decide on a new format you need to think about how that format would apply to existing games. I'm not sure one format is going to fit all - for example: Darkfall, WoW, and Free Realms all are MMOs. The folks looking for a new game are going to want to know if they are good games for them. So target audience is going to have to be identified and any grades/rating reflect that fact. Ideally a member of the target audience would be the one doing the reviewing.
What would be nice is to know how each of the following grade a game: raider, crafter, pvper and soloer. I expect there are some reviewers that can fulfill multiple roles but even then we're talking a lot of reviews because MMOs evolve and what was experienced on day one will be totally different within a few months.
Ultimately if I know the reviewers game background and if they themselves are willing to shell out bucks for the game that is more information than any score could ever give me.
The biggest thing that is needed with your scoring system is some kind of consistency in the ratings. I remember at one point a review for Age of Conan gave it a very low score (like a 6 maybe, I didn't go look it up). That was probably one of the lowest reviews ever. To put it in comparison, Vanguard scored better. So did Matrix. So did SWG for the NGE review!
That really ticked me off. Sure AoC had issues, but worse than the NGE? Worse than Vanguard? Seriously? I do not know if the reviewer was a big PvPer (as so many at MMORPG.com are. Nothing made me chuckle more than early reviews of DCUO that started aout by talking about the issues like bad chat but then got all giddy as a school boy talking about how great the open world PvP was and that made the game AWESOME....anyway) so maybe he was just upset that all of his Sieging dreams did not come true.
Regardless, scores should be meaningful. if you give something a 4, any game that gets a 4 needs to be in that same category. Maybe the final score should be tweaked by the Editor to make sure that there is consistency.
I agree with this. Then you can also add a final score so you have something for everyone. More options is always nice imo. Although, personally I don't have major problems with the current system also, so..
Eat me!
I think arieste just became my personal hero.
Talk about hitting the nail on the head.
Its like when EQ2 went F2P and we all started asking for direct answers to direct questions in detail on how it would work and what it would do and we kept getting blown off then got upset and were totally ignored.
PUSH. Get the answer. You're reporters, so do your jobs, be fearless, and get it.
Lemme keep it simple...
Return to your roots.
Score a game on several key "MMO" specific features and toss in a bonus (x2) score for "innovation" or "creativity". Total up the 1-10 scores, divide by the number of catagories and be done with it.
The most important thing is establishing specific items you feel are key to the MMO genre you represent (grouping, guilds, chat etc). Any game missing one of the pillars of traditional MMOs will get a 0 in that catagory (housing for instance), but they would be able to make up for it with something "innovative" (Rifts, action combat, voice acting).
Heck with reviews, I would be happy for some SOE outtage coverage here as of late. If armchair analysts have been putting up a slew of serious updates and articles - I am most eager to see what actual analysist without weak wills and with integrity can do. Anxiously awaiting those news pieces. Especially since they do have contacts with developers and can give us honest feedback.
In any case, this isn't the thread to talk about whether or not you think our reviews are biased, etc. it's a place to talk about the direction that the format of our reviews should take in the future. We could continue arguing, but it's just going to boil down to the same stuff it always does. People are going to make accusations and I'm going to respond to them and no one will be more convinced of anything either way.
If people want to think we're corrupt, then that's their perogative, I suppose.
So, if I overreacted,then I apologize. I'm tired of having to defend myself from accusations against my integrity on a daily basis and I lashed out.
In any case, let's get this back on topic please.
Cheers,
Jon Wood
Managing Editor
MMORPG.com
I dislike the overall number for a MMO. No matter if its numbers or letters. As long as there is no consistency between different reviewers, then that score is useless. You guys have to remember that the readers only compare scores and not include 'oh but thats reviewer x and he gives relatively high scores compared to reviewer y' etc.
I do like a pros/cons list though. But instead of just naming some random features or whatever the reviewer thinks of, why not make this list consistent? Make a list of points to which you put the MMO against. The reviewers personal view can then be put in his conclusion at the end of the review and maybe with added comments if the pro/con list isnt complete enough. Maybe its an idea to hold a poll for which things should be on the list.
Then there were those previews which then end up with a recommendation to buy or not. I dont see why that is necessary at all. Isnt the idea that the reader makes up his/her own mind after reading the review? The problem with this recommendation is that a reader might have complete different expectations from a MMO then the reviewer has. This just asks for disappointed ppl. So I rather would see a pros/cons list instead with maybe some sections that are more general.
Something like this.
Graphics : A description about player toons maybe, armoursets, textures etc.
Animation : Is it choppy, does using a skill break off animations etc.
Music/Sound : What does it add to the game? Is it just generic music that you will soon turn off?
World : How immersive, guided paths?
PVP : (could break if up in types even if necessary or list it behind this) FFA, arena, RvR etc
Crafting/Gathering: recipedrops, nodehunting etc
Housing : Instanced or available at all
A list thats used with every review, with behind it a short description and then put in Pros if its exceptional, or Cons if its bad. Everything else just in seperate neutral section or so. If a summary would be made like this, then an overall score isnt even necessary. You can leave that up to the readers and their voting system. But anyone who wants to try a new MMO, finds a summary like this maybe more usefull.
Reviews for me serve the purpose of getting a inital overview of things, reading multiple site reviews and general player comments on a game is a good thing sure but the only review I would ever really trust is my own internal review - what i mean is the first month in a game will make a much bigger impact on potential future play than any review will
As far as rating are concenred, like the review themselves they are subjective and personal to the reviewer and I only take them lightly, getting included onto metacritic isnt such a big deal, most of us are wary of those scores now anyway so maybe running a system that doesnt conform to metacritics system would make you look more open/genuine (i am not saying to me you dont now)
So really, what ever you decide is fine with me
The main issue with the review is that it should be descriptive of features but not judging them. Why? because some feature or lack of them are good for some players and bad for others. For instance, if you state that instancing is bad, that is pretty relative even is bad it will be more important for some people than others. Using descriptive terms like quest oriented or mob grinding oriented, open world, someway instanced or fully instanced, are the things that the gamers need to know to take their own decision. I don't need to be told what I should like. I need to know how the game is in order to anticipate if I am going to like it or not.
I would really like it if a review wasn't released after only say 4 hours of playing. Or even 24 hours. Reviewers need to take long than that before they start actually saying its great.
And a game should NEVER be reviewed in beta stages. Games are always fun when they're new and shiny.
I'd like to say what happened when I started Rift. I got in on first beta. For the first few days of playing (about 10 hours), it was alot of fun. Everything was shiny and new, it was pretty to look at, true. But then I started noticing the flaws. The fact that there were too many mobs in one spot, that they spawned too fast, that Rift's started to become an annoyance and a chore. And then I started noticing that actually there's very little about the game thats actually different from any other game. Even the multi souled idea has been done elsewhere. And there are major flaws in even minor functions. Like the map is pretty much totally useless. The functionality of the brokers is very limited even with the most recent update they did to it, and its definitely much more limited than EQ2 which has been out for far longer. I started out with a renewing sub every 6 months. By the time beta was over, I had changed it to 3 months. Within a week after release, it was down to month to month. I barely log in, and if I do, I play for an hour or so and then log. For gods sakes, the game company just released a week long trial period and thats not a good sign when its only been open two months so obviously I'm not the only one to feel this way.
These games are releasing before they're finished and none of the reviewers ever mention it. If you're going to review, then review it completely, and not only after a few hours of play. The company isn't willing to give you more than 4 hours to try it out? Then maybe there's a reason for that. They don't want you noticing what the game is actually lacking. Mention it. Its a big deal. Be honest. Be open. Don't say its good if it isn't.
The scoring system, I think is fine. It could use less numbers. I prefer 1-5. But as far as preview, review, whatever, listen here.
The reason most people get upset about your reviews is the unbelieveable inconsistantcy at which you review. The latest failure was with FF14, where you guys were showing a clear bias to not review that game. Many people thought you were paid, some people thought you guys were just fanboys of square (aren't we all). The question being, why did FF14 get such grace when other MMO's that released in better shape didn't.
Consistancy is what you need. Whatever you do, however you do it, be consistant. As a professional reviewer you have to be over sensitive to showing bias.
While I'm on the subject. The "Buy it now" logo or whatever, is a terrible idea. Just god awful. If you wanted to breed conspiracies, this is definately the route to go. A review should be a review, give insight to the game and how it plays. A simple buy it now button does none of that, and just says that the reader should buy it. This is niether a good way to review, or not show bias.
I think that people who are inclined to read reviews are smart enough to understand that waiting longer for a review is not only warrented, but a smart move as far as MMORPG's go. You can't review MMORPG's without an end game. Perhaps a multi part review for all MMORPG's, review early, mid, and end game for all of them.
But again, whatever you do, be consistant.
Gauge
EDIT:
In response to the last post by the OP. The ranting about bias and things of that nature are coming from long time users of this site who had no problem when you were reviewing when the site opened. They are angry because the reviews have not been getting better, they have been getting worse. The moderators are getting worse, the selling is getting worse. Everything is getting worse.
I'm sure sometime in your life you felt a product that you like sold out and lost what you held dear about it. That is how a lot of us feel about mmorpg.com right now. It's hard to let go of a community you've been a part of so long. Many of us had an unsurmountable respect for the unbiased nature of this sites reviews.
IMO it is impossible to talk about reviews on this site without this subject coming up.
Also before I get discredited, Like many I was a long time reader before I decided to post. I started on this site sometime in the very early 2000's...I don't know the exact year or date.
It's quite clear that your sites integrity is in question. Be it deserved, or not it seems to be the problem. This is because of what ??? Soft reviews? Not asking the tough questions ? Unrealisticly high reviews of games that later turn out to be crap?
The whole Payola/Swag issues is really but a symptom of the larger issue. To attack people that are only doing as you ask them to do is not very smart either. You admitt your system is flawed and you ask for input.
Be honest, If a game is crap say so.
If a game is the same old pig with a different shade of lipstick, Say So.
Players are looking for someone that will hold the dev's feet to the fire and make them introduce something new and exciting. Live up to their own hype and deliver what they promise. If they don't hammer them hard.
Asking if we prefer numbers or letters in our reviews is like putting a bandaid on the sinking Titanic. If you lack the Integrity and your readers trust. No system will mean anything.
The problem with a review is that it is something that someone who has paid for the game gives to someone who hasn't. This means it's one person's individual experience appealing to one person's individual characteristics. The MMORPG experience is going to last about a month before you can give a real review. But after a month it stops being a review and starts being a critique.
Website: http://www.thegameguru.me / YouTube: http://www.youtube.com/users/thetroublmaker
It's not possible to review a game in a way that will suggest whether or not it will last with any player. That's something the player needs to decide.
What the reviewer can and should do is to review the game based upon what the game is trying to accomplish, the genre it's trying to fit in (or the ways it crosses genres) and whether it's successful in doing what it sets out to do.
Also the parts that make up the whole.
A game might be so fun your head bleeds but the crafting might suck or be non-existent. To someone who hates crafting this could be a plus. To someone who requires crafting a huge minus.
Let's take "Lord of the Rings online", a game that I have a subscription to but haven't really played in earnest in quite a few months.
It's not a bad game and does some things realy well. But what grade or score does one give it? It does many things well so should it get an A? A B+?
Well, it falls short in the pvp side of things so giving it a high score will immediatley cause those who love/want pvp to scratch their heads because for them a game with bad pvp or no pvp might not be worth playing.
Some games might have everything that a hardcore player might want but are released with a lot of bugs. The first inclination is to give it a failing grade. But if it's playable and offers game play that hasn't been in recent releases then maybe it has worth to a player who is looking to build his own hut and make his own tools and who doesn't care about some bugs.
I suppose in the end what one needs to do is to get in their and just describe the experience.
Is it buggy, does it include all the promised features, do those features work as intended and are they done well? Is the art design good and consistent throughout game world, are the graphics (lighting, shadows, DX "whatever") done well or does it feel dated?
Is the writing good, does the game guide the player or does it expect the player to find his/her own way?
Just telling it "like it is" is worth quite a lot to a player. Not "game play sucks" but "game play incorporates a lot of repitition" or "game play requires the player to have his/her hand in many facets of the game or they wont' be compeitive", etc.
I completely understand the need for a letter or number grade because it sums up the reviewers thoughts. However, I don't think one grade is going to tell the whole story or even make an honest impression. Again, if one hates pvp then LOTRO could be an A. If one wants/requires pvp then it could be a D.
The other thing is that, as the article suggests, 1-10 isn't good because anything below 6 is never used as it's failing. The converse could also be true. Should a game get a 10? what constitues a 10? Nothing is perfect so a 10 shouldn't be a "perfect" score. 10 should be achievalbe. Or 1.
or better put, the highest and lowest scores of whatever system you use should be achievable.
Godfred's Tomb Trailer: https://youtu.be/-nsXGddj_4w
Original Skyrim: https://www.nexusmods.com/skyrim/mods/109547
Serph toze kindly has started a walk-through. https://youtu.be/UIelCK-lldo
Nice discussion item. Here's my thoughts:
the reviews, it should be obvious that there need to be two, one near launch or close afterwards, call it "launch review", "progress report/review" or "initial review" and make it clear that it's not the definitive review but more of an early report.
That review covers like about the first 40-80 hours of gameplay and what can be said about the game as experienced by the reviewer in that time.
Then, after a month or 2 after launch a second review follows that can hit all the areas of gameplay that an MMO offers, especially it'll pay attention to the endgame content that was left out in the initial launch review, how much of a switch it is in gameplay or not, and how the new MMORPG as a whole stands. For some MMO's, the reviewer will have been able to hit level cap and the endgame content sooner just like other players (eg DCUO, CO), for other MMO's it'll take longer to hit that level cap and the endgame content (eg Aion), the point is that this second review can take the endgame content into account, because this is where often the (in some cases 'bait and') switch takes place in gameplay, and it's often this type of content, the amount and variety in it, that has to provide longterm value to an MMORPG, the reasons why people keep being subbed to it.
If deemed fitting, the rating of an MMORPG will then be corrected to reflect this progressive insight better.
Or to put it shortly, the initial launch review will focus on whether it's fun to play the game in the first place as far as can be determined and guessed with a mere 40-80 hours under the belt, why the second definitive review will showcase a wizened outlook upon the new MMORPG, one that will focus on its longterm value and what arguments it offers for players to be subbed to it.
About the rating, I see no problem with it, although as said above, the rating should reflect progressive insight which a follow up review a few months after launch can give.
And maybe different game aspects could be rated, plus a short 3 sentence commentary accompanying the rating, why a rating is given like it is. I don't think that a game rating should just be an average of the rating of the subratings of that game's gameplay aspects, the overall rating should show how these separate game elements combine into the overall gameplay experience.
Note: and if you want to show site visitors that you're willing to listen to their recommendations and viewpoints regarding how they want to see reviews being done, then just collect a number of the best and most well received suggestions, present them in a few concrete options and then let forum visitors vote on them in a poll, with the addition that ideas can slight changes can still be added to the most popular option based upon emerging ideas from the debate about it
The ACTUAL size of MMORPG worlds: a comparison list between MMO's
The ease with which predictions are made on these forums:
Fratman: "I'm saying Spring 2012 at the earliest [for TOR release]. Anyone still clinging to 2011 is deluding themself at this point."
If you really want to improve your reviews it is actually quite a simple path. Whether you use scores of 1-10, 1-00, A-Z is irrelevant. What is important is that YOU as the managing editor set standards which reviewers will use to judge a game
You want game companies to be held accountable for shipping incomplete and buggy games? Well then make that a cornerstone of your reviews. Don't leave it up to various writers to value this. It results in contradictory reviews and scores.
All the criteria should be the same for each review. Come up with a STANDARD scoring system. For instance:
25 Completeness- Does the game feel complete, polished and free of bugs? Does it meet a minimum acceptable level of content?
15 Graphics- Self explanitory
15 Story/Lore/Gameworld - How developed is the gameworld. Does it feel immersive...
15 Innovation - Does the game improve upon what has come before?
10 UI - How intuitive is the interface.. how customizeable?
10 Social Tools - Does the game foster player interaction?
10 MISC - Reviewers choice
That's just a sample.. it doesn't matter if you make it A-Z, 1-100 or anything else. Simply have clearly defined categories which we can use to compare from review to review. Make sure the reviewers are trained to understand what these guidelines mean so we don't have instances where a reviewer says he doesn't judge a game on how incomplete or buggy it is at review time because it has "potential" to be fixed later. Oh yeah.. the word "potential" should be banned from reviews as once you go down that path it becomes a PREVIEW and nor a REVIEW. Grade a game on what exists at the review time. Go back in 6 month and review it again if you liek but please dear God never EVER give points in a review for what might "potentially" be added or fixed in the future...
Lastly, make sure the review categories mirror the standards and values of the site. If you as the managing editor truly feel that games that are buggy and incomplete should NOT be rewarded with great reviews then make sure that category exists in your review AND it is weighted more highly than the other categories!
All time classic MY NEW FAVORITE POST! (Keep laying those bricks)
"I should point out that no other company has shipped out a beta on a disc before this." - Official Mortal Online Lead Community Moderator
Proudly wearing the Harbinger badge since Dec 23, 2017.
Coined the phrase "Role-Playing a Development Team" January 2018
"Oddly Slap is the main reason I stay in these forums." - Mystichaze April 9th 2018
When I first read the thread title I was hoping you were going to actually rate our critiquing. I was thinking, "OK this guy has some serious courage" to go up against the kind of people you find on some forums. The kind of people that would be like,
What? You are critiquing our critique on your critique? Well here is my critique of your critique of us critiquing you on your critique:
But then I can see why one would think better of it. I would love to hear the positive and negative comments the writers have about some of the things you guys read from the posters after a review though. I was hoping to have some light shed on why players react the way they do to your reviews and how you guys deal with it. I am glad that you were able to better explain the way you envision game reviews for the site and sharing some of the complexities of online game reviewing that many readers rarely think about. There are probably a lot of positive things players say about reviews.
Personally, I like the re-reviews and would even like to see 3-month, 6-month, and anniversary reviews for some of the games that interest me the most. I don't have the time or money to check out all the games that look good to me initially and I can appreciate what reviewers have to juggle as these games progress past launch. I pay attention to the forums here because you can get a fairly accurate impression on whether or not developers are able to step up to the plate after launch or if they blow it, in my opinion. Initial reviews can be spotty for online games. I even expect them to be. But the review combined with feedback from players here, trolls and all, paints a pretty clear picture for me, most of the time.
Vault-Tec analysts have concluded that the odds of worldwide nuclear armaggeddon this decade are 17,143,762... to 1.
I don't see many problems with the reviews as they stand. I like the whole pre-release - launch - re-review thing myself. Not a big fan of the scoring sistem but what you could replace it with? That's for the wiser minds than my own. What I would like to see is more than one opinion on one review, after all one persons tripple A is another persons Fly for fun. (No offence to ment to any flyf fans out there.) For instance review as normal then a small section of counter opinions like "Jon thinks this, Mike thinks this" e.t.c e.t.c. Whatever you do there is going to be some unsatisfied customers out there. As much as I love this community, and I do. Sometimes we come accross as some of the most negative, self entitled, and frankly paranoide people I have ever seen on all the internets. I feel the more information we get the less moronic conjecture there will be. But I am probably deluding myself
The thing is we are mostly aloud to act like the worst kind of total turdburger and people expect the staff to act with total professionalism at all times. On a platform with (Mostly) this much freedom of expresion I think that's nigh on impossible. Also from some of the things I have been reading in this discusion I get the idear that some posters belive the staff are all taking huge backhanders from Blizzard and flying around in private jets. Let's keep a sense of perspective here shall we? Just my own opinoion, no offence ment. I still enjoy reading every article on this site.
Obviously, a game journalist has to review a game ASAP. But it does create a problem with MMOs, because they are expected to be played for a long, long time. And too many games are fun for the first 5 hours, and then the rest of the game just stinks. Its hard to review what the game is like at level 50 when you're doing a review based off 3 days of play or less. Therefore, I suggest a level range review. As an example, if a game has 50 levels review every 10 levels. 1-10, 11-20, 21-30, etc. Once the reviewer has done all the levels, then he/she can give a synopsis. Of course,a problem here is how will the reviewer find time to do this when he/she was expected to play the game for 5 hours, and tell the world all about the game.
Maybe this can be left up to player correspondents. That's a good way for someone wanting to get into the journalism to gain some experience.
Honestly, I don't even bother to look at scores. I just read what others players say, and I find that I agree with what the majority say.