Oh I believe that developers listen to feedback... I have always believed that.
Sometimes its good, sometimes its bad and sometimes its taken somewhere that wasn't asked for.
Issues in general:
1) The loudest voices are usually not the majority of players.
2) As seen with many MMO's (SWG a good reference) the "forum game" is in general a very small percentage of the "game". Thus people who "play the game" and "don't play the fourms" log in one day to a totally screwed up experience.
3) If a lot of negative feedback is given do you actually see a drop off in subscriptions to tie these things together (aka is it just some unhappy vocal people but most people are happyily playing).
They are all kind of the same thing and obviously tied together.
On the other hand I also know that developers sometimes seem:
1) To lie
or
2) To be mis-informed
or
3) have very potent drugs in their systems.
Tho I use SWG as a reference to some of this..
Dark Age of Camelot and Mythic Entertainment are THE emboidement of ALL of these things. Which somehow seems ironic... and that goes back many years when a certain someone was still doing the community updates on the Camelot Herald.
My favorite two arguements from developers when their views do not mesh with feedback:
1) You have rose colored glasses. (yet developer cannot explain why their awesome idea have resulted in one server and the loss of some 55,000 to 60,000 concurrent online players).
2) We were bleeding subscribers (yet developer cannot explain why they had 250,000 subscribers when they were bleeding but only 50,000 when everything is "ok").
Is it really that hard to see why the people who give feedback but don't post on forums think they are not heard?
I think the entire subject is to wordy to get into and the article was a good read and illustrates much of what I think personally. Oddly most people who played a game and left almost always have a common time frame that the game was "no longer fun" even tho they may not agree on much else.
That "time frame" when things were no longer fun is 99% tied to a change that was made which no one seems to have wanted or asked for.
Now how all that ties together is... changes are made based on feedback.
That feedback doesn't represent the majority of the player base.
Or was most likely given by someone who just rage quit and isn't coming back anyway. (which is why the people left behind don't know why the game was changed the way it was... which just moves the cycle onward).
Obviously Devs have access to data we don't.. but the main issue is lack of communication. If there was more open and constructive communication then more people would be happy. Happy players = subscribers = money = happy devs with jobs.
I think a bit part of this is simply the design and implementation process at most MMo studios as well. If I have learned one thing from all my many years on "test" servers, submitting feedback and talking to the occasional dev it's that by the time nearly anyone in JQ Public category gets to see a "change" or "New feature", it's too late. The train has already left the station so to speak. You may be able to slightly, very slightly tweak some minor details with good feedback, but the change/expansion is basically happening anyways because the dev team has already spent XX days of paid time on it and no managing exec is just going to willingly write if off and say "oops, mabe that was a bad idea after all".
I loved the article, and I so miss the golf course.
I think the problem is that developers are champing at the bit with ideas they want to implement and when they see player feedback say: " We want the Frontiers to be bigger and the keep setups to change, running in a cirlce up to the lord room is silly" , developers attach their ideas to what the player has said instead of just giving us what we asked for.
Example:
Players": We want a bigger frontiers with more travel paths to run around in.
Developer: Oh sweet! Let's add triple the size of the frontiers and add in places that are "Too Steep" to run across. While we're at it, let's add in boat travel routes instead of portals. But we can add in a system of teleports too at every keep, that way people will only need to run from keep to keep if the portal system is broken. Oh and we can add bridges at every keep.
Players: We want keeps design to be updated. Currenty they are all just an open box with one room and one entrance to the lord.
Developers: Oh nice, we can add in destructible walls. That will take forever to destroy giving it the feel of a real world assault on a keep. Let's also add in towers that are shaped like a box and have on entrance to the keep lord. Oh we can put swamps around the keeps so people will run 1/5th their normal speed, further slowing down RvR and travel. Don't forget to put a lot of water around keeps so people dont lose health while sneaking up on the castle. It makes it near impossible to escape the keep if they are failing too!!
It was simple. We wanted bigger keeps that had better design. We didn't ask for standstill combat and blanket additions to turn RvR from dynamic warfare to keep sieging. We wanted bigger frontiers to run around in with multiple access to them so when the milegates were zerged or camped by the stealthers we had an opportunity to port into a second location via port!
Part of the problems with feedback in my opinion are :
Developers looking for a moment to exploit feedback to implement designs they want. Then blaming the players saying "You asked for this!"" No.. that's not really what we asked for...
Developers really not caring about feedback. " When we created Dark Age of Camelot we always had it in mind that siege warfarce would be a core concept of game design and the focus or our RvR experience where large battles of over 200 players would crash together." ---- Except that's not what was bringng enjoyment to the game for everyone. The players loved many aspects of RvR not just the large battles and keep sieges.
Developers not asking the right questions to narrow down exactly what that feedback is looking for. - So when you guys say you are unhappy with keep design what exactly are you asking for? Is it the difficulty, the shape, the lord room's tower in particular?
Here's hoping some miracle worker with a good listening skills can put together a game that could be dubbed DaoC 2: The Realm Wars Created Right.
...but the change/expansion is basically happening anyways because the dev team has already spent XX days of paid time on it and no managing exec is just going to willingly write if off and say "oops, mabe that was a bad idea after all".
I think this is where the best companies (like Blizzard) really differ from the rest of the pack.
Nobody always has perfect ideas, and throwing ten times as much money in the wrong direction just results in a pile of crap that's ten times bigger.
The difference is that good designers aren't afraid to pull the plug on bad ideas, eat their losses and get it over with. Most companies have a feedback disconnect on the INSIDE. No amount of "customer community dialogue facilitators" are ever going to fix that.
Good article, and one that can be expounded on somewhat.
The utility of feedback isn't just sabotaged from the customer end. While customers do tend to give crappy feedback ("I like this class, so make them capable of doing everything and I'll be happier"), we also have to consider the flaws on the other side of the screen.
What flaws? Let's say a lead developer has a certain notion of how raids should work. They put some folks on a design they want, then go out to get feedback on it. But if they aren't objective enough, they start skewing their viewpoint on the feedback. If feedback is negative to the new design, they start looking for reasons for the negativity other than poor design on their part. If the feedback is positive, they will frequently refuse to put the design through rigorous testing to make sure nothing really bad will come from it. They sabotage the customer feedback this way by not remaining objective themselves
Or let's say that a certain developer believes a class is imbalanced and wants feedback on it. But the developer never bothered to play the class themselves; they only take second-hand accounts of what the class is doing in the game to base a decision on. No matter whether the reaction is positive or negative, they make a poor decision because they aren't getting any actual face-time with the class, just using other players to gauge the extent of the change. They sabotage the customer feedback here by being less informed on the issue than the customer is, and that's just plain not acceptable (after all, we're paying them to be professionals so we don't have to be).
Blizzard and SOE have both been infamous for being somewhat sloppy about checking their data in a timely fashion, and often seem astonished when the data supports something the community has been trying to tell them about for some time. The fact is that you often see better math skills posted on the official forums at times than you see in the patch notes when some change is made, and that's just shameful. I don't expect devs to be perfect, but some of the seesawing being done in many MMOs these days is getting ridiculous. If they don't have the supposed math skills needed for programming computations of this nature, why were they hired in the first place?
Perhaps that is what we need in Quality Assurance departments in these companies: a division between customer feedback, and hard number crunching. Get some sociologists working on the customer end, do some questionaires, see if they can pick up the signal out of the noise on forums and such. Then get some folks, ideally programmers and mathemeticians, to look at the numbers being debated, see how things add up, and if not find out why.
One last thing that might help is if some of us customers stopped acting like $15 a month meant that these people had to do what we told them to do. We should also be a bit more objective. Is a class overpowered because they beat another one? Or is there a defect in the loosing class that needs to be addressed, rather than nerfing the other class? Is something "challenging", or is it really just too difficult to the point of stupidity? We're paying to play a game with other people, not above them.
I think if we were more realistic about what was going on in a game, and the companies were more interested in being more careful about analyzing potential issues, we'd get amazingly better results.
Another good article from Sanya. I remember the grab bags and Sanya's CM days for DAoC, although I played in Europe, not the US. The Herald Grab Bags were us Europeans' way of getting most of the info available and I remember back in the day that Sanya raised the bar for CM feedback.
Semi-related to this thread we have an event occuring in Warhammer Europe on Monday 27th July which is related to a lot of the issues raised in this article. Rather than waffling on, I'll just link you to the thread here on mmorpg.com,
In my opinion two very important points are missing. I accept your expertise in the area of community relations as a given and perhaps you didn't make your blog article exhaustive, however, since you've posted your thoughts and allow for replies I'll put in my two coppers.
First I read nothing about attempting to define the community. You seem to define "community" as if it's something static and monolithic and that it should be accepted without some attempt to understand it's component parts. It stands to reason that knowing just who you are dealing with is an important element in determining what to do with community feed back. You kind of approached this idea when you were writing about direct feed back, but it was done in a way that made it sound like the poster/written feedback was almost inherently untrustworthy. Focus groups have some of that same "unreliability" built in as what one focus group absolutely hates another might absolutely love. Defining the demographics of community gives context to the feed back received. It also helps you anticipate what kind of response is expected on your end.
Second, while feed back can be relied on to point to potential problems in any product I don't think it can be expected to point the developers to the reason for the potential problem. To use your own example if no one is complaining about a class of character then it is probably hideously overpowered. However it does nothing to tell you exactly why it's overpowered. Usually the combination of several factors makes something overpowered. Feed back usually points to the result (i.e. XYZ class is overpowered because it always pwns me and that's just not possible because I'm an uber player with mad skillz) instead of the root cause of the problem. I think that feed back should be used to bring attention to potential problem areas, but it then becomes incumbent on the Devs to study the potential problem and make a determination on exactly what the fix to that potential problem is or even if there is one necessary. You touched on this with your thinly veiled poke at Dark Age of Camelot but again stopped short of making the point and instead satisfied yourself with saying something along the lines of, if everyone is playing it and no one is complaining then it's broken.
As a community relations person your bias is always going to be toward the "open" end of the communication spectrum where devs slug it out on public message boards with the actual players. I understand that one of the best ways to get people vested in and playing a game habitually is to listen to their imput and respond to it. However I think that it is important for the Devs to maintain some dettachment from the community to avoid stampede type changes that the mercurial nature of communities dictates. It is a given that no matter what people have some of them won't like it, will scream for change, trumpet the change when it happens and then a week later sit down and pine for the old way. That doesn't completely devalue community feed back, but it does put it in it's proper perspective.
In conclusion it is simply this. The developers are the keepers of the original vision and direction of the game. The community is probably mostly ignorant of this original vision and direction. Therefore without context and careful consideration feed back is not a complete or reliable tool for the developers to use.
And shame on you for playing a Fire Controller.
Many a small thing has been made large by the right kind of advertising.
Comments
lol, Hibernia's original PVP Zone is the golf course she was talking about. I miss those days.
Oh I believe that developers listen to feedback... I have always believed that.
Sometimes its good, sometimes its bad and sometimes its taken somewhere that wasn't asked for.
Issues in general:
1) The loudest voices are usually not the majority of players.
2) As seen with many MMO's (SWG a good reference) the "forum game" is in general a very small percentage of the "game". Thus people who "play the game" and "don't play the fourms" log in one day to a totally screwed up experience.
3) If a lot of negative feedback is given do you actually see a drop off in subscriptions to tie these things together (aka is it just some unhappy vocal people but most people are happyily playing).
They are all kind of the same thing and obviously tied together.
On the other hand I also know that developers sometimes seem:
1) To lie
or
2) To be mis-informed
or
3) have very potent drugs in their systems.
Tho I use SWG as a reference to some of this..
Dark Age of Camelot and Mythic Entertainment are THE emboidement of ALL of these things. Which somehow seems ironic... and that goes back many years when a certain someone was still doing the community updates on the Camelot Herald.
My favorite two arguements from developers when their views do not mesh with feedback:
1) You have rose colored glasses. (yet developer cannot explain why their awesome idea have resulted in one server and the loss of some 55,000 to 60,000 concurrent online players).
2) We were bleeding subscribers (yet developer cannot explain why they had 250,000 subscribers when they were bleeding but only 50,000 when everything is "ok").
Is it really that hard to see why the people who give feedback but don't post on forums think they are not heard?
I think the entire subject is to wordy to get into and the article was a good read and illustrates much of what I think personally. Oddly most people who played a game and left almost always have a common time frame that the game was "no longer fun" even tho they may not agree on much else.
That "time frame" when things were no longer fun is 99% tied to a change that was made which no one seems to have wanted or asked for.
Now how all that ties together is... changes are made based on feedback.
That feedback doesn't represent the majority of the player base.
Or was most likely given by someone who just rage quit and isn't coming back anyway. (which is why the people left behind don't know why the game was changed the way it was... which just moves the cycle onward).
Obviously Devs have access to data we don't.. but the main issue is lack of communication. If there was more open and constructive communication then more people would be happy. Happy players = subscribers = money = happy devs with jobs.
I think a bit part of this is simply the design and implementation process at most MMo studios as well. If I have learned one thing from all my many years on "test" servers, submitting feedback and talking to the occasional dev it's that by the time nearly anyone in JQ Public category gets to see a "change" or "New feature", it's too late. The train has already left the station so to speak. You may be able to slightly, very slightly tweak some minor details with good feedback, but the change/expansion is basically happening anyways because the dev team has already spent XX days of paid time on it and no managing exec is just going to willingly write if off and say "oops, mabe that was a bad idea after all".
I loved the article, and I so miss the golf course.
I think the problem is that developers are champing at the bit with ideas they want to implement and when they see player feedback say: " We want the Frontiers to be bigger and the keep setups to change, running in a cirlce up to the lord room is silly" , developers attach their ideas to what the player has said instead of just giving us what we asked for.
Example:
Players": We want a bigger frontiers with more travel paths to run around in.
Developer: Oh sweet! Let's add triple the size of the frontiers and add in places that are "Too Steep" to run across. While we're at it, let's add in boat travel routes instead of portals. But we can add in a system of teleports too at every keep, that way people will only need to run from keep to keep if the portal system is broken. Oh and we can add bridges at every keep.
Players: We want keeps design to be updated. Currenty they are all just an open box with one room and one entrance to the lord.
Developers: Oh nice, we can add in destructible walls. That will take forever to destroy giving it the feel of a real world assault on a keep. Let's also add in towers that are shaped like a box and have on entrance to the keep lord. Oh we can put swamps around the keeps so people will run 1/5th their normal speed, further slowing down RvR and travel. Don't forget to put a lot of water around keeps so people dont lose health while sneaking up on the castle. It makes it near impossible to escape the keep if they are failing too!!
It was simple. We wanted bigger keeps that had better design. We didn't ask for standstill combat and blanket additions to turn RvR from dynamic warfare to keep sieging. We wanted bigger frontiers to run around in with multiple access to them so when the milegates were zerged or camped by the stealthers we had an opportunity to port into a second location via port!
Part of the problems with feedback in my opinion are :
Developers looking for a moment to exploit feedback to implement designs they want. Then blaming the players saying "You asked for this!"" No.. that's not really what we asked for...
Developers really not caring about feedback. " When we created Dark Age of Camelot we always had it in mind that siege warfarce would be a core concept of game design and the focus or our RvR experience where large battles of over 200 players would crash together." ---- Except that's not what was bringng enjoyment to the game for everyone. The players loved many aspects of RvR not just the large battles and keep sieges.
Developers not asking the right questions to narrow down exactly what that feedback is looking for. - So when you guys say you are unhappy with keep design what exactly are you asking for? Is it the difficulty, the shape, the lord room's tower in particular?
Here's hoping some miracle worker with a good listening skills can put together a game that could be dubbed DaoC 2: The Realm Wars Created Right.
I think this is where the best companies (like Blizzard) really differ from the rest of the pack.
Nobody always has perfect ideas, and throwing ten times as much money in the wrong direction just results in a pile of crap that's ten times bigger.
The difference is that good designers aren't afraid to pull the plug on bad ideas, eat their losses and get it over with. Most companies have a feedback disconnect on the INSIDE. No amount of "customer community dialogue facilitators" are ever going to fix that.
To the OP:
Great article.
Thanks!
Good article, and one that can be expounded on somewhat.
The utility of feedback isn't just sabotaged from the customer end. While customers do tend to give crappy feedback ("I like this class, so make them capable of doing everything and I'll be happier"), we also have to consider the flaws on the other side of the screen.
What flaws? Let's say a lead developer has a certain notion of how raids should work. They put some folks on a design they want, then go out to get feedback on it. But if they aren't objective enough, they start skewing their viewpoint on the feedback. If feedback is negative to the new design, they start looking for reasons for the negativity other than poor design on their part. If the feedback is positive, they will frequently refuse to put the design through rigorous testing to make sure nothing really bad will come from it. They sabotage the customer feedback this way by not remaining objective themselves
Or let's say that a certain developer believes a class is imbalanced and wants feedback on it. But the developer never bothered to play the class themselves; they only take second-hand accounts of what the class is doing in the game to base a decision on. No matter whether the reaction is positive or negative, they make a poor decision because they aren't getting any actual face-time with the class, just using other players to gauge the extent of the change. They sabotage the customer feedback here by being less informed on the issue than the customer is, and that's just plain not acceptable (after all, we're paying them to be professionals so we don't have to be).
Blizzard and SOE have both been infamous for being somewhat sloppy about checking their data in a timely fashion, and often seem astonished when the data supports something the community has been trying to tell them about for some time. The fact is that you often see better math skills posted on the official forums at times than you see in the patch notes when some change is made, and that's just shameful. I don't expect devs to be perfect, but some of the seesawing being done in many MMOs these days is getting ridiculous. If they don't have the supposed math skills needed for programming computations of this nature, why were they hired in the first place?
Perhaps that is what we need in Quality Assurance departments in these companies: a division between customer feedback, and hard number crunching. Get some sociologists working on the customer end, do some questionaires, see if they can pick up the signal out of the noise on forums and such. Then get some folks, ideally programmers and mathemeticians, to look at the numbers being debated, see how things add up, and if not find out why.
One last thing that might help is if some of us customers stopped acting like $15 a month meant that these people had to do what we told them to do. We should also be a bit more objective. Is a class overpowered because they beat another one? Or is there a defect in the loosing class that needs to be addressed, rather than nerfing the other class? Is something "challenging", or is it really just too difficult to the point of stupidity? We're paying to play a game with other people, not above them.
I think if we were more realistic about what was going on in a game, and the companies were more interested in being more careful about analyzing potential issues, we'd get amazingly better results.
Another good article from Sanya. I remember the grab bags and Sanya's CM days for DAoC, although I played in Europe, not the US. The Herald Grab Bags were us Europeans' way of getting most of the info available and I remember back in the day that Sanya raised the bar for CM feedback.
Semi-related to this thread we have an event occuring in Warhammer Europe on Monday 27th July which is related to a lot of the issues raised in this article. Rather than waffling on, I'll just link you to the thread here on mmorpg.com,
http://www.mmorpg.com/discussion2.cfm/thread/244858/Warhammer-Europe-27th-July-2009-Europe-Stand-Together.html
For further info, feedback or if you wish to support the event, drop by my blog at warofalts.wordpress.com
Roo Stercogburn
In my opinion two very important points are missing. I accept your expertise in the area of community relations as a given and perhaps you didn't make your blog article exhaustive, however, since you've posted your thoughts and allow for replies I'll put in my two coppers.
First I read nothing about attempting to define the community. You seem to define "community" as if it's something static and monolithic and that it should be accepted without some attempt to understand it's component parts. It stands to reason that knowing just who you are dealing with is an important element in determining what to do with community feed back. You kind of approached this idea when you were writing about direct feed back, but it was done in a way that made it sound like the poster/written feedback was almost inherently untrustworthy. Focus groups have some of that same "unreliability" built in as what one focus group absolutely hates another might absolutely love. Defining the demographics of community gives context to the feed back received. It also helps you anticipate what kind of response is expected on your end.
Second, while feed back can be relied on to point to potential problems in any product I don't think it can be expected to point the developers to the reason for the potential problem. To use your own example if no one is complaining about a class of character then it is probably hideously overpowered. However it does nothing to tell you exactly why it's overpowered. Usually the combination of several factors makes something overpowered. Feed back usually points to the result (i.e. XYZ class is overpowered because it always pwns me and that's just not possible because I'm an uber player with mad skillz) instead of the root cause of the problem. I think that feed back should be used to bring attention to potential problem areas, but it then becomes incumbent on the Devs to study the potential problem and make a determination on exactly what the fix to that potential problem is or even if there is one necessary. You touched on this with your thinly veiled poke at Dark Age of Camelot but again stopped short of making the point and instead satisfied yourself with saying something along the lines of, if everyone is playing it and no one is complaining then it's broken.
As a community relations person your bias is always going to be toward the "open" end of the communication spectrum where devs slug it out on public message boards with the actual players. I understand that one of the best ways to get people vested in and playing a game habitually is to listen to their imput and respond to it. However I think that it is important for the Devs to maintain some dettachment from the community to avoid stampede type changes that the mercurial nature of communities dictates. It is a given that no matter what people have some of them won't like it, will scream for change, trumpet the change when it happens and then a week later sit down and pine for the old way. That doesn't completely devalue community feed back, but it does put it in it's proper perspective.
In conclusion it is simply this. The developers are the keepers of the original vision and direction of the game. The community is probably mostly ignorant of this original vision and direction. Therefore without context and careful consideration feed back is not a complete or reliable tool for the developers to use.
And shame on you for playing a Fire Controller.
Many a small thing has been made large by the right kind of advertising.