Howdy, Stranger!
It looks like you're new here. If you want to get involved, click one of these buttons!
Quick Links
General: Saturday Debate: Beta, Test or Trial?
Stradden
Managing EditorMember CommonPosts: 6,696
This week, Dan Fortier and Derek Czerkaski check in with our Saturday Debate. The question is, are beta test supposed to be for serious testing purposes, or are they merely a free trial of the game?
Dan Fortier:Back in the early days of MMORPGs, the beta phase of game development was nothing more than a tool for the designers to see how their ideas translated from paper to reality and to work out the bugs that would have otherwise been passed over. The list of players that were asked to join the early stages were those who were committed to testing and were eager to sacrifice their time to improve the stability and playability of a game that they were interested in.
How far we've fallen since then.
You can read the whole debate here.
Cheers,
Jon Wood
Managing Editor
MMORPG.com
Comments
For instance the forum where the beta testers post maybe only have one community person meaning that a lot of
interaction is lost.
Of course the devs will be busy working on the game but they need to watch out for what the players want and hopefully
build something will make the devs proud and players happy.
I remember when i was betatesting outforce not that great of a game but the beta experiance was 100% fun
was talking and playing with programmers one day and had a blast chatting and playing with sound tech the next.
Of course you cant demand that focus to the player when it comes to larger games but still something in that direction
would be fun..
-Semper ubi sub ubi!
always wear underwear
Beta's in MMOs are a complete and utter mess, in more ways than one.
They have become a free trial. The majority of people wanting to get into an MMO beta want to A) Get a trial of the game before they buy,
MMOs have started advertising access to beta as part of the pre-order package. Hello? What are they thinking? Both RF Online and Light And Dark did this, both of them had people cancelling their pre-orders in droves. Why on earth would you give your key player base access to an unfinished game.
And while we're on the subject of unfinished. The state of MMOs in beta is a disgrace. I know its become the norm for an MMO to be 'unfinshed' at release and have months of bug fixes, but come on... Where are the internal testers? Games should not be entering beta when the whole thing is still a confusing mess. Look at Vanguard, they're adding new (bugged) features less than two weeks before launch. If I was making an MMO I wouldn't be adding anything to the live/beta server that hadn't be tested internally to a high degree and had all noticable bugs corrected. Beta should be testing the bugs that are impossible to catch with a small internal testing team.
MMOs are like loners in the game industry, they do there own thing and refuse to learn and evolve from the industry as a whole. I'm getting sick of it.
One of the things that I felt from the very beginning that hurt auto assault was the main focal point of the concept of the game driving. I suggested they do things to make it more fun to play, but it fell on def ears they didn't even respond to my post about it and I was in closed beta telling them this so I waited and when it hit open beta I started testing it out some more again, but it's like nothing had changed so I got bored even quicker and left and was kind of like good luck to them I see this one flopping because there wasn't enough entertainment value because it felt way to much like a repetitive grind since the driving part of it was so generic and bland. It wasn't at all what you would come to expect from a next gen mmorpg there was nothing innovative enough to keep enjoyable enough to want to keep playing. It felt like a cut and paste job, but with important parts missing so basically a hackjob. The vehicles all felt the same sure some were a little bit faster and some wider and heavier, but that was like the only noticeable differences between them and while the actual class roles of them were different that wasn't the focal point of the game in the first place it was driving which that aspect was just sub par. It felt nothing like a modern race car game or car game in general does. There should be a little more control to breaking than tapping the button and stopping entirely. How can miss such a simple feature like that and expect a car game to be good. I think most people who tried auto assault were expecting something a long the lines of a massively multiplayer style version of twisted metal meets carmageddon except with rpg features mixed in. Instead they got a terrible implementation of that because they chose to ignore the important features. They focused all there time and effort on the classes, but ignored the driving and handling features too much.
I think development teams need to rethink the whole beta testing process a bit because of things you mentioned like testings looking to discover bugs to exploit later and just playing to figure out the best way to level once the games launched if they decide on buying it. They should focus more on closed beta testers a bit and create more focused testing situations so players can't know everything about the game. Skip open beta all together just phase players in gradually till release and each phase bring new players in that way. The people who sign up all get a chance to test the game, but none of them get to see it all or know it all till release time then have like a one week free trial at launch time for players still interested.
But a large-scale beta test is the worst of both worlds. The first beta test I was in was for DnD online. As a systems engineer, I think I'm used to a bit more analytical effort than most people, and after the first two hours of play, I had six pages of detailed notes on things I thought should be fixed or improved.
Then I stopped and thought: "OK, fine. But what the heck am I supposed to DO with all of these notes?"
There was a beta forum, but it wasn't something that could be part of any design process; it was just like any other MMORPG forum: a shouting match between children (well, at least I HOPE most of those dyslexic narcissists were children!) Trying to provide useful input in a forum like that would be as much use as spitting into the wind.
Sure, they had the bug reporting software in the game itself. But why the heck would I want to work as an unpaid Quality Assurance tech? Sorry, but I don't see the point to that.
In other words: it's not the player's decision as to whether they are beta testing the game; that is entirely up to the designers. If I feel like I'm part of a team where my feedback is appreciated, and I'm doing something I like to do, I'll stick with it as long as I can spare the time, whether I'm being paid or not.
Otherwise, it's nothing but a demo where the developer hopes you'll forgive some glitches. A bunch of idiots can tell me it's some sort of sacred bond, and I'm somehow obligated to spend a hundred hours of my time on it even if I don't enjoy it. But guess what? I don't care.
I usually agree with Dan, but here, I will agree with Derek.
First and foremost, the beta tester must either be willing to be saints...or be interested in the game. Personnally, when I apply to a beta I feel like the game as a lot of potential...if my interest is too high, I don't want to be in the beta, I want to wait and be rdy as a racer for game launch.
See, let's take a concrete example, I apply for Vanguard beta and was invited in the late phase. However, even in my application form for the beta, I was talking about...anyone guess? Yes, raiding been a problem. Now that it is stated, once they let's me in...what will I do? Nope, I didn't start talking about raid non-stop. I find a few bugs (prolly already found bugs but not yet fixed), I try a few quests, I check how my computer can run the game and feedback about it and my feeling as if this is playable or not. But then, I start digging a LOT more into the FAQ, to know if I can find this game fun or not in the long run, and I found out, as I expect, that raiding will prevent me from having fun in the long run. My interest dwindle to nothingness, so I didn't beta anymore, I stop testing anything from there. See, even if I disagree about the lack of instancing, it is not a gamer killer. If I disagree about tradeskills been enforced, grouping been enforced or anything else (the list would be long), it is still not game killers. I could still enjoy the game, despite many disagreements. However, when it comes to raiding...deadline. It is a game killer. They know that before letting me in, I mean, I am talking about that since 2002 (EQluclin) and I am on this site and have over 5k posts, most of them talking of that problem in a way or another and use to have a signature about that as well!
Personnally when I do a beta, I have 3 goals: 1- Find as many bugs as possible. 2- Feedback a LOT about bugs and my point of view about each, they can discard my point of view, however I must organised my feedbacks so they can see the problem and then my opinion and just discard my opinion. 3- Judge the game; design-wise.
Yes, no matter what peoples will say, I always judge a game when I am in beta. Will I want to play that game? In the case of DDO, the answer was no and it was firm. They lost 1 sold box (a competitor gain that money). The pre-order beta cost them 1 sold box. In the case of Auto Assault, I really like what I saw. I didn't feedbacks as much as I would have love to, but I did my best and actually enjoy that; so I buy a game I would otherwise not have buyed. In the case of Vanguard, it is grey as with a raid-free server I would enjoy the game, but with raiding forget it, it is not going to happen. I won't start or stop my arguments about raiding because I join a beta, because it rains or because of whatever. After my time in Vanguard, it confirm everything I was thinking, the game would be nice and I would be calling it home and even go buy a new computer, however, there is still the same critical condition, no raiding enforced in any possible way my twisted mind can conceive...raid-free servers would work, but it wouldn't have needed to be as drastic; back in time...now with so little time, I don't see any other possible way to get my interest and a purchase. Vanguard is nice, but not enough with raiding enforced, if I want a nice game that enforce raiding, I will go get BC. What does it have that WoW doesn't offer? Enforced raiding is enforced raiding, 1%, 20% or 100% is all the same and all unacceptable.
See, if I disgree about something, I say it, and then I am done and it is over (ever saw me go endlessly saying that a game MUST incorporate instancing? No, I do believe it would be an edge for ANY game, but I don't care that much about that). If I disagree to the point it turn a sweet product into hell, then my duty is to get them to understand this, and that it will cost them at least 1 player, prolly many more.
I still feel I didn't make them understand that raiding is killing their game enought, that I have a duty to say it more, and loudlier than ever! I don't think they understand just how badly raiding is crippling their game (or WoW, the more successfull the game, the harsher is raiding crippling them); I wish I could make them understand but I obviously fail at explaining my point...even in 5 years. I don't care if they think ill of me, good of me, I want them to understand just how bad raiding is on the game. Raiding is like a muscular enhancing drug. It is not good at all in the long run. And I feel they are too casuals to understand, that the only peoples who understand me, well they are with FoH and actually turning me ridicule without much troubles and I don't care too look like a fool, but I wish I could convey them to understand this point, as they obviously don't. I am not the ideal messenger, but I guess I am the only one caring enought to do it, so there I do it...until the point will be understood, be it tomorrow or in 20 more years!
It would be easier to just raid, and far less time consuming...and I would get, individually, what I want. But...that wouldn't help them in the least. And raiding would have nasty effects on my personnality, raiders turn very sour in time.
So yes, games will be judged by peoples playing in betas. However, I don't think beta reviews will change much in the opinion of potential buyers. Conflicting design would however. Graphics, lags, whatever technical can be overcome, a weak design is there to stay. Would have I buyed Vanguard without beta testing it? Nope, so they didn't lost a sold boxe in my case, but they gain more feedbacks, maybe they don't want this feedback, this is not for me to judge. But DDO actually lost a sold boxe by allowing me in beta, as I would have been buying it before trying it, and I was appealed.
Would I ever play a LOT before the release of a game and give TONS of beta feedbacks? Unlikely...I want to keep my focus for when it matter, when it is the official release of a game. However, I may give quite some few nice feedbacks, depending on the situation.
- "If I understand you well, you are telling me until next time. " - Ren
I, personally, feel split on this whole debate. Yes, I'm there to see if the game is fun, of course, but I'm also there to find bugs and report them. No, I will not stick around and report bugs for a game that I feel has no potential, or that is complete drudgery. However, I do my very best to be forgiving of any game that I think might shape up into something I would pay a monthly fee for (or play for free, if that's how the game is set up).
I don't see how one can be on either side of the fence on this one, really, and be a "true" beta tester. Fun factor is essential. Reporting bugs and making suggestions is essential also. We do this not to make a game that does exactly what we want, but to help developers see where we feel the game is lacking. If we all do this, then our collective subjective opinions coalesce into something approaching true objectivity as these ideas support or contradict each other.
Quitting a beta, to me, sends a clear message that the game does not have enough going for it for me to spend any more of my time evaluating and bug-testing it. As betas happen in phases, if the game has some merit, then I would return to again evaluate and test the game. I am always delighted to be part of any beta test, but developers and Dan need to understand that I'm not being paid to improve this game, so the only currency that matters is how the game impacts my level of enjoyment. I'm not asking for a six-digit salary, of course, but I do expect at least minimum wage!
If I buy a game and get a pre-order beta invitation, I'm there to play the game. With a possible small side-order of insight into the process.
Far and away I'm finding, in both scenarios, the inability to quickly and easily report bugs/thoughts about the game play. Offering the testers easy access to an in-game forum/bug reporting system/GMs themselves could potentially increase the number and quality of the reports. How often do I feel like sitting down at my computer with a pad of paper and a pencil? How often do I feel like getting up to find one if I do come across a bug?
Ultimately, the developers should be putting more of their time into listening to the beta testers and getting the problems fixed. If I were playing in a beta, having patches daily to fix the smaller bugs and weekly to fix the bigger ones would be a great sign that our voices were being heard. A forum post by a community relations expert saying, 'We hear you and all the things you want fixed are on the way!' is little comfort in an age of unfulfilled expectations about games.
Early beta tests typically have a vibrant community that is pounding away at the game, they go through several wipes and test test test. The communities of early beta tests are also made up of a more mature crowd of gamers, they normally play with or against the devs, are able to communitcate effectively and understand how the beta process works as many of them have been doing it for SEVERAL years, I know I have been beta testing since before some people around here had even graduated from their speak n spells :P Some of us, because of the quality of our testing even get invited by parent companies or publishers to test single player games I can think of a half dozen or so that I have taken part of.
I am starting to believe that Mr. Fortier either plays devil's advocate, or has little to no comprension of software development processes or creative processes in general. If testers didnt bash systems etc then the developers will think everythign is happy go lucky and go about making mistake after mistake. There is a game releasing very soon where the developers consistently ignored their beta testers from the very early stages, have lied repeatedly on this site and many others and are soon to release a broken and unenjoyable product. Myself and other testers weren't there to try and change the developer's vision... that wasnt our task. Our task was to provide feedback on systems, quests and basic mechanics. There is another game where the developers have taken a completely different approach. We, the testers, told them what we felt was missing from the game and they pushed back their release date from 2006 to sometime in 2007 to get in everything we have told them they need. Including a complete rewrite of portions of the skeletal system on all npcs.
Beta testing is an integral part of the development process. A company can hire hundreds of people to come and sit inhouse on systems the company has to provide to do the same thing we do for free. And very likely put in many more hours than hired people can and will do.
Are all beta testers actually testers? no, not saying they are. But the developers are able to see who is and who isnt and all but one game that I am currently testing actually went out of their way to make the games better because of the input from the testers. One of the last interviews I had read from Mr. McQuaid he bemoaned the fact that his testers werent submitting bug reports.... I know I for one submitted hundreds upon hundreds of reports and the last time I played a great many of those bugs were still in game, after a complete rewrite of that section of the game world.
A tester is only valuable if the company desires honest feedback and will listen to their testers. Otherwise they are waisting their money and our time.
http://www.speedtest.net/result/7300033012
I help manage User acceptance tests at my firm, and believe me, user reported bugs are much harder to decipher and replicate, (and therefore fix) so I can understand why beta's (particularly open betas) probably provide little useful information.
At best, Dev's would look for trends, or multiple reports of similar issues, and work their way from there. I can't see any way for them to deal with all the issues that are reported.
As for some folks just using it as a free trial, I'm think that's probably true. I know when I was in Vanguard beta I reported a few bugs and rated all the quests (while that system was in place)
"True friends stab you in the front." | Oscar Wilde
"I need to finish" - Christian Wolff: The Accountant
Just trying to live long enough to play a new, released MMORPG, playing New Worlds atm
Fools find no pleasure in understanding but delight in airing their own opinions. Pvbs 18:2, NIV
Don't just play games, inhabit virtual worlds™
"This is the most intelligent, well qualified and articulate response to a post I have ever seen on these forums. It's a shame most people here won't have the attention span to read past the second line." - Anon
I help manage User acceptance tests at my firm, and believe me, user reported bugs are much harder to decipher and replicate, (and therefore fix) so I can understand why beta's (particularly open betas) probably provide little useful information.
At best, Dev's would look for trends, or multiple reports of similar issues, and work their way from there. I can't see any way for them to deal with all the issues that are reported.
As for some folks just using it as a free trial, I'm think that's probably true. I know when I was in Vanguard beta I reported a few bugs and rated all the quests (while that system was in place)
Well if it is going to be "difficult" to sift through all those bug reports, then I really have to question the goals of the beta test cycle or the management of the process.
It was the first game I ever tested that was completely and utterly devoid of any entertainment value for me. I realized that I didn't want to help make that game any better, because the game lacked enough promise for me to care about the product's quality. Also, when you hear about their development team playing WoW together for several hours a day, rather then improving their product, it just goes to show they have a problem with prioritizing.
In my opinion, if you're bored of your own game, then how do you think your testers feel?
Waiting for something fresh to arrive on the MMO scene...
I help manage User acceptance tests at my firm, and believe me, user reported bugs are much harder to decipher and replicate, (and therefore fix) so I can understand why beta's (particularly open betas) probably provide little useful information.
At best, Dev's would look for trends, or multiple reports of similar issues, and work their way from there. I can't see any way for them to deal with all the issues that are reported.
As for some folks just using it as a free trial, I'm think that's probably true. I know when I was in Vanguard beta I reported a few bugs and rated all the quests (while that system was in place)
While what you say has merit thats why the devs need to be selective in whom they choose. Not just going for people in huge guilds or that run fan sites. But people that are able to bring a critical eye to their product and assist them in tracking down bugs. As a software engineer I for one know that all developers and in house people are somewhat limited in what they can and what they will try to do to any software system. As a developer if you know that, for example, putting numbers in a text field isnt going to work you may very well, on a subconscius level, not put those numbers in. It is just the nature of the thing. Not saying anythign bad about anyone but that is why it is important to get many many different people pounding away on a system.
http://www.speedtest.net/result/7300033012
The transition from MSGS to SOE went relatively smoothly, but a large amount of dev -> tester communication came to a stop. And after they re-opened a number of the areas they had closed off for "balance and bug fixes" only to find myself reporting the same 20 bugs from the first 2 hours of play that I had reported nearly a year ago well... it became harder and harder to take the time to write up the clear and concise reports that I take personal pride in.
Then came the flood of people. The testing community became an absolute sham during phase 4 and everything seemingly went from bad to worse.
Now this is just not vanguard. There have been other games with developers that have the same mentality. The Outpost by Sierra (several years ago, a single player game) was absolutely horrible to be a part of. Beta testing horizons
And of course there are polar opposites to these. Ultima Online betas are still a cherished memory. Not to mention a couple I am currently involved in (under NDA) where the developers take a very pro-active stance and strive to make their game as enjoyable as they can even if it involves pushing back release dates and completely rewriting a few systems and their subsystems. It all depends on the developer. And there are a handful out there that "get it". Too bad the majority, in my experience, do not.
http://www.speedtest.net/result/7300033012
As Kyleran said, sorting the feedbacks is an issue in itself.
I never work on a big project like a MMO or even a regular RPG, merely on tiny little flash games...and in-house, not even going outside, during an afternoon, they got over 500 bugs reported by 10 "testers". In fact, the real amount of bugs is prolly around 200, so 300 are only a repetition of the same bugs, who are quite obvious as they are more spotted, which in turn make the reader feel like he is reading all the same stuff, over and over again.
Honestly, these companies should have someone with strong reading skills to sort these bugs report (aka the marketing departement could possibly fill this role, just like the designers but with more ability to bear the tedius aspect of this chore). See, the LEAD PROGRAMMER, aka the guy who is SOLVING these bugs, may have very little skills when it come to classifying TEXTS entry, see if you talk to these guys in 01000100011100 they are happy, but if you start using grey, kinda black, not really white, you get them confused. And most bugs report, they tend to be messy to decipher since we all have our way of writing...time consuming. The LEAD PROGRAMMER is not someone you want to put his time sorting TEXTS entry, but is the company ready to put someone who has some programming skills(not much) and heavy reading skills as the first person sorting bugs? Most companies are not ready to do so and it just fall on the shoulders of the programmers, who saddly are usually not the ideal candidates to go through extensive texts entries.
Finding the bugs is the first step. Sorting them is the second step and if it is not planified, it is going to be done by programmers. Finally, solving bugs is the final step and they might even involve talking between the programmers and the designers before been resolved.
Keeping it simple is good. But when you have 934734048 betas testers, it ain't going to be simple, you prolly should have the right guy to classified these texts entry, and not everyone can do a fine job at that and enjoy it....which is also part of the reason why it often fall on the shoulders of the programmers, even if it shouldn't. Ideally, the programmers should received filtred feedbacks, either from the designers, from the marketing team, from the PR, whoever has the TIME and strong reading skills. The guys correcting bugs may have access to the original reports, but their work should be based on what come from other peoples. See, bugs solvers don't need to hear me rant about raiding, they have a job to do...so bugs reports should be filtered for them, rewritten if necessary but most likely just copy/paste some parts.
1 feedback from a beta tester may be completely useless, it may be about 1 bug, it may be about 2 bugs, it may be about 5 bugs! If you have rushed staff, they are not likely to see the whole of the message. Each feedback should be short, concise and straight to the point...but often these persons who find the most important bugs, they don't have the skill or the will to put it so sweetly divided...so having someone sorting the feedbacks would be a huge improvement during beta and bugs hunting. (not to forget that the person sorting these bugs, if that person is also beta testing the game, may see other bugs that where not reported, that person has a broader view of the bugs found).
- "If I understand you well, you are telling me until next time. " - Ren
I agree that BETA testing has lost the "testing" part. I remember being part of a couple game BETA's years back, and one day you would be a level 2 killing in one area, and the next hour or day you would be level 55 in another area, because the designers needed that area tested out, and then the next day you would be another different level in another different are testing what they needed tested. Today people sign up and get to play the game..and every now and then are asked for their opinion and thats it. I think that is one key reason why you see so many bugs in games today...
Just my $.02
I would like to point out you also got balancing to do in beta
With in house staff you will not catch tricks combos and loop holes players will find to either lv or become unbalancingly powerful
Better to adjust the classes in beta then have to nerf a class after release
Frankly, I believe the old adage: “You get what you pay for.” While it might sound nice in theory to get a bunch of “free” beta testers, I highly doubt that the vast majority of them are truly beneficial to the beta process.
Don’t get me wrong, I know people who have taken betas very seriously and actually “worked” to find bugs and exploit content so they could report their findings to better the final game. However, most people I know see beta as either A) personal bragging rights or
a way to get a “leg up” on others. The latter concept seems, in my opinion, to be becoming worse. I knew of one guild that eagerly applied to betas in order to find exploits and then “not” report them so as to give their own guild a way to cheat come official release.
~Synexis
yeah and then you have the botters and item sellers getting in and ploting wher items are and bug that they dont report for later exploitation and also test of programs they should not be using
but then any time they have an open beta its going to have the games biggest fans the ones waiting for years to play and then the next in line are the bad guys and then any one else who stumbled across it
The average player will not have the experience or the foresight necessary to create a well designed game. He/She will complain about the experience as it pertains to them or their chosen profession/class/race. Their suggestions and demands will change with the seasons and as they become better players they will require more challenging material, but not until they are ready for it. With literally millions of players now, everyone will be complaining about the game because each aspect doesn't perfectly fit their current wants.
Maybe what the MMORPG develoeprs need is an impartial board that could certify the expertise of beta-tester applicants might be a good idea. They could compile information on players of previous betas and let the developers send feedback as to that players usefulness and the structure, intelligence and overall efficacy of that player's remarks and ideas. Then you would receive a handle from this board and would use that when applying for beta spots. Then companies could have real beta tests with real suggestions that allow for a strengthened game experience, not a tug-of-war in the community or between players and devs.
The idea of letting pre-pay customer's in as beta-members is a horrible idea and an eggregious error. I don't want to play a game that was beta-tested by kids who were basically just playing the game early, giving no feedback on character balance, world design, quest structure or bugs/exploits. That'd be like if automobile companies sent out prep-cars to 16-year olds who just got their license. What good does it do to receive feedback from someone who has no experience to weigh the subject matter against?
I also think that open-betas, or stress-tests, should be handled by the company if they are interested in ridding themselves of the undersirable characters that weasle their way into betas. Betas aren't preview events. GW did preview events and they were fantastic, they helped showcase the game and they were designed to show-off the gameplay experience, not as a tool for gathering information to help design the game.
If you ask me, its not really a dabate here. Pretty simple if you break it down logically. Real and true "testing" of a game, finding bugs and fixing playablity of a game, is really a job. If you want it done correctly you are going to pay people that know what they are doing to come in and take care of business. If you think that you are going to just pick up anyone off the street and expect that they are going to do all the work for you because they feel priveledged that they get to play an unfinished and buggy game for free then you don't live in reality. The author of this article is just plain crazy to think that the general public should be loyal to a game because the creators of the game allow them in early. Do you think the creators are going to be loyal to them in return? Come on, this is business and, as the old saying goes, business is business. If game creators really want someone that is going to be dedicated to finding all the bugs and come up with ideas to improve it then they need to go out and pay someone that knows how to do it properly. If you think your going to get someone to do quality work because they get to do it for free then you obviously need to join us in the real world, Dan Fortier.
For a long time, before MMORPGs became "hot", there were quite a few people interested in helping to shape the experience. While there are dedicated testers and "bug-finders", the scope of most MMORPGs creates the need for more than just standard testing.
I think it boils down to this; MMORPG developers actually need to work on forward thinking. Both their games and their development strategies need serious rethinking. This is a genre that has subsisted, until WoW came along, on a small crowd. Now that they're popular, developers are still using the old fashioned methods, but trying to attract a new audience.