After an admittedly long absence from our allotted iDevBlogADay schedule, I would like to share my thoughts about the difficulties of beta testing. We have conducted a few beta tests in the past for iOS products and I am beginning to understand the underlying problems that probably many other dev teams are struggling with: beta tests are simply not effective enough. Yes, we have great tools to push test builds to prospective testers (like Testflight), so the technological hurdle is gone. The much bigger challenge is finding the testers who can give you real, useful input over a sustained period of time.
We are currently in the middle of the beta test of our latest, greatest game and here we have a unique case: we are in the process of doing a total makeover of our earlier game, Soctics. Fortunately for us, we have managed to involve a few fans of the original game to beta test for us. I admit this is not a common scenario for most developers so I won’t analyse this delicate case deeply, but very briefly the situation is:
- Upside: They are already familiar with the original game so they require less instructions for testing, and as they are already loyal to the game they are very enthusiastic
- Downside: Easy to alienate. As the users got used to the original feature set, there is resistance from their side when feature changes occur (opinion is often going to be biased)
As for why we are doing a total makeover for a game that has been on the store for more than 2 years, that’s a different story and it’s worth its own blog post. So it’s great for us, we have enthusiastic players who have been using our game for several months, some of them even for more than a year (how common is that in the iOS app ecosystem, I wonder?). But what about when you are creating a brand new game?
Your Developer test device quota is so precious because of the restricted number of devices you can register, you really have to think carefully how you spend them. Let’s say that you start blogging about your fancy new game, even posting some photos / teasers / early gameplay trailers. If you do it properly it will give some insight into what the game is really about and you will get some initial excited applicants who seem to be very excited about testing the game for you. We have done a beta application form for Fontify last year with Google Docs, it was fairly simple and easy to distribute, I can heartily recommend checking it out! So the next phase is picking the best candidates for your beta purposes based on the answers they gave you on the application form. Perfect! You’ve got the 5 / 10 people you were looking for, the testing can begin!
You share the first test build on TestFlight, everyone is madly grabbing it. Then you wait for the results. And then wait some more. Finally, a couple of feedbacks start trickling in. Some observations won’t be news to you, there will be some bugs you have overlooked.
You create a few more builds and you will exponentially get less and less feedback, it’s like the door of a treasure vault is closing before you are able to go in and grab the things you want.
In the end, you won’t get that much information from the testers. You can then go and get a second batch of people and try to squeeze some more data out of them and then you decide that OK, beta is finished, now lets release the game. But is your game really ready to go before the public? Is it really giving the best first impression it can give? Is it really well balanced, enjoyable, easy to use? Does your game simply suck? Maybe. It could also happen that you have not found the proper people to show it to.
How can we expect to optimise when we select a bunch of random people out of nowhere?
I think beta testing needs much more planning than most people realize.
- Distinguish several beta phases
- Specify explicitly what has to be tested in each phase (eg. Phase I: singleplayer core gameplay, Phase II: multitasking support and multiplayer issues, Phase III: additional gameplay elements (eg. more advanced units/buildings/, bigger maps, more difficult puzzles etc.)
- Find the best people for each phase: don’t accept just about anyone for beta testing. Most iPhone users will not even be aware of the true meaning of it. Testing != playing
- keep the beta intense, stick with a pace, keep new builds coming and give immediate feedback to every tester who sends you a bug report / improvement suggestion
Testing != playing. Keep that in mind when you are sorting through applicants who are accustomed to using one of the most refined electronic devices ever made. Average iOS users are not used to testing and trying to break things, sending bug reports and thinking about improvement possibilities because when they came out from the Apple Store with their new phone/tablet, they’ve bought a “magical device”.
I believe that separating phases and then allocating the best people for each phase is the way to go. How effective were your previous beta tests? Do you have any insights to this topic? Let us know!
nice discussion, I agree with the above. A few more thoughtful tips
- Install analytics like Flurry in your app, even if just for beta testing. Track as much info automatically so you don’t rely only on posted feedback. Some users are quiet, but you want to know how they scored/progressed – this is how.
- I’ve had great success recruiting from TouchArcade.com for games. If you are having trouble finding volunteers, try there.
- Offer a name in the credits to those who submit written feedback (if that’s your style). It’s more incentive for users to openly express their POV.