UIX and Focus Testing: How user feedback lets us make a better game
UIX testing + focus testing + meeting for bug triage
*Please note that some contents shown in the videos in this update are still subject to approval by Paizo
As we mentioned before, while developing interfaces we use our UX lab to get the most reliable feedback from the players. Today we want to tell you more about how we do it and what we get as a result.
Before we get the testing started, we need to decide what we want to learn from our tests - whether certain interfaces work as we planned, if players like to complete quests, or even how long they remain interested in playing within certain time frames.
After our goals are set, we create a draft script with the help of our colleagues from UX lab, which consists of the following parts:
1. Introduction – when the player gets acquainted with the game and receives instructions. Here, it’s extremely important for the player to realize that he must play exactly as he would normally play at home. In other words, he can help us only if he forgets he’s helping us. :)
2. Testing description.
3. Things we (Owlcats) should pay special attention to.
4. Questions that arise during testing.
5. Final interview.
The next step is selecting the respondents. Usually, for our testing we invite 6 to 9 participants who belong to one of three groups: those who are familiar with the tabletop version of Pathfinder, those who used to play other roleplaying board systems and those who never played any roleplaying board games. At the same time, all the respondents are crpg players and hence belong to our target audience. The number of respondents, that belong to each of these groups, depends on the goal of our testing. In case of our interface tests, we’re equally interested in opinions of representatives of all the groups mentioned above. If we are testing game mechanics, we want feedback from players familiar with RPG systems. And if we’re going to test the learning curve, we want comments from players unfamiliar with RPG systems. It’s simple logic – those who have the most difficulty figuring out what’s going on in the game are the most help to us.
After we have selected our respondents, testing begins. On average, each test takes three days, with three respondents per day. Each player needs approximately three hours to undergo the testing - introduction takes a half an hour, playing the game is two hours, and the last half hour is the final interview.
The testing itself looks like this:
• We meet the respondent volunteer and treat him or her to something yummy. :)
• We introduce the respondent to our UX lab specialist (a developer is always present during the testing, taking the opportunity to watch the respondent’s game in “live” mode).
• We give the respondent a quiet room to play in with no distractions – just you and the game. Oh, and a chair. :)
• We tell the respondent about the game, but do not mention a word about the goal of our testing (This is to keep the results unbiased.)
• We tune the equipment: eye tracking, cameras, microphones. Using these devices, we record everything that happens in the testing room and broadcast it live to the observation room: the player’s words, their facial expression, gaze direction, and so on.
• Then the testing starts. During the game, the player can share their experience in any form they see fit.
• After that comes final interview with the UX specialist who will ask questions about your gameplay experience.
During the testing, all the player’s comments are written down by both the developer and the lab specialist. Together with the video records, these notes will later be used to create a final report.
After the testing is complete, UX specialists compile all the data they’ve collected and make a summary report. Its form depends on the initial goals of the testing. In most cases, they just take the list of these goals and add their extended remarks, screenshots and thoughts on what’s working properly and what’s problematic.
Here is an example of such a report:
The final stage comes when the team reviews the reports, discusses new-found problems/inconveniences, and decides how to solve them.
UIX testing is an extremely useful tool, and it gives detailed feedback on how users perceive our game. But we realize that this method has a substantial drawback – the number of testers is too small, and this testing is extremely time-consuming. As a result, representation and statistical validity of the results we get can be questionable. That’s why we use UIX-testing only to detect some nuances and non-apparent bugs of usability in interfaces that have already passed several iterations of testing.
To collect more representative feedback data on our game, we use another classic approach – focus testing.
To perform focus testing, we invite the same categories of players we use for UIX-testing, but the numbers are bigger – 10-20 people in each category. It's important to note that in this case territoriality doesn’t matter much , so we can invite people from other countries to participate in our testing.
To perform the testing, we assign one developer to each player. It’s important to us that every game developer – and especially every game designer – take part in an iteration of focus testing at least once. Imagining how people would play your game is one thing, but seeing how they do it in real time – is something completely different.
The process of focus-testing includes these activities:
• Beforehand, we prepare computers with the game installed. We try to use various laptops, as examples of most potentially “inconvenient” hardware (monitor too small, keyboard not comfortable enough) :)
• We meet the respondent volunteer or reach them via Skype. If the testing is done via Skype, we ask the participant to adjust the camera so we can see their monitor and – ideally – their face (if there are two cameras available)
• We lock inside the special room where nothing would distract us from the game We do our best to provide a quiet game environment with no distractions.
• We tell the respondent about the game – briefly describing the events in the game up to the beginning of the test, explain how the controls work, and so on.
• Testing begins. The point of focus-testing is to avoid giving hints to the player. Instead, we observe their actions and notice what wasn’t clear enough, or what went wrong/not as it should. To make this easier, the player is invited to comment on their in-game actions in detail as they are playing.
• Final questions. Usually we ask the player to give us his general opinion on the game (and also, is the game interesting enough to buy it – the criteria of “parting with my one’s own money” is often the most effective appraisal of how compelling the game is ☺), describe the most annoying and the most pleasant elements.
All the remarks of participants are added to a special table, which looks like this (example):
During the last test we performed in July, we collected about 1.4K comments. After preliminary sorting and merging, this gave us about 600 suggestions and bugs for discussion. Once we have those, then it’s time for the most interesting activity - issue resolution brainstorming. Some of these discussions were extremely heated, and more yet to come!
The last focus-test triage resulted in about 340 new tasks.
Even though we discuss the game’s problems in these triage meetings, these discussions are very motivating for the team, because each decision improves our game, not in some abstract way, but for real people playing it.
In conclusion, we’d like to remind once again that we can’t wait to start a new, most representative test – our first Alpha test stage – and it starts this week. We hope we’ll gather massive feedback and lots of comments, for it’s important to us to understand how you like playing our game and what we should do to improve your experience!
Hail to the kings!
Owlcats.