QA Design Bug Standards

The current QA Design lead on the PRC team is gone for a week to take care of some Dragon Age stuff at Gencon, so for this week I’ve been put in charge of QA Design on PRC.  It’s a big week too because we got 2 new term testers, and it’s my job to train them, delegate tasks, etc…  I took it on myself to write a training document for them, and passed it on to the higher ups hoping they use it for future hires.  There’s no company secrets here, so I figured I share it, since I’m rather proud:

---

This document details the bug submitting standards for QA when dealing with issues of Design specifically. If you are new to the position, or just in need of a refresher, please read this over, and maybe print the bullet points for reference.

Your job:

QA Design’s job is to play through the content with an eye on design. Design “bugs” are not as simple as something working or not (though don’t ignore those bugs either!), but tend to be much more subjective. As the player, how do you find yourself experiencing the game? Monitor your own emotions when playing the game. When you catch negative emotions arising such as frustration or boredom, or suspect your reaction to the content is not what the designer intended, such as finding something cheesy, or disliking a character you’re supposed to identify with, ask yourself why you feel this way. What could have been done differently to create a more positive experience, one the designer was more likely to have been aiming for?

QA Design’s job is not to re-design a level, it is to help the designer accomplish his or her vision for the content. This content is the result of many hours of hard work, late nights and much brain-storming, and as such it is important to remember and empathize that these projects are often their babies. When making a suggestion to a designer about changing some portion of their content the goal is to be working with them, together, trying to achieve the best possible experience for the player. You are not looking to criticize, but help the designer find opportunities to improve their creation. There are times you will have make your case, which is where knowledge on topics such as narrative, cinematography, story structure, even psychology, all come in handy. The more you can familiarize yourself with the many applications of design, the more opportunities you’ll recognize for further improvement, the more you’ll be able to emulate the role of the player, and you’ll have that knowledge to help you make your case to the designer. It is important to keep all this in mind when submitting QA Design bugs.

Common issues to look for:

•  Guidance: How does the level layout guide the player? Is it clear what your goal is and where you’re supposed to go next? Does it feel fluent, fun and understandable? Does it make sense and mesh well with the story? Is it confusing or easy to get lost? Do you know WHY your character is going to their next destination?
• Level Art: Are levels pleasing to the eye while still being effective? Do the transitions and settings make sense? Are areas too crowded or too barren? Too dark? Too repetitive? Are several levels too similar?
•  Combat: Is it not challenging enough, or else too challenging? Are the combats fun and engaging? Could enemy placement be better? Does the loot feel balanced and rewarding? Are there too many fights, not enough? Are there too many similar fights? Do they need to be more strategic?
•  Emotional Impact: Do scenes have the intended emotional impact? Are there any distractions or unintended implications?
•  Dialogue: Are conversations too wordy? Do they explain coherently what they are meant to? Does the dialogue fit the character delivering it? Proper wording?
• Canon: Does everything belong? Does it fit? Is it understandable? Is anything out of place within the greater canon of the game’s universe? Always be aware of how cinematics, dialogue, etc, fit in with the overarching story.
•  Pacing: Do you find yourself getting worn out by too much of the same thing? Are fights too common, too spread out? Is there too long a stretch with nothing engaging? Has it been too long since a plot development? Are there so many plot twists it gets confusing? Is it repetitive (i.e. cutscene, fight, cutscene, fight…)?
•  Fun: Most importantly, is it fun? Fun is when all these pieces come together to be more than the sum of their parts. But sometimes it’s not achieved and all you get is a technically proficient level that just doesn’t feel fun.

A few pointers for filing well-written, non-confrontational bugs, and to help save on un-necessary extra work for both you and the designer:

• Always check for duplicates. Designers’ time are very valuable, and they don’t want to waste it dealing with bugs that they’ve already seen.
• Make your bug very clear. Design bugs are by nature subjective, and it’s your job to make your bug as objective and clear as possible, so the designer can get the gist of your suggestion at quick glance, and reproduce it easily.
• Repro steps! Do not ignore this field. Keep it in point form, and list it as though you were telling someone who’d never played the content before, because sometimes as the bug gets passed around, it will end up on someone who HASN’T had a chance to the play the content yet, and they need to be able to get to the issue at hand with as little wasted time as possible. Ex: “-Load included save game - Talk to NPC named Bob - Select option 2 - Notice reaction is not consistent with conversation”
• The more useful information you include, the better. Sometimes words alone are not enough to explain a situation. Whenever possible, include a screenshot. When the issue in question is hard to spot, use an image editor to highlight the issue.
• When possible, include a save-game. This helps keep the repro steps short, and saves the designer’s time by allowing him/her to jump straight to the issue.
• Use the title field wisely. Have the map abbreviation and a quick summary of the issue in the title. You want the designer to be able to be able to recognize it in their bug list at quick glance. Ex: “TRP – Path by caravan blocked”
• Don’t be confrontational. When submitting an opinionated bug, try to word it from the player’s perspective, and word it as a suggestion. Do not state it as fact and certainly not as a demand. You are looking to work WITH the designer for the best experience of their content. Ex: “Suggest changing Eastern layout of level, player may find the circular nature of the caves too confusing”.
• Be prepared to open a dialogue with the designers. Your bugs may be returned with comments added, asking further opinion, informing you of future plans which may cause you to revoke your suggestion, or sometimes just to explain why the suggestion was not implemented. When a topic is too lengthy for a comment post, talk to the designer! Getting to know the designer makes it easier for a continual open dialogue to form, makes it easier to understand the overall vision the designer has for the content, and to keep up to date on planned changes. You also don’t want to become a faceless QA member who’s only point of contact is constantly suggesting changes to their creation.
• File all bugs. You may have a conversation with a designer, or send a suggestion in an email and feel your voice has been heard. File a bug for every suggestion, for visibility. Bugs are tracked, and checked by more people than one designer, so it’s important to have all bugs documented and on file.
• Don’t be stubborn. It’s easy to get attached to an idea, but sometimes it’s not feasible to implement, and sometimes it’s just your own opinion. Choose your battles, and back up your opinions with objective research.