Common Playtesting Myths

I am fanatic about testing my games, and for the past 6 months, I’ve run 12 - 25 tests in per month (not including playing solo or with co-designers on projects). My background is in design and design research and I run user studies for a major retailer. User Experience design (UX) all about analyzing behaviors and using data from studies to drive change, and techniques from UX research have been incredibly valuable in my playtesting and design process for tabletop games as well. I wanted to start my posts on board game design with a series on how to improve the quality and quantity of your playtests. Would love to discuss

 the finer points of playtesting if people are interested.

Suggestions for more myths? Add em in the thread!

Myth #1:
"Start testing with your family and friends, then strangers."
You see this often listed as a first step in posts about playtesting, and there are many successful designers who got designs published starting this way. But you’ll notice, almost every time you see the advice to start testing with friends, all of them follow up with: “but then, make sure you get your game tested by people who don’t know you.” I think most people reach for their family and friends to test because that is what is easy/familiar, not because it’s necessarily the fastest or best way to get a game tested to completion. My view is you should jump straight to testing with strangers. I’m not saying NEVER show your game to to family and friends, just that their feedback isn’t as useful.

1. Your family and friends are unlikely to be honest with you.
   You’ve probably heard this before, but no matter how many times you tell them that you want honest feedback, they will be biased and try to avoid hurting your feelings.
   1b: “My friends aren’t like that, we’re all mean to each-other.”
   It doesn’t matter. Everyone has subconscious biases, and people that know you will have a harder time separating a critique of the game from a critique of you. Just because your friends are more honest than most (or you perceive them to be) doesn’t mean that their feedback is as unbiased as a stranger’s.
2. Your social and family circle is too closed.
   Even if everyone is different, the fact that you are friends and family means you likely: come from similar backgrounds, are more likely to share political and social views, are likely to have had similar levels of education, are likely to have the same primary language, and many more hidden factors. In short: even if you have a diverse group of friends and family, you think more alike and have more in common than you and a stranger. Your friends and family are less likely to catch things during testing that you haven’t thought of.
3. Testing isn’t a central part of your relationship, so you are changing the nature of your social dynamic.
   Let’s say I loved singer-songwriter music, and I have a friend who’s a really good musician. I’ll go to her shows. and if she invites me over to listen to a preview of her new album and give her feedback, I’d do it. But if EVERY time I came over, she wanted to play her album, I’d start to get annoyed.

Myth #2
“You shouldn’t make your prototype look nice”

This is very common design advice in the board game community, but is only true to a point. Spending a small amount of extra time (for me, this is typically ~2hrs per iteration, maybe a little less) to make your prototype look better is worth it. Now I’m not saying go out and commission artwork, or spend an extra 100 hours designing the perfect custom font for your card titles, but it is worth the time to go slightly beyond bare bones and make sure your prototype is clean and professional. Much of the job of game components is communicating information, and your prototype needs to do that as effectively as possible. Learning graphic design makes you able to communicate information more clearly. So do it.

1. Having clear card, board, and other component layouts decreases teaching time and confusion during the game.
   How players feel about the experience of a game is connected tightly with it’s pacing and narrative arc, and stopping to parse or interpret information during gameplay messes with that pacing. If your players need to spend an extra 10 min per game reading your handwritten cards or parsing the confusing chart you’ve laid out on your game board, you’ve not only lengthened your testing time, but you have negatively impacted their play experience.
2. Having a clean looking prototype attracts testers to your game.
   People want to pick up and play games that feel nice. My next post will be about recruiting play testers. This is way easier if you can show a nice looking prototype. It doesn’t need to look like a final product. It just needs to show clear effort and professionalism.
3. The key is being able to do this quickly.
   You don’t want to get bogged down making your prototype pixel perfect, but it’s worth cleaning it up. Learning fundamental graphic design and layout principles is worth the time investment, because they are foundational skills you can apply across all of your prototypes, rulebooks, sell sheets, publisher pitches, email templates, etc.

Myth #3:
Playtesting is about making a series of small iterations.

This one is half right: because the design process is all about iteration. My issue is people often approach their game testing as though they are conducting a quantitative study, and each small change represents an experiment against a control of the previous version. I think it’s more helpful to approach playtesting using qualitative research techniques, using technique from usability studies or ethnographic research to base playtest structures on.

Realistically, you will never have enough tests to statistically finish your game using a “scientific” A/B testing method. It’s better to make lots of changes at each iteration during earlier parts of the design process to see how the core experience and feeling changes, then make smaller incremental changes toward the end of testing. 

The majority of the balancing of your game should likely be mathematical + a little bit of manual tweaking. If you just essentially use “trial and error” over and over, you don’t have enough real data to conclude your game is balanced.

That being said:

Myth #4
Your game needs to be balanced


There are many different definitions of balance but for simplicity’s sake, let’s say a game is balanced when players of equal skill stand a statistically similar chance of winning. If a game has multiple core strategies or paths to victory, balancing between those strategies means that each on of those paths stands a statistically similar chance of winning if executed well.

Your game doesn’t need to be balanced to be fun. In fact, the PERCEPTION of balance/fairness is more important than whether your game is actually balanced. Players will continue to have fun as long as they believe that they have a fair chance. For example: let’s say you have an asymmetrical game. You’ve extensively balanced the skills and abilities of the three factions so they all stand an equal chance of winning. However, the counterplay from Faction C that allows them to stand up to faction A is not intuitive, and requires a few playthroughs of the game to understand. If the first 4 times a player plays your game, faction C loses, they’ll conclude the game isn’t balanced and shelve it. Similar to this are games where there are prescribed opening moves. If people take said “standard” moves, then the endgame is balanced, but if anyone deviates, they essentially hand the victory to another player. Here, your balance is getting in the way of players having fun and meaningfully participating during the duration of the game.

Some designer/publishers don’t consider balance while learning the game a real issue, and focus their development only on expert players who can really plumb the depths of tactics. Ignacy Trzewiczek for example, tells a story in his book about an army for Neuroshima Hex that has absurd win-rates (Steel Police) until people learn to play against it, at which point he concludes that it is in fact balanced. That was expansion content, but again I think you have a problem if it requires weeks of playing constantly for players to achieve the level of knowledge to make the game balanced. In my view this represents a flawed design.

Final Thoughts
Playtesting is an incredibly important process. Hopefully, some of the lessons I've listed here help you in getting your games testing and getting the most out of those tests.

I think I'll likely be doing some more in this series:
Recruiting Playtesters
How to run Playtests like a Researcher
Getting Better Playtest Feedback
Interpreting Playtest Feedback like a Researcher