What kinds of questions do you ask playtesters?

Some you asked:
How long did it take to play?
What rules did you have to look up or not understand?

Also:
Please clearly write your name as you would like to see it in the rules.
Was there anything you felt was over/under powered?
Other suggestions for improvement...

Apologies for the wall of text. I tend to over-answer and over-explain myself (though with the best of intentions). I don't have a lot of experience getting feedback from play testers, but I have had several classes that have dealt with surveys and what makes them effective, as well as personal experience designing surveys and interpreting results, so here are my thoughts. Hopefully they're of some use - take them for what they're worth:

I think the key to a good survey is focus. More than anything, I'd suggest making a list of information you want to find out from the play testers and prioritize them. I'm sure you've got a catch all field for additional comments at the end so use that to your advantage. Don't be anxious about having overlooked something - responders will use that catch-all field to let you know if you have. In fact, I find I'm actually more likely to write something in the catch-all field at the end if the survey is made up of specific, detailed questions than ones that are too broad or open ended.

Getting a good indication of people's true gut reactions is crucial. I'd ask testers something like, "On a scale of 1 to 5, 5 being best, how would you rate the quality of your experience with this game overall?" before I asked them any thing else, because you want that initial gut reaction. If you ask people to analyse/justify their initial reaction, they often over-analyse things and actually give less accurate responses. If you're doing an electronic survey, I'd consider going as far as to asking this on a separate page so that people can't second guess themselves and go back to change it. (You can find an interesting study on this phenomenon here, if you're interested - if you can't access the full write up, the abstract has all you really need: http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1991-17498-001 )

How easy a survey is to respond to will dictate how many (and how complete of) responses you get more than anything else. Targeted, specific questions are actually easier for people to answer than broad, general ones. Having people rate things on a scale from 1 to 5 (with 5 being best) is a great way to do things because people are more likely to give you an accurate reflection of their first impression. It's much easier to rate the speed of setup, how smooth the turn progression went, etc, than it is to answer a question like "Can you think of anything that slowed down the game or phases of the game that could be more streamlined?" For questions that don't lend themselves to a rating scale, people should ideally be able to answer with one or two words.

Also, don't fall into the trap of trying to make the responders do you job for you. While you want to ask for suggestions (in the catch all and perhaps about specific elements), trying to identify potential problems should take priority. Questions like, "How could this game be made better?," are hard to answer, so most people will leave them blank. And while it's worth asking just in case someone does come up with a great suggestion, most of the ones that you do get aren't that useful. You're the expert on game design (or at the very least, the expert on your game), and your play testers usually aren't. Play testers are experts in an area you aren't: their experiences playing your game. If you construct a survey that centers on your play testers' expertise, you'll get a more accurate picture of they're experiences and you'll be better able apply your expertise (tweaking your game so that other people have a better experience playing it). Like I said, there's no harm in asking for suggestions - you'll just get better results realizing your (and your play testers') areas of expertise and playing to them as much as possible.

Here are some questions I thought of that might yield useful responses (some are open ended, some lend themselves to a rating scale, some could go both ways, depending on how they're phrased). They just came off the top of my head, so this list is by no means definitive (or even balanced), but hopefully there's something in here that will be of some help:

• About how long did it take to set up the game?
• How difficult was the setup? (scale: "It went smoothly" to "It was complicated and/or confusing")
• How does this game compare to other games you enjoy? How is it different? How would those similarities and differences affect your likelihood, either positively or negatively of buying this game? (Could be open ended, could have possible choices along the lines of: Positively, because this shares elements of games I already enjoy; Negatively, because it is too similar to games I already own; Positively, because I do not own many games like this; Negatively, because this game does not have much in common with games I already enjoy; Neutral)
• Rate the clarity of the instructions (could even get more specific and have them rate clarity of instructions for setup, player turns, global events, etc). Might also phrase as, "How clear were the explanations for the rules of the game?" and "Were you able to navigate the rule book easily/Was the rule book well organized?" (since I've played many games where the rules themselves were easy to understand, but were presented in a confusing manner)
• About how many turns did it take for you to "get into" the flow of the game?
• Were there any phases of the game or elements of play which stood out as engaging? Were there any which stood out as boring or which took too long? (Similar to obligatory "What part of the game did you like best/least?," but might elicit different response from more experienced gamers, especially for something like a tabletop wargame)
• Is this a game you see yourself playing frequently? If not, why? (possible response options: It's not my type of game, it took longer to play than I usually have time for, needed too much space to set up, etc)
• Would you pay [projected list price of $x] for this game and purchase it either for yourself or as a gift for someone else? If not, why? (perhaps with options: Not a game I would normally buy; I would be interested in that game, but would not consider buying it unless it was on sale; I like this type of game but there are similar games available for a better price; etc)
• Were players able to devise a "best" strategy and use it repeatedly? What was that strategy? How might the game be better balanced so that other strategies were more viable?
• How quickly were players able to make decisions? Did the game progress smoothly from turn to turn (or phase to phase), or where there portions which routinely acted as speed bumps?
• Did you feel confined in the number of choices you had when making decisions? Were you overwhelmed by too many choices?
• Did you end up adopting and "house rules"? If so, why? (i.e. lack of balance, unclear rules, desire for more/fewer options, etc)
• Did the game feel like it was missing a crucial element? Were there elements that felt extraneous?
• How many times did you play the game? (If more than once) Was the game more enjoyable, less enjoyable or equally enjoyable during subsequent plays?

I like to bluntly ask 'what didn't you like about this game', or 'what didn't feel right'.

Go into the conversation with the attitude that something must be wrong, and you want them to tell you what it was - that way they're more likely to think of something than they would be if you ask whether anything was wrong.

I feel like such a dork!

I've just been giving my testers a piece of paper labeled "Notes and Observations" and letting them put what they feel. This has worked well for finding the hitches in rules and such, but after seeing this i'm so excited to give a much more directed feedback form.

excuse me while i go make my form...

Ask is the key here; Questionnaires can ask about things you already know are a problem or might be a problem. Problems tend to crop up in unseen ways. I prefer to interview the players so that my questioning can be guided by the answers I get.

I start with something like. "What didn't you like about X"

Just a note to say that I cross posted some of your excellent responses to this question at the dev forums in bgg. I figured that others could benefit from the super great words :)

http://boardgamegeek.com/article/9500936#9500936

I recently needed to put together a survey for blind playtests of my new game. Of course, I couldn't put everything on the survery that I wanted to ask but I followed a lot of the ideas in the thread.

The Abbey - Playtest Questionnaire.pdf

One more comment on the price question... I actually find that this is a good way to get more game feedback than you might initially think. If playtesters respond with a lower price point that you were going for, push them on why. I've had folks tell me that the game was too simple and/or not engaging enough and thus they put a lower price tag on it.

In one way, it makes no logical sense... you could make a tic-tac-toe set with diamond 'x's and pure gold 'o's. It won't be a cheap game, but it is simple. But you've got to look past that and engage with their actual reasoning as to the price point.

You could use or modify the one found on unpub.net as well. Good luck!

The boring parts.

I often like to ask if there are parts that players find to predictable.

"Which parts do you find predictable or boring?"

These parts often are the first boring parts of a game, indicating a decline in the fun that players have.
A second question following this one would be if they think it was because of their opponent(s) or the game itself. After all, it might be a player that immidiately goes "systematic".

"Was this predictability/boring part due to the game? Or was it your opponent(s)?"

Then I also know, if the opponent went automatic, if the game is obvious balanced or not.

To conclude, I can ask the opponent(s) why they did the same thing over and over.

Well, I have many more. But they are not general questions. More in dept questions.

PS.
Good thread.