Interpretation of statistic results with simulators

Since game balance issue's belong in the mechanic department. I place this topic here.

I am not going to blog this. I rather have it as a discussion. And I would like you to share your experience. Or ask questions that I might answer or take a look into if I did not think about it yet.

---

Simulators are only used when a game has many possible outcomes by luck or choice. And you test these to see if the possible strategies are balanced. And/or a game in a whole. Perhaps you even want to know the results when comparing 2 or 3 different mechanics that are roughly the same. Only to see which one provides the best results.

I am certainly not talking about if a game is fun or not. You can only find that out by play testing in the first place. And by now, you already should have done so, before even thinking about a simulator.

In this entire topic, I will talk about mechanics. But you also may read as statistic numbers like health or dice rolls or any other variable that you can think of.

When you know that simply play testing will not give you the numbers you want. That is when you need to decide on making a simulator or not.

A little time investment, might, safe time.
And it is this "might" that you need to consider.

These are the most important questions that popped up in my head during the making of a simulator.

Here is my experience thus far. Feel free to add.

***

Communication when you get help.

At first, you need to learn how to make a simulator. Well, let's just say, 90%? of game board designers simply don't know how to program. And that is ok. While I had some experience 10 years ago, time sure changes everything. So I simply asked help. And finally got it too.
When you seek help. You need to have a good communication. Best would be to explain your game first. Making sure that your helper understands the game. And every mechanic that is needed for the simulator. And above all, you want to trigger him/her to wanting to help.

***

Are you sure the simulator works correctly and sufficient?

Each and every mechanic that have influence on the outcomes of a game are surely needed. But it is important to know if you can program these or not. If you can put it down on paper, you can surely program it.

But what you need to put down on paper is not only the mechanics. Also, how a player would apply certain mechanics. In other words, what decision would they make? If a player has 2 or more options. You need to consider if each option is a gamble, or a logical option. It is best to test only the logical options and pretend you have a skilled player.

If a game is very complex. It will not do you any good when leaving out mechanics. Unless you are absolutely sure you are only to test games where only a few mechanics are tested. In fact, it "might" be wise to test only a few mechanics.

You can also build up a simulator by adding 1 mechanic at a time. Your results will automatically change too. Only for the better. But you don't know when it is good enough.
Eventually you might coma at a turning point where you need to change the simulation completely. This only because you know that a player acts completely different when a certain mechanic is applied.

***

Are the results any good? Do the results really provide the information that you need?

Well, you only find out by testing your simulator.

Noooo!!!
Was the first time.
So I tested a little 36 infantry vs 6 tanks. The result was completely useless. The infantry got obliterated, while in theory, this should be relatively balanced, where the infantry surely loose but occasionally would win. The result told me nothing at all.

Obviously I only knew how many times the infantry would win. That is 0 out of 10000 times. But how hard did they loose? My game tracks health. And this health might be close to 0 of the survivors. And that is the actual result that I needed.

So you need to add something to the results. In my case, every win requires the simulator to record the health and give an average in the end. I was happy with these results.

***

Is the information that you get accurate enough?

Obviously, a couple of tests are needed to see the inaccuracy that your randomness creates.

But there is more to this. When adding mechanic 1 by 1. During that process, you get more and more possible outcomes. What you now need to consider is the amount of iterations in your test. (Is that the correct word?) If a game has 2 possible outcomes. Is 2 iterations enough? By instinct, most of you will say. Obviously no!
But is 4 enough? 10? Perhaps 100?
You don't know. But there are 2 ways to find out.

One way is to calculate the iterations that you need.
For this you want a certain accuracy. I could write an entire page about this. But by simply googling, you can find many ways to reach your answer.

The second way is to discard calculating. You only do this when your simulator has grown complex enough. That you don't even know how to calculate any more. You simply conduct a test with a respectable amount of iterations. And you record the results. You do this at least 10 times. And you watch how inaccurate the results are to one another.
If your results are 10% inaccurate. And you want to bring this down to 1%. Simply multiply the number of iterations by 10.

***

And is the simulator really worthwhile your time?

By some point, you will ask yourself this question again.
You need to consider if you have combined enough mechanics that you want to test. Perhaps you need to separate mechanics again and have several simulators side by side.
You also need to consider the amount of iterations. While at first 10000 would take 2 seconds. By now it is 14 seconds. When you need to increase the number of iterations too for better accuracy. You simply need more time as well.
Eventually 1 simulation might take half an hour. And at this point you will probably stop.
Now you need to decide for a last time if the simulator is good enough or not.
In several area's, you can take back a step. But if you are not willing, you have reached a dead end.

On the other hand, if you do have your results. Congratulations! Now it is your time to test your game. And make the changes that you need.
But remember to change your simulator along with it.
Which will always provide the danger of being a waste of time.

In the end. You might decide on making your game more, or less, complex. The simulator grows alongside of it. Your simulator will also provide you with numbers. Which you decide upon what they have to be. But you can only change the mechanics for the input and simulation.

***

I hope this has given more insights for using simulators. I also hope for you to share experience if you have them. Or giving links to other topics regarding this.

Thank you for reading,
X3M