Do you create game rules or start prototyping first?

If your question is on PROCESS, you sort of do both at the same time, since you cant plan for everything your players can do with your system. These problems are found out from playtesting

That is why a game has several version. Each version is changed based on feedback and observations from playtest sessions.

There is no such thing as 1 prototype, 1 playtest, 1 revision, final product. If it was that easy there would be far more products in the market

But if your question is on HOW TO START, you need rules to play. It's a must, if not your players have nothing to work with. These rules however have to be to the best of your solo ability to make sense of the workings of your game. It allows your playtesters to get a feel of what your game is about, how its played, and the experience you are trying to convey or achieve.

You have to playtest your own game before asking others to try it out. Usually you would ask ppl closer in your social circle to try it out. if feedback is positive there, you move on to outer rings of your social circle, eventually reaching public playtesting. If the public playtesting is positive, you basically have a potential product to sell.

I take a pretty long time to get to actually writing out rules, like in paragraphs and everything. I scribble out scattered rules here and there in a sketchbook as I come up with them, then change them so frequently in the beginning that which version i'm using at the moment is just in my head. If a game calls for it i'll make little quick guides for players to see what actions they have available, and list turn phases.

However, I test on my own quite a bit before getting other players involved. Having something written out may be useful when teaching other players the game, so right now I'm considering getting started on some written rules before bringing a few people in to test a new game.

I'm very much a rules first person, usually I'll have an idea for a mechanic and theme (it can be either) that I think would be interesting, and then I'll build up the outline of the rules, working through them in my head. When I've got something that feels reasonably complete I'll write out a full rulebook (as much as possible). I like to have this done before I start working on the prototype as it allows me to make notes and update rules on the fly more easily. This rulebook will usually go through a bunch of revisions, get covered in red pen and crossings out, but I think I'd struggle to remember everything without it.

When I do get into prototyping I usually get over-excited and spend far too much effort on it, so another benefit for me of working out the majority of the rules first is that I don't end up wasting too much of what I actually make for the first model. I like good-looking prototypes as I want to be as excited about the material side of the game as I am about the rules.

When I've got something that looks pretty decent I'll run through it a couple of times by myself just to see if there are any glaring holes, but I don't often find solo play testing to be very helpful. I'd rather spend the effort on a presentable prototype and get it out into real testing.

I write down a rule once I am satisfied with the outcome during the prototypes test play. But I mostly do test plays first.

Of course, in the end I need to place the rules in a good order. And scrap the obvious.

And my prototypes are only white paper with text or numbers on them. For test plays with other people I make proper prototypes.