GDS Scoring System 'flaw'

I think that participation in the GDS presupposes a certain level of maturity on the part of the participants such that actions such as self-voting, refraining from voting so as not to "help the competition", and voting on weaker entries simply to dilute support for "your main competitors", would be recognized to be in poor form and against the spirit of the competition. Anyone who needs to resort to these tactics to win simply for the right to put a GDS win in their mental trophy case needs to get a life, big time.

Having said that, there is one legitimate concern with voting that I've seen expressed recurrently, and it's that people are often torn between voting for what they believe to be the most promising game and the one that conforms the best to the requirements of the competition. It may be helpful to come up with an "official" ruling on which of these represents the "right" way to vote, or else perhaps have two votes, for "best design" and "best solution to the challenge".

If we had to pick one way to vote (best game or best challenge solution), I'd advocate the former, actually. The contest requirements are useful as guideposts, but it's very difficult to actually conform to all of the restrictions in each challenge, and I strongly suspect that any of the entries that see further development will shed in an eyeblink those elements that were tacked on simply to meet the challenge. To my mind, it makes sense to vote for the best game, as long as it doesn't completely flaunt the contest requirements. But I understand the opposite view and think it has merit as well. Perhaps we could take a poll as to a preferred way to vote, and then make this the law of the land. Or just leave it as the free-for-all that it is now.

-J

I don't mind leaving the GDS unchanged. There's nothing important at stake here, and I don't think we need excessive formality in the rules.

But if we decide we do need a change, then I think Mike (Hamumu) makes good sense, in his post just above this one. If you're counting opinions, I agree with Mike.

Perhaps have two requirements for voting:

A) In order to win you must vote (as already stated).
B) You can't vote for your own submission.

Regarding A, it's not really fair to throw out all the votes for an entry just because the designer didn't vote. In that case, I think it would be best for the moderator to gently prod the entrant to vote.

I know a lot of people, because of their maturity, won't vote for their own entry anyway. However, some people honestly will think their entry is the best or that may just want to 'bump up' their chances of winning, whether subconsciously or on purpose. If it comes down to two close entries, and one person voted for themself and beat the person (or even tied for that matter) who didn't vote for themself, then obviously the system if flawed. I think your design will have to be obviously good if you win and and all the votes you received are from your peers.

Just my dipence,
-Darke

1 - Zom are you in this months contest?

2 - Its not "stupid"; the voting is not a game its a job. I often fall in to the same trap. I want to make everything a game. but voting is not a game the game ends when you turn your subbmission then you just have the job of voting. As Matt said we are all mature and we know what the spirit of the job is to try and pick the best which leads me to #3

3 - For the first time I disagree with Jeff. If we are picking the best game and not the best that fits the requiments then why do we have the requirements? The "Charm" of the GDS are the requiments. The whole rest of the fourm is dedicated to everyone trying to make the best game they can. What I mean is we are already doing that out there. In my oppinion its FUN to try and come with a game that meets the requirements.

4 - We are all infatuated with rules or we wouldnt be here. Each one of us could come up with ways to vote if we had to. Its what we love to do. THat doesnt mean that what we are doing is flawed.

5 - I feel that if a entrant doesnt vote then we should asume that they had a good reason and move on. I agree with Jeff that if they didnt vote on purpose then they need to get a life but I have a hard time beleiveing that that would ever be the case here.

ok I am done with my rant.

- Dwight
THE CURENT GDS CHAMPION! :P

Yup...see first post.

Sorry. I thought you told me in live chat that you were not going to enter. I misunderstood.

My whole rant was an atempt to argue that The GDS is fine although we have this same disscussion about the GDS each month we find it to be good each time and that maybe we should not take it too seriously. I meant no hard feelings. I beleive you want the best for the comunity.

At first glance, during my first GDS about 4 months ago, I thoguht it was a flaw too. Then after the disscussion that month I realized it was not, that it was just on our honor to vote honestly.

- Dwight

I'll start with a quick reminder that the voting for this current GDS Challenge (December 2005) will use the current system -- each voter gives me their top 3 games through a PM. And I'd like to encourage everyone to be to please submit a vote for this current Challenge. ;-)

With that said, I've seen a few things in these posts that have inspired the following idea for a new voting system ... It would consist of 2 parts -- a "gameplay rank" and a "challenge score" ...

Gameplay Rank
There would be 3 categories for this:

• Best of the Best - At least 1 and not more than 2 games that the voter felt were the very best games submitted based purely on the quality of the perceived gameplay.
• Close Second - From 0 to 2 games that were close to being the best-of-the-best, but just missed the cut. (I think having 0 games in this category would be rare, but might occur if the voter thought there was a huge drop-off after the best 1 or 2.)
• Worth a Mention - Any number of games that should at least get a mention.

Challenge Score
For the top 2 categories ("Best of the Best" and "Close Second" -- this would range from 1 to 4 games), the voter would give a score as to how well each game met the Challenge criteria using the following scale:

• 3 - Met all Challenge critieria well
• 2 - Met most Challenge criteria ... shakey in a few spots
• 1 - Didn't do so well with the Challenge critieria

Scoring
I would give points to each game as follows:

• Best of the Best = Challenge Score + 2
• Close Second = Challenge Score
• Worth a Mention = 1 point

Best total score would still win.

Self-Voting
Entrants are encouraged to list their game in one of the 3 categories. In whichever category they list their game, they must also list at least 1 other game with the same or better Challenge Score. This means an entrant can give their own game a "Best of the Best" Challenge Score of 3, as long as they give the same to another game too.

Sample Vote Message
This might seem like a lot of extra work for the voters on first glance, but I don't think it is. I could see the vote PM to me looking like this:

Most people have been submitting comments and rationale for their votes ... sometimes several paragraphs. So this additional scoring granularity shouldn't be too bad of an extra burden.

So ... whadayathink?

-Bryk

[/][/][/]

I think this is a clever hybrid system that addresses nicely the concerns I've seen expressed. Nice job! But...

I can't emphasize enough what a bad idea I think this is. In a sense, the contest is already "unfair" since entrants are also judges, but allowing players to vote for their own games just pushes it right over the edge. People simply cannot evaluate their own work objectively, and the outcome of the contest will be tainted if this is allowed.

-Jeff

I'm agreeing with Jeff here. Very clever hybrid system, but allowing people to vote for themselves isn't a good idea. Some people may even feel obligated to vote for themselves. Just don't allow votes for your own entry.

When folks vote, just have them make sure to list their own entry at the top, that way you don't have to cross reference your list of entries.

-Darke

It is impossible to objectively voted for your own game. While every other game has a rules set that you are being introduced to for the first time with an 800-word limit to get the idea across, your game has had effectively, an inifinite amount of "words" (the concept in your head), that has been boiled down to the 800. You know exactly what your game is supposed to be, while the others have a limited description.

I see no reason why the current format needs to change; it's simple, and it's really more of an exercise than a contest anyway.

But if you need to change, I wouldn't mind something like you have 15 points to spread around any which way you want, so if there are three top games you like equally, you could give each 5/5/5, or if you have a single game you really like but are having trouble deciding between two games for a third place vote, then you could vote 7/4/2/2, etc.

Hey. Speak for yourself! I'm sore every month!!!! (stupid judges!)

That's a good point, Mike -- but our current system also has this problem because all scores are posted for all entries. I remember a couple months ago that someone received absolutely no votes at all. Ouch! On the other hand, I like the fact that all scores are posted so I can see where I fell -- even though I'm usually in the bottom 1/3 of the heap.

On a different subject (from a different author), the whole "if it's worth keeping it's worth fixing" sentiment strikes me as a bit off-the-mark, because it implies what we're using is broken. Whether it's broken or not stems, I think, from your expectations as an entrant. As I understand it, the current system forces you to be "stupid" (not vote for yourself, thereby decreasing your chances for winning) or "selfish" (vote for yourself, thereby possibly depriving another game of equal merit from winning). But these alternatives are only relevant if you're playing to win, no?

I think most of us play for the challenge and the possibility of creating designs we wouldn't normally produce. It's also great to get some peer review. If these are the reasons we play, the current system provides a decent "relative rating" scheme -- especially since Bryk posts a summary of each entry's total votes. Anyone can look at his design and say, "Of the 14 other voters, 2 thought mine was 3rd best and 1 thought it was second best. Not bad!" (Unless you're Yogurt, of course, in which case you usually get to say, "Oh, 80% of the people thought my design was the best. Wow, I need to do better next month."). Those results don't change if you vote for yourself because you would automatically disregard your own vote from your tally (i.e., you know how you voted -- only other people's opinions matter).

I think Bryk's hybrid system has some neat features. I especially appreciate being able to judge on two axes (best overall design and best challenge entry). I think it's more complex than we need -- especially if it leads to more work for him.

Finally, I agree with Jeff that self-voting is just a bad idea, period. Even if one could be emotionally objective about one's own entry, the fact remains that we all have much more information about our own entry than the others.

K.

Which brings me back to, "If it ain't broke, don't fix it."

Yeah, there are a lot of different ones. The reason I like assigning default comparisons is because then everyone's input carries the same weight, namely nC2 pairwise value judgments, instead of, say, "A>B" vs. "here's a total ordering".

Perhaps it'll sound better if I reverse the polarity of the neutron flow:
"If it's worth preserving, it's worth optimizing."

(And for those who missed the no-voting-for-yourself rule already in effect, 'selfish' is not voting at all, rather than voting for yourself.)

Hey Zom,

I don't think you really offended anyone. The phraseology of the original post did come on a bit strong, but as you're re-expressed it, I think your intent is much more clear. Darn text media and its lack of nuanced communication. :)

I just tried to summarize our situation, then realized that it was just a rehash of previous debates. The GDS has been down this road before. That's how we ended up with the system we have. If the main problem is "to vote or not to vote", the community has already spoken: you're on your honor to vote if you enter, and you are not allowed to vote for yourself.

My guess is that you are more concerned with giving entrants the best possible feedback on their designs. This begs the question, "What kind of feedback do people want?" and "how much time are they willing to put into judging via a new system?"

The current system represents a reasonable compromise between feedback received, time spent judging, and Bryk's time spent crunching the numbers.

You made several good points about the current system -- like "if we feel that it's not important, maybe we should focus on more important things," and "'Don't take it so seriously' sounds more like an argument to consider the results meaningless."

Fair enough. So -- outside of the "to vote or not to vote" question, which most people feel is now a non-issue -- where does the current system break down and how could we improve it?

So far, there seem to be two answers to this:
1) Not enough resolution to select between close designs.
2) Lack of ability to judge specific aspects of designs.

Can anyone add any more?

Once we agree on the direction in which to optimize, we can start talking about how to optimize.

K.