apterous.org

Conundrum Difficulty System - How It Works

Recently we have received many complaints that the conundrums are too difficult. This is a problem for us because we want to cater to players of all standards, which means setting potentially obscure or very hard words for top players, while also making the game interesting and competetive for less experienced players. After some thought, and some valuable input from Ben Hunter, here's how our new difficulty balancing system works.

First of all, conundrums are graded for hardness, from 1 (talc - very soft) to 10 (diamond - very hard). This grading is based on the data we've gathered from players' previous attempts to solve them, and I am much indebted to Michael Wallace for his statistical wizardry in producing these numbers. Some conundrums did not have enough data for us to grade them, so they are temporarily ungraded; for practical purposes, they function as Grade 1, but we hope to eventually place them in the correct grade.

At the beginning of each game, we take the ratings of the two players (for this variant and dictionary) and take the higher of the two. If one of the players is a bot, we ignore it and use the other player's rating. If the player has only a provisional rating, having played fewer than 11 games in this variant/dictionary, then they are considered to have the lowest possible rating (although see the proviso below). We then look at how many players on the site as a whole have a rating lower than this number. For example, if the higher-rated player has a rating of 1700, this would be around 88% of the total.

The range of conundrums available depends on this percentage. If the percentage is between 0 and 10, we only set them conundrums from Grade 1. If the value is 10% - 20%, the players may be set conundrums from Grade 1 or Grade 2. So on up, until players rated in the top 10% (so that 90% - 100% of players are rated below them) can be set conundrums from any grade. Notice that ungraded conundrums count as Grade 1, so novice players may occasionally be exposed to a very difficult conundrum which has not been graded yet.

And that's the system.

Questions & Answers

Why let top players be exposed to very easy conundrums?

One reason for this is that top players will want to practice conundrums from all parts of the spectrum; very easy ones to improve their "quick reaction" spots, and very hard ones to extend their knowledge and deduction skills. Another is that we do not want to penalise them too harshly in terms of achieving high scores, max games etc.

I played another variant and got very hard conundrums, why?

For the standard variant, unrated players are considered to have the lowest possible rating, because they are probably completely new to the site. For all other variants, unrated players are assumed to have the highest possible rating, and therefore will probably face difficult conundrums. This is to prevent players new to a given variant from scoring easy max games, and encourages them to instead work towards gaining a full rating. Since the difficulties are computed from experience in the standard variant, they won't necessarily be accurate in any other variant anyway. We simply don't have enough data to do per-variant ratings.

I got into a tie-breaker and got a very hard conundrum, why?

If the game is tied going into the final round, we set the difficulty cap to 10 (the maximum), so there's a good chance that you'll get a really tough one. This isn't unfair since, if neither player can solve it, the game is still a draw so another conundrum will be set. This gives us a good way to gather data for the hardest conundrums, which would otherwise rarely be seen by most players. If the first tie-breaker goes unsolved, the cap is reduced to 9 and so on, until (hopefully) one player eventually solves one and the game ends!