Last position people are awarded positive rating changes.

[atiabjobayer]

Used Algorithm: mmr

Consider a situation where 3 user participated in a contest[user1, user2, user3]. Consider it's a programming contest. Let's say their scores are [300/500, 200/500, 0/500].

Now the basic sense suggest that since the user3 scored absolutely zero, he should get a negative rating change. But what happens is, if user1 and user2 have higher rating than user3 before the contest, then user3 receives a positive rating change, even if he scores zero.

Is there any way to prevent this behavior? Doesn't seem right at all.

[EbTech]

Hmm it's hard to tell what exactly happened without going through the math in detail, but this shouldn't usually happen. From the 0/500 player's perspective, the algorithm treats the result as 1 loss against the 300 player, 1 loss against the 200 player, and either 1 loss + 1 win or 0.5 loss + 0.5 win (depending on the split_ties parameter) against themself. Losses never help, so you might like to try removing one or both of the winners from the match to confirm that having fewer losses results in a higher rating for the 0/500 player.

That leaves the question of why a player's rating would increase from a neutral result, where they play only against themself. The answer is that probably mu didn't change, but sigma decreased; if we publish something like mu - 2*sigma, then the published rating will be higher. sigma changes a lot for new players, but stabilizes later so you'd no longer see this effect. Of course, a 1-player match is completely uninformative so sigma really shouldn't be updated, but Elo-MMR's measure of information gain is based on the limit as the number of participants goes to infinity.

[Plantain]

I am seeing the opposite case - first place players have their rating go down after their first placing, even with relatively mature players (>10 rounds)
Can that happen for the same reason? How do I debug this?

[EbTech]

It might happen if it's not a true 1st place, i.e., in the event of a tie. Otherwise, check that you followed the contest data format and 0-indexed place numbering exactly as the README says. Among the files that get generated, you can also find a player's calculated "performance" score for the match they won, which should exceed their previous and current rating.

[Plantain]

I believe it's happening because perf_ceiling is set - I think if the ceiling is set to 2000, and a competitor with ranking 2100 enters and wins, their rating should not lower. Is that a bug?

[EbTech]

Ah yes that would do it. I didn't anticipate perf_ceiling being set below the maximum rating of allowed contestants. This can be adjusted but, probably such a contestant shouldn't be entered into the rated standings at all, for a contest whose ceiling is rated lower?

[Plantain]

What I am trying to achieve is setting a ratings 'cap' on lower level contests (perhaps with less serious adherence to rules), such that you can only reach the highest ELO's by flying higher level contests. I think there is still valuable signal if a 2500 ELO contestant enters a 2000 ELO capped contest and gets beaten by a 1900 ELO, so I don't think they should be ignored, but I'm not sure what the right behaviour is either.

Perhaps the 2500 ELO contestant should have a 'virtual' 2000 ELO for such contests?

[EbTech]

The trouble with allowing players to participate at a ceiling below their current rating is that there is no opportunity for them to gain points, so it's never worthwhile for them to be rated.

On the other hand, the ceiling can be set higher than you might expect. Even if the ceiling were 2500, no player will actually acquire a 2500 rating unless they win every single time forever without fail.

[atiabjobayer]

@EbTech the issue still persists. Today I had this one contest where 30 people scored positive(more than zero), and 15 people score zero. So all those 15 people were awarded position 31.

And then once I ran the rating calculator, many of them people tied at position 31 got positive rating. How do I get rid of this situation?

[EbTech]

If the tie is between 31st and 45th, their 0-indexed rank should be [30,44]. Ratings are usually positive, but rating.mu should decrease for these losing players. Displayed ratings might still increase for relatively new players, because those are purposely deflated for newcomers and then gradually restored to rating.mu (which starts at 1500 by default).