Friday 26 February 2010

Olympic Metrics

I don't mind using medal counts as a metric for Olympic success, and I fully acknowledge that Canada badly failed in their "Own the Podium" campaign, but I think that the heavy focus on medal counts misses a lot of information. Clearly a fourth place finish is far superior to a fortieth place finish but we don't see that in medal counts. Another problem: sports like hockey or curling award only two medals, while most others, like short track speed skating, award large batches of them. If country A has an excellent hockey program, at best it's rewarded with two medals, if country B has an excellent short track speed skating program it can result in 8 or 10, but it doesn't seem to follow that country B is 4 or 5 times better at winter sports than is country A. I've thought of a few other simple metrics that would help communicate additional information.

a) Average percentile or ordinal/athlete: Take the average of the sum of the ordinal of the finish of all participants from a given country. Or if that would bias too heavily to sports with fewer participants, use percentiles. DNF or DQ count as one place lower than the last qualified finisher.

b) Average percentile or ordinal/event: Same as (a), except that we factor in breadth of participation so that breadth of qualification and participation is factored in. If a country has no participant in an event, we count that as a last place finish. If a country has two or more participants in an event, we take the average of their ordinals or percentiles.

c) Sport counts rather than medal counts: Given a metric for success in an event, e.g., best percentiles or most medals, we then identify a finish/sport. That number is then used to measure overall success. So, for example, a country finishes with two golds in hockey, they get 1st in hockey. They finish with 4 medals in short track but another country has 5 (with some weighting for ordinal, of course), they get second in short track, etc.

I don't think these need replace medal counts, but I think they'd be useful supplementary metrics. Next step, calculate these for the 2010 Games. (Don't hold your breath.)

Friday 19 February 2010

Short tournaments

Short tournaments are notoriously bad for choosing the best team. I read an article a few months ago on the likelihood that a soccer game will select the best team and the fact of the matter is that a single game isn't a very good experiment. (short summary) In light of this I've been thinking about the optimal setup for a very short tournament of the sort necessitated by the Winter Olympics, assuming we want the tournament to be maximally likely to "choose" the best team. The article considered, among other things, intransitivity as an indicator of the effectiveness as a game, i.e., if Team A beats Team B which beats Team C which beats Team A then we have an intransitivity. A lot of intransitivity suggests the games aren't very good at selecting winners.

I suspect the intransitivity will be lower in hockey, not because the games are better experiments but because the difference in relative ability is much greater between teams. But, there are also tiers of skill levels, I suspect. If we consider games between the top 5 or 6 teams in the tournament I suspect we'd see close scores and fairly high intransitivity. So, in a good tournament, I think the thing to do is to have early knockout of bad teams and allow for more games between the really good teams. I don't suspect that the best way to set up the tournament is to have a long round robin (3 games/team) after which zero teams are eliminated. If Norway loses 8-0 to Canada, they should immediately move to a consolation pool, the US shouldn't then be forced to waste time playing this team. But this tournament eliminates zero teams after the round robin, although they do give the top four teams a bye through one of the playoff rounds. It will have Russia and the US and Canada wasting a lot of time, three games, on potentially far lesser skilled teams and then move to single elimination games in the playoffs. I think it's a very poorly designed setup if our goal is to be able to crown the truly best team.

x-posted to blogspot.