Results (ongoing)
https://1drv.ms/x/s!ArCAuhmbxCeRgQdbFut ... I?e=ndCRqh
https://forum.icmforum.com/viewtopic.ph ... 40#p708620
Doubling the Canon 2021 Update - Results!
well done angel!
wonder why participation is low this year when we're all mostly sat around at home not doing much else, maybe the plague has been brutal to some cinephiles
wonder why participation is low this year when we're all mostly sat around at home not doing much else, maybe the plague has been brutal to some cinephiles
They were missed.twodeadmagpies wrote: ↑Sun May 02, 2021 10:25 am wonder why participation is low this year when we're all mostly sat around at home not doing much else, maybe the plague has been brutal to some cinephiles
I should have brought this up earlier ...
- Bayesian averaging is the wrong thing to do in the first place. it would be correct if random people had seen each film, because then high ratings would probably be a fluke, and you'd want to regress to the mean. But when a film doesn't get many votes, it's not random people who have seen it. It's going to be people who have seen a lot of films. So those ratings are going to come from the only people who can fairly compare the underseen films with the widely seen ones. And if the sample isn't random, the whole assumption behind Bayesian averaging is wrong.
- The rules also ask the people who have seen the most films to give the lowest ratings; a film in the top 600 is supposed to get a '6'. Someone who has only seen 3000 films is giving 20% of films a 6. Someone who has seen 60,000 films is giving 1% of films a 6.
So Bayesian averaging treats the ratings of the most knowledgeable voters as if they were the least informative ratings, and doubly punishes those raters if they followed the ratings guidelines. It's not the right thing to do, but it might not matter much if the effect was small. But when I look at the WA column in the new ratings spreadsheet, I notice two things:
- first, it seems like the wrong average is being used in the calculations (it seems to be using 3.33, the average from last year, instead of 3.41, the average of the actual data from this year)
- but way more importantly, the averaging seems to be mixing in eight average ratings. That is way more than I was expecting (I thought it would use maybe three), and more than is justified by the number of ratings past DTC polls assumed were meaningful.
When I read the icheckmovies forum discussion a year ago that proposed using a Bayesian average, my impression was people were upset that Douglas Sirk's eleventh best film didn't make the list but some obscure Albanian thing did. If the goal of DTC is just to list the films the real TSPDT voters would have ranked 2001st-3000th, then using a heavy Bayesian average is the right thing to do -- then Hitchcock's 26th best film will probably make the list, and any underseen film from Burkina Faso won't. If instead the idea is that some films are systemically excluded from the TSPDT list because the TSPDT voters have biases in their viewing, and if DTC is trying to propose an alternate canon of films that TSPDT voters aren't even considering, then the Bayesian averaging formula that's being used right now is bananas.
I appreciate all the work on DTC, as always, but I hope you'll look at the rating calculation again! I'm worried with the current formula that the final list will end up much more anodyne than before, and will be much less valuable as a result..
- Bayesian averaging is the wrong thing to do in the first place. it would be correct if random people had seen each film, because then high ratings would probably be a fluke, and you'd want to regress to the mean. But when a film doesn't get many votes, it's not random people who have seen it. It's going to be people who have seen a lot of films. So those ratings are going to come from the only people who can fairly compare the underseen films with the widely seen ones. And if the sample isn't random, the whole assumption behind Bayesian averaging is wrong.
- The rules also ask the people who have seen the most films to give the lowest ratings; a film in the top 600 is supposed to get a '6'. Someone who has only seen 3000 films is giving 20% of films a 6. Someone who has seen 60,000 films is giving 1% of films a 6.
So Bayesian averaging treats the ratings of the most knowledgeable voters as if they were the least informative ratings, and doubly punishes those raters if they followed the ratings guidelines. It's not the right thing to do, but it might not matter much if the effect was small. But when I look at the WA column in the new ratings spreadsheet, I notice two things:
- first, it seems like the wrong average is being used in the calculations (it seems to be using 3.33, the average from last year, instead of 3.41, the average of the actual data from this year)
- but way more importantly, the averaging seems to be mixing in eight average ratings. That is way more than I was expecting (I thought it would use maybe three), and more than is justified by the number of ratings past DTC polls assumed were meaningful.
When I read the icheckmovies forum discussion a year ago that proposed using a Bayesian average, my impression was people were upset that Douglas Sirk's eleventh best film didn't make the list but some obscure Albanian thing did. If the goal of DTC is just to list the films the real TSPDT voters would have ranked 2001st-3000th, then using a heavy Bayesian average is the right thing to do -- then Hitchcock's 26th best film will probably make the list, and any underseen film from Burkina Faso won't. If instead the idea is that some films are systemically excluded from the TSPDT list because the TSPDT voters have biases in their viewing, and if DTC is trying to propose an alternate canon of films that TSPDT voters aren't even considering, then the Bayesian averaging formula that's being used right now is bananas.
I appreciate all the work on DTC, as always, but I hope you'll look at the rating calculation again! I'm worried with the current formula that the final list will end up much more anodyne than before, and will be much less valuable as a result..
The formula is not written in stone so thanks for your comment, flip.
- Yes I'm using the average of the last 5 years (3.33) because it does affect all the films rated these last 5 years. Otherwise this year's rates could not be combined with previous rates (obtained with a different formula).
- 8 are the minimum votes required to make the list.
I am not a mathematician, I can only give my lay opinion: It seems that the average (3.33) and the cutoff (3.6-3.7) are so close that the influence of the formula is not very strong in this area (and this is the only area where a film with few votes might not make the cut because of the formula). Believe me if I tell you that I share your concern and I hope you are not right.
- Yes I'm using the average of the last 5 years (3.33) because it does affect all the films rated these last 5 years. Otherwise this year's rates could not be combined with previous rates (obtained with a different formula).
- 8 are the minimum votes required to make the list.
I am not a mathematician, I can only give my lay opinion: It seems that the average (3.33) and the cutoff (3.6-3.7) are so close that the influence of the formula is not very strong in this area (and this is the only area where a film with few votes might not make the cut because of the formula). Believe me if I tell you that I share your concern and I hope you are not right.
Even though I've been gazumped to the actual nomination, delighted for Babylon, what a film. And a good launchpad for Central Park, hope more people see it by next year and get it over the line then.
Kamome shokudô (2006, Naoko Ogigami)
My favorite discovery this year! There are good movies, bad movies and therapeutic movies (you start with a headache and leave feeling good). The latter are beyond valuation.
Congratulations to MrCarmady.
My favorite discovery this year! There are good movies, bad movies and therapeutic movies (you start with a headache and leave feeling good). The latter are beyond valuation.
Congratulations to MrCarmady.
delighted, good bounce-back from its defeat in the cupLetterboxd list updated.
I wish to thank all the enthusiastic participants. See you again next year!
I wish to thank all the enthusiastic participants. See you again next year!
Thx angel!
