[Paint Touches] How accurate are KenPom's preseason rankings?

[PaintTouches]

Took a broader look at preseason vs final rankings in KenPom for the Big East post reformation, and it turns out the guy knows what he's doing.

In the immortal word of Brian Fontana, 60% of the time it works every time.

https://painttouches.com/2022/10/19/how-accurate-are-kenpoms-preseason-rankings/

Forreal though, this kind of accuracy is seriously impressive.

[Newsdreams]

Facts no matta

[NCMUFan]

How accurate was he with last year's team?

[Uncle Rico]

Took a broader look at preseason vs final rankings in KenPom for the Big East post reformation, and it turns out the guy knows what he's doing.

In the immortal word of Brian Fontana, 60% of the time it works every time.

https://painttouches.com/2022/10/19/how-accurate-are-kenpoms-preseason-rankings/

Forreal though, this kind of accuracy is seriously impressive.

Analytics and predictive models don't decide the results and will be wrong, a lot in some cases!

However, anyone ignoring them or dismissing them are fools

[cheebs09]

How accurate was he with last year's team?

We overperformed by 31 spots. I believe some of this was in the article about Wojo before he got fired. Not a lot of overperformance in those years with some significant underperformance.

[tower912]

Yes.  Wojo consistently failed to make a team better than the sums of its parts.

[PointWarrior]

I like this Kevin McNamara guy....

[brewcity77]

I like this Kevin McNamara guy....

Like most online Providence fans, he really revels in making willful stupidity his brand.

[Pakuni]

Took a broader look at preseason vs final rankings in KenPom for the Big East post reformation, and it turns out the guy knows what he's doing.

In the immortal word of Brian Fontana, 60% of the time it works every time.

https://painttouches.com/2022/10/19/how-accurate-are-kenpoms-preseason-rankings/

Forreal though, this kind of accuracy is seriously impressive.

Now do AP and USA Today polls

[TAMU, Knower of Ball]

Now do AP and USA Today polls

He can't, those only go to 25, not 359

[Pakuni]

He can't, those only go to 25, not 359

1. He didn't do 359. He did 11, i.e. the Big East.
2. Throw in the others receiving voters and you can go to 40ish.

[TAMU, Knower of Ball]

1. He didn't do 359. He did 11, i.e. the Big East.
2. Throw in the others receiving voters and you can go to 40ish.

1. He did the Big East but looked at how those 11 teams fared in KenPom which ranks 359 teams. It's a lot harder to rank 359 teams than the top 25.
2. No you can't. The the top team in others receiving votes is not #26.

[PointWarrior]

Hopefully our analytics this year will be better than Kenpom's projections based off last year.

[Pakuni]

1. He did the Big East but looked at how those 11 teams fared in KenPom which ranks 359 teams. It's a lot harder to rank 359 teams than the top 25.
2. No you can't. The the top team in others receiving votes is not #26.

1. No it isn't.  He's dumping data into an algorithm. The only skill here is in creating the best possible formula. Once that's done, ranking 350 teams is no different or harder than ranking five.
2. Says who?

[TAMU, Knower of Ball]

1. No it isn't.  He's dumping data into an algorithm. The only skill here is in creating the best possible formula. Once that's done, ranking 350 teams is no different or harder than ranking five.
2. Says who?

1. ..... that's the whole point. He created an algorithm that can rank 359 teams better than most eye tests can
2. Logic. If 100/100 voters think that Marquette is the 26th best team in the country,  they will get 0 votes. If one voter thinks that DePaul is the 25th best team in the county but every other voter thinks theyre ranked somewhere 90 or below, they will get 1 vote. Which team is more well regarded? The one with 100 voters thinking they are 26th best (but 0 votes) or the one with with 1 voter thinking they are 25th best and 99 voters thinking they are 90th best or worse (but 1 vote).

[Pakuni]

1. ..... that's the whole point. He created an algorithm that can rank 359 teams better than most eye tests can
2. Logic. If 100/100 voters think that Marquette is the 26th best team in the country,  they will get 0 votes. If one voter thinks that DePaul is the 25th best team in the county but every other voter thinks theyre ranked somewhere 90 or below, they will get 1 vote. Which team is more well regarded? The one with 100 voters thinking they are 26th best (but 0 votes) or the one with with 1 voter thinking they are 25th best and 99 voters thinking they are 90th best or worse (but 1 vote).

1. Did he, though? You keep saying it, but you're not showing it. You're not comparing his predictions against an eye test or anything else. You're comparing his predictions against his predictions. Basically, the argument here isn't that KenPom's preseason rankings can accurately predict how good a team will be. It's that KenPom's preseason algorithm can accurately predict (sometimes) what KenPom's postseason algorithm will say.
Small sample size, but I made this post in August that shows, at least among the top 25 (because that's all we've got to compare against) KenPom's preseason rankings are no more or less accurate than the AP preseason poll in predicting final rankings.

https://www.muscoop.com/index.php?topic=62634.msg1464165#msg1464165

The claim that he can rank 300+ teams better than any eye test - which actually hasn't been shown here - is specious because there is no AP poll or any other eye test ranking 300+ teams.

2. You've correctly identified how the AP poll works, but how is that different for team 23 or 24?

[Uncle Rico]

1. Did he, though? You keep saying it, but you're not showing it. You're not comparing his predictions against an eye test or anything else. You're comparing his predictions against his predictions. Basically, the argument here isn't that KenPom's preseason rankings can accurately predict how good a team will be. It's that KenPom's preseason algorithm can accurately predict (sometimes) what KenPom's postseason algorithm will say.
Small sample size, but I made this post in August that shows, at least among the top 25 (because that's all we've got to compare against) KenPom's preseason rankings are no more or less accurate than the AP preseason poll in predicting final rankings.

https://www.muscoop.com/index.php?topic=62634.msg1464165#msg1464165

The claim that he can rank 300+ teams better than any eye test - which actually hasn't been shown here - is specious because there is no AP poll or any other eye test ranking 300+ teams.

2. You've correctly identified how the AP poll works, but how is that different for team 23 or 24?

KenPom is more accurate than the AP Top 25 in the numbers aren't a slave to the moment.

Xavier of 2018 is a good example.  Record was great and that's the goal, not being ranked high in the AP or KenPom but not a classic 1-seed and never higher than 11 all year in KenPom.  Losing in the 2nd round was an upset, but not particularly surprising.

As for start of the season, the AP and KenPom will be wrong about a few teams.  Over the course of the season, I'd trust KenPom much more than the AP.  That is changing as more voters are looking at the analytics to see if it matches the eye.

We argue about this because I think some people interpret the analytics guys on this board trying to say the numbers matter more than wins.  If Marquette is 14 in KenPom like X in 2018 and gets a 1-seed, I don't care.  Give me the 1-seed and I hope the analytics driven posters agree. 

Also, the majority of fans know the polls and don't care about any analytical numbers.  We're conditioned to look at the number in front of the team when it's in the paper, on CBS sports dot com or see it on the TV screen.  It's comforting when it's our team and infuriating when it's the other.  We want to protect that number and hurt the other teams number.  There's a world for both

[mu_hilltopper]

Probably need something besides averages.    If KenPom misses by 20 each year, but half the time it's -20, the other half +20 .. it looks like he's right on target.

That being said .. wow.  No need to play the games and get sweaty.

[Pakuni]

KenPom is more accurate than the AP Top 25 in the numbers aren't a slave to the moment.

Xavier of 2018 is a good example.  Record was great and that's the goal, not being ranked high in the AP or KenPom but not a classic 1-seed and never higher than 11 all year in KenPom.  Losing in the 2nd round was an upset, but not particularly surprising.

As for start of the season, the AP and KenPom will be wrong about a few teams.  Over the course of the season, I'd trust KenPom much more than the AP.  That is changing as more voters are looking at the analytics to see if it matches the eye.

We argue about this because I think some people interpret the analytics guys on this board trying to say the numbers matter more than wins.  If Marquette is 14 in KenPom like X in 2018 and gets a 1-seed, I don't care.  Give me the 1-seed and I hope the analytics driven posters agree. 

I agree!
Once the season is underway and there is a reasonable amount data from a team to consider, KenPom is great.
I'm only questioning the value/accuracy of his preseason predictions. Now more than ever when most teams and their personnel go through so many changes year after year.

[TAMU, Knower of Ball]

1. Did he, though? You keep saying it, but you're not showing it. You're not comparing his predictions against an eye test or anything else. You're comparing his predictions against his predictions. Basically, the argument here isn't that KenPom's preseason rankings can accurately predict how good a team will be. It's that KenPom's preseason algorithm can accurately predict (sometimes) what KenPom's postseason algorithm will say.
Small sample size, but I made this post in August that shows, at least among the top 25 (because that's all we've got to compare against) KenPom's preseason rankings are no more or less accurate than the AP preseason poll in predicting final rankings.

https://www.muscoop.com/index.php?topic=62634.msg1464165#msg1464165

The claim that he can rank 300+ teams better than any eye test - which actually hasn't been shown here - is specious because there is no AP poll or any other eye test ranking 300+ teams.

You acknowledge here that KenPom's preseason numbers can somewhat accurately predict what KenPom's postseason numbers will be.

I agree!
Once the season is underway and there is a reasonable amount data from a team to consider, KenPom is great.
I'm only questioning the value/accuracy of his preseason predictions. Now more than ever when most teams and their personnel go through so many changes year after year.

Then you acknowledge here that KenPom's in season and postseason data is great. So if the preseason rankings can somewhat predict the "great" in season and postseason data. Doesn't the speak to the value/accuracy of his preseason rankings? Again, no one is saying it is gospel. Just one tool that has a history of being decently accurate.

2. You've correctly identified how the AP poll works, but how is that different for team 23 or 24?

It's not different. That's why once you get past the teams that are in the top 25 on every bracket, it's not an accurate measure. And this is part of the point. It's possible to look at a measure and see flaws with the structure without even looking at the results. Flawed structures can still post accurate results, but it doesn't mean that the structure isn't flawed. And to be clear, there are flaws with metrics like KenPom as well, just not as many.

[Frenns Liquor Depot]

Probably need something besides averages.    If KenPom misses by 20 each year, but half the time it's -20, the other half +20 .. it looks like he's right on target.

I think he used absolute numbers which eliminates this point

[UWW2MU]

I think he used absolute numbers which eliminates this point

Using absolute helps, but still doesn't tell us much since many of the teams are on the higher end of the rankings.  The margin for getting it wrong/right narrows.   Being off by 7 on a team that averages a rank of 10 is not as impressive as being within 7 of a team that averages a rank of 70.   We need to know the p-value

[muwarrior69]

So how does the KemPom preseason rankings compare to the Big East Coaches preseason Poll/ranking?

[Dr. Blackheart]

You acknowledge here that KenPom's preseason numbers can somewhat accurately predict what KenPom's postseason numbers will be.

Then you acknowledge here that KenPom's in season and postseason data is great. So if the preseason rankings can somewhat predict the "great" in season and postseason data. Doesn't the speak to the value/accuracy of his preseason rankings? Again, no one is saying it is gospel. Just one tool that has a history of being decently accurate.


It's not different. That's why once you get past the teams that are in the top 25 on every bracket, it's not an accurate measure. And this is part of the point. It's possible to look at a measure and see flaws with the structure without even looking at the results. Flawed structures can still post accurate results, but it doesn't mean that the structure isn't flawed. And to be clear, there are flaws with metrics like KenPom as well, just not as many.

Just to be clear, Pomeroy doesn't predict or model rank, he predicts Adjust O and D, then ranks on the difference between the two (Adj EM). The differences in Adj EM at the top and bottom are much wider than in the majority middle as a rank forces distribution.

Said another way, the differences in rank are not the same along the curve. You have stated this in the past but just to reiterate it.

As to the Top 25 discussion, most voters rely on Pomeroy (confirmation bias) or they are homers and vote the home state public school. Pomeroy or other statistical models at least take the subjectivity out and are consistent in their objectivity. All are great for fan engagement and discussion.

[tower912]

How long until the games start so we can argue about actual things and not abstracts?