Why the large discrepancy between Pomeroy and other RPI sites?

[Spencer Pratt]

I have never understood why over the years Pomeroy usually has us ranked considerably higher than most of the other guys calculating RPI rankings.  Currently, we sit at #26 according to kenpom.com but are way down at #68 at realtimerpi.com.  The same can be said for many other teams as well.  Nebraska - #35 Ken Pom, #87 Realtime RPI.  Iowa State- #32 Ken Pom, #82 Realtime RPI. 

On the other side of the spectrum, you have a team like Cleveland St. ranked #70 by Ken Pom but #30 by Realtime RPI. 

How much weight, if any, does the committee put into Pomeroy's numbers and should we as fans place more importance on one ranking over another?



               
                 

[Marquette84]

I have never understood why over the years Pomeroy usually has us ranked considerably higher than most of the other guys calculating RPI rankings.  Currently, we sit at #26 according to kenpom.com but are way down at #68 at realtimerpi.com.  The same can be said for many other teams as well.  Nebraska - #35 Ken Pom, #87 Realtime RPI.  Iowa State- #32 Ken Pom, #82 Realtime RPI. 

On the other side of the spectrum, you have a team like Cleveland St. ranked #70 by Ken Pom but #30 by Realtime RPI. 

How much weight, if any, does the committee put into Pomeroy's numbers and should we as fans place more importance on one ranking over another?
               

Pomeroy and RPI use vastly different criteria. You can read the details at these two sites:

Pomeroy: http://kenpom.com/blog/index.php/weblog/ratings_explanation/
RPI:  http://en.wikipedia.org/wiki/Ratings_Percentage_Index

While some argue that Pomeroy is more accurate, RPI is what the committee uses at selection time.  If you're arguing with a friend who is the better team, use Pomeroy.  If you're trying to guage a team's chances at making the tourney, use RPI.

[ChicosBailBonds]

I have never understood why over the years Pomeroy usually has us ranked considerably higher than most of the other guys calculating RPI rankings.  Currently, we sit at #26 according to kenpom.com but are way down at #68 at realtimerpi.com.  The same can be said for many other teams as well.  Nebraska - #35 Ken Pom, #87 Realtime RPI.  Iowa State- #32 Ken Pom, #82 Realtime RPI. 

On the other side of the spectrum, you have a team like Cleveland St. ranked #70 by Ken Pom but #30 by Realtime RPI. 

How much weight, if any, does the committee put into Pomeroy's numbers and should we as fans place more importance on one ranking over anoter

RPI looks at wins and losses only, no scoring differential.  That's one major reason right there.  RPI also gives a bonus for road wins and a penalty for home losses.

[dsfire]

RPI tries to rate the quality of a team's wins and losses to date, while pomeroy's system tries to measure the quality of a team's play for predictive purposes.  I believe I heard the pomeroy numbers were provided during one of the mock selection committees that were done last year, but there's certainly no official emphasis on them.  For what it's worth, my impression is that the emphasis on RPI has been dialed back a lot in recent years.

As a fan, feel free to put importance on whatever system you like.  Certainly helps arguments if you understand what the numbers mean, though.

edit: Pomeroy explains some of his philosophy here: http://kenpom.com/blog/index.php/weblog/ratings_explanation/  Oops, that's the same link as in Marquette84's post

[Nukem2]

Don't know that the Pomeroy #'s were made available to thhe NCAA committee ( though I would not be surprise and certainly individual members certainly look at KPom ).  In any event, the selection of teams and the seedings have certainly mimicked the KPom rankings far more than the RPI.  What I've read is that the RPI is used more for administrative purposes than selection or seeding.  when I look at the bubble and seeding, I personally ignore the RPI and look at KPom.  KPom "represents" reality far more than the  RPI.

[MuMark]

Actually the selection committee makes use all of the major computer information not just RPI.

During the mock committee process many of the participants were surprised that RPI was never talked about.

Members get lots of information and its up to each person to decide how much weight to give them.

"Body of work" is the term mostly used in deciding who gets in.

[Lennys Tap]

RPI is not nearly as important as it once was. Last year there many (9?) teams with higher RPI than us that didn't even make the tournament. We got a #6 seed. So much for the almighty RPI.

[ChicosBailBonds]

Don't know that the Pomeroy #'s were made available to thhe NCAA committee ( though I would not be surprise and certainly individual members certainly look at KPom ).  In any event, the selection of teams and the seedings have certainly mimicked the KPom rankings far more than the RPI.  What I've read is that the RPI is used more for administrative purposes than selection or seeding.  when I look at the bubble and seeding, I personally ignore the RPI and look at KPom.  KPom "represents" reality far more than the  RPI.

Yes, they are and this is directly from two friends that have been on the committee in the last 10 years.  Sagarin, Pomeroy, RPI, Massey, etc, etc are all provided.  The RPI carries a lot of importance, however, because it was developed by the NCAA.  Some of the other ratings are predictors of outcomes and not necessarily power rankings of teams, many people get confused on the differences.

[ChicosBailBonds]

RPI is not nearly as important as it once was. Last year there many (9?) teams with higher RPI than us that didn't even make the tournament. We got a #6 seed. So much for the almighty RPI.

Yet there were also teams that made it ahead of others because they had a better RPI ... it goes both ways.  It's important but it's used selectively is the way it was put to me.  Often times to help break ties or distinguish objectively between 4 or 5 teams. In some cases it will break the way you stated above, but in other cases it will break the other way. 

[MuMark]

"made it ahead of others because they had a better RPI"

You know they made it....you don't know that the RPI was the reason.

It could have been a bunch of reasons and was likely a combination of factors not just the RPI.

[ChicosBailBonds]

"made it ahead of others because they had a better RPI"

You know they made it....you don't know that the RPI was the reason.

It could have been a bunch of reasons and was likely a combination of factors not just the RPI.

Correct, any number of factors are used.  I guess the best way to state it from the two folks that I know that have been on the committee is that the RPI can help or hurt but it's one of many tools at the committee's disposal.  When they are making an argument for or against a school, someone strongly pushing a school may use RPI to advocate their inclusion while another member may use RPI to prove the school doesn't belong.  But, it's one of MANY tools.

[Spencer Pratt]

It would be interesting to know how knowledgebale to committee members truly are regarding specific teams they either place in or out of the tourney.  Have these guys actually watched multiple games and come to a logical conclusion based on what they have seen, or are they simply comparing resumes on a piece of paper and choosing one over another.  I know that the "eyeball test" is always referenced, but I have a feeling some of the committee members haven't even watched more than a half of basketball on certain teams they are deciding on. 

[ChicosBailBonds]

It would be interesting to know how knowledgebale to committee members truly are regarding specific teams they either place in or out of the tourney.  Have these guys actually watched multiple games and come to a logical conclusion based on what they have seen, or are they simply comparing resumes on a piece of paper and choosing one over another.  I know that the "eyeball test" is always referenced, but I have a feeling some of the committee members haven't even watched more than a half of basketball on certain teams they are deciding on. 

Most of the committee members are either Conference Commissioners, large university AD's, or even school presidents.  They certainly see their fair share of basketball at their own schools and conferences, but they can't watch it all...they have day jobs.


http://www.bracketography.com/selection-committee/

[MuMark]

They watch a bunch of games especially when it gets closer to SS.

Bilas has talked about how the goal is to have the 34 best at large teams. To judge which teams are best you have to watch games.

Chicos is right though nobody can see them all.

[esotericmindguy]

Do yourself a favor and compare Kenpom to vegas....I think you'll see who's more accurate when the chips are down.

[marquette99]

The committee never officially will make pomeroy an official criteria, because every point you win bu ups your rating.  You don't want to encourage buzz to leave the starters in to beat depaul by 50 instead of 30. The bcs originally allowed systems like pomeroy until florida and others ran up 70-0 scores.

The problem is rpi gives no credit for one point losses at places like vandy and louisville.

Unofficially, you better believe the selection committee looks at everything.

[Wareagle]

I have never understood why over the years Pomeroy usually has us ranked considerably higher than most of the other guys calculating RPI rankings.  Currently, we sit at #26 according to kenpom.com but are way down at #68 at realtimerpi.com.  The same can be said for many other teams as well.  Nebraska - #35 Ken Pom, #87 Realtime RPI.  Iowa State- #32 Ken Pom, #82 Realtime RPI. 

On the other side of the spectrum, you have a team like Cleveland St. ranked #70 by Ken Pom but #30 by Realtime RPI. 

How much weight, if any, does the committee put into Pomeroy's numbers and should we as fans place more importance on one ranking over another?

If money is on the line, chose Pomeroy.  I've won two NCAA pools mainly by using KenPom's numbers. 

[Marquette84]

Given the debate on whether Pomeroy or RPI was a better indicator of making the tournament, I thought some facts might be useful to insert into the discussion.  Using 2010 tournament data, I wanted to see if either RPI or Pomeroy had a better track record of predicting both the field and the seed.

--I considered only the top 12 seeds (48 teams), plus the top 48 ranked teams in RPI and Pomeroy.  I figured the 13-16 seeds were all tournament champs that would not have been picked at large, and thus irrelevant to this discussion.  

--The only exception to the above was 13 seed Siena, included because they were ranked #32 in RPI (#59 in Pomeroy)

--I created a predicted seed from Pomeroy and RPI rank, using 1-4 ranked teams as predicted #1 seeds, 5-8 as #2 etc.  

--I calculated the difference (up or down) between the actual seed from the RPI/Pomeroy predicted seed.

--I Ignored any tournament seeding rules such as teams from same conference meeting before 2nd round or protected seeds forcing a move up or down a seed line by another team.

--Once I had the deviation from predicted seed, I did a sum of the absolute value for each ranking

For example--Villanova was a 2 seed.  Pomeroy had them 21st, which equated to a 6 seed, so Pomeroy was off by 4 seeds.  RPI had them at 16, which equated to a 4 seed, so for Villanova, RPI was off by 2 seeds.


Conclusions & Observations.

--In total, Pomeroy was off by 116 total seed lines, RPI off by a total of 112, making RPI just slightly more accurate,

--The biggest deviation on Pomeroy was New Mexico.  They were ranked 54th, equating to a predicted 14th seed.  They actually received a 3 seed.  BTW, RPI had them ranked 14th, or an expected 4 seed.

--The biggest deviation on RPI was Marquette, ranked 55th in the RPI (equating to a 14th seed.  Actually received a 6. Pomeroy had us at 33, or a predicted 9 seed.

--Pomoroy had 4 teams in their top 48 that did not get bids:  26 Dayton, 37 Va Tech, 46 Miami, 47 ASU.  Only 3 teams with an RPI over 48 made the tournament.

--RPI had 5 teams in the top 48 that did not get bids:  36 Dayton, 38 URI, 47 VCU, 47 Kent St., 48 UAB.  Six teams with an RPI over 48 made the tournament.

In short, based on 2010 data, Pomeroy is ever so slightly less accurate for predicting the NCAA field and their seed.  

[Hards Alumni]

In short, based on 2010 data, Pomeroy is ever so slightly less accurate for predicting the NCAA field and their seed using my criteria.

fixed.

[dsfire]

For what it's worth, I do think the comparison of how pomeroy's system fares at picking teams versus RPI is interesting and can be worthwhile.  One of the pitfalls, however, is that the numbers continue to be updated during the postseason, so they aren't actually reflective of what the predicted teams were on Selection Sunday.  RPI numbers from that period are probably easier to find since RPI is pretty useless after that point.  Might be able to find both on the website with the dance cards.

And grading either system on seeds is kinda harsh, since seeds can be moved up or down for any number of reasons (which isn't to say that actual bracketologies shouldn't be graded on seeds).   S-curve numbers would be more accurate but I don't think those are released.

[Marquette84]

fixed.

What did you fix?  Anything? 

I find it interesting that you supposedly "fixed" the only post in which someone used actual facts rather than gut feel.

Apparently, you have some issue with the way I made the comparison, but as far as I can tell you neither identified any inaccuracy in the method used, nor did you offer any alternative or suggestions that might improve the accuracy.

For what it's worth, I do think the comparison of how pomeroy's system fares at picking teams versus RPI is interesting and can be worthwhile.  One of the pitfalls, however, is that the numbers continue to be updated during the postseason, so they aren't actually reflective of what the predicted teams were on Selection Sunday.  RPI numbers from that period are probably easier to find since RPI is pretty useless after that point.  Might be able to find both on the website with the dance cards.

The RPI isn't updated after the final conference tournament games.

And since the subject was predicting how accurate the ranking system was with respect to predicting tournament bids and seeds, the fact that RPI doesn't include those games is helpful.

And grading either system on seeds is kinda harsh, since seeds can be moved up or down for any number of reasons (which isn't to say that actual bracketologies shouldn't be graded on seeds).   S-curve numbers would be more accurate but I don't think those are released.

But that was precisely the point--whether one is better than the other at predicting seed.  People are making unsubstantiated claims that Pomeroy is more accurate with respect to predicting who gets in the tournament and where they'll be seeded. 

The reality is that Pomeroy appears to be no better than RPI when it comes to predicting seed.. 


RPI is no longer updated after the the final confernece tournament games.

I don't know if Pomeroy updates to include CNAA tournament games.

[NersEllenson]

What did you fix?  Anything? 

I find it interesting that you supposedly "fixed" the only post in which someone used actual facts rather than gut feel.

Apparently, you have some issue with the way I made the comparison, but as far as I can tell you neither identified any inaccuracy in the method used, nor did you offer any alternative or suggestions that might improve the accuracy.


The RPI isn't updated after the final conference tournament games.

And since the subject was predicting how accurate the ranking system was with respect to predicting tournament bids and seeds, the fact that RPI doesn't include those games is helpful.

But that was precisely the point--whether one is better than the other at predicting seed.  People are making unsubstantiated claims that Pomeroy is more accurate with respect to predicting who gets in the tournament and where they'll be seeded. 

The reality is that Pomeroy appears to be no better than RPI when it comes to predicting seed.. 


RPI is no longer updated after the the final confernece tournament games.

I don't know if Pomeroy updates to include CNAA tournament games.

Pretty sure you missed the humor in the "fixed" comment..all he was basically saying was that instead of taking the time to read through your long e-mail...people could get the conclusion and findings of your analysis in 1 sentence.

[dsfire]

But that was precisely the point--whether one is better than the other at predicting seed.

Your post started with a sentence about making the tournament, which is not exactly the same as predicting seeds.  My point was simply that a good system would (hopefully) have a strong correlation with predicting at-large bids and S-curve numbers, but the correlation would drop when look at the seed numbers due to the tweaks the committee makes for things like not facing a conference opponent within the first two rounds.

RPI is no longer updated after the the final confernece tournament games.

I don't know if Pomeroy updates to include CNAA tournament games.

The kenpom numbers definitely include tourney games.  RPI, it probably depends where you get the numbers from.  http://www.realtimerpi.com/2008-2009/rpi_123_Men.html appears to have Duke's 2009 tourney games factored in but the following season doesn't include tourney results (though it has an entry for their first round opponent).

[Marquette84]

Your post started with a sentence about making the tournament, which is not exactly the same as predicting seeds.  My point was simply that a good system would (hopefully) have a strong correlation with predicting at-large bids and S-curve numbers, but the correlation would drop when look at the seed numbers due to the tweaks the committee makes for things like not facing a conference opponent within the first two rounds.
The kenpom numbers definitely include tourney games.  RPI, it probably depends where you get the numbers from.  http://www.realtimerpi.com/2008-2009/rpi_123_Men.html appears to have Duke's 2009 tourney games factored in but the following season doesn't include tourney results (though it has an entry for their first round opponent).

Well, all I can say is that my first sentence did say "making the tournament" but the rest of the post also included the accuracy of seed.  I apologize if I wasn't clear that my post analyzed both.

While your point that the committee's S-curve would be more accurate, you also mention that we don't have access to it.  Makes it tough to know if we're in our out, and what seed we might receive.  As fans, we only have access to the public rankings such as RPI and Pomeroy.

For some reason, there is an general attitude that the Pomeroy rankings are far superior as indicators of whether a team will make the tournament and with what seed.  Take a look at the comments just in this thread--and these are representative of comments made any time the RPI is brought up:

• When I look at the bubble and seeding, I personally ignore the RPI and look at KPom. 
  
• KPom "represents" reality far more than the  RPI.
• Last year there many (9?) teams with higher RPI than us that didn't even make the tournament. We got a #6 seed. So much for the almighty RPI.
  
• The problem is rpi gives no credit for one point losses at places like vandy and louisville.
• If money is on the line, chose Pomeroy.  I've won two NCAA pools mainly by using KenPom's numbers.
  

When I see so many one-sided comments, I became curious as to the reality.  Is Pomeroy really that much better than RPI when it comes time to guessing what teams are in and where they're seeded?  Almost everyone here seems to think so. 

So I used the 2010 tournament field to see.  And I have to admit that I was surprised by the findings.  Given the conventional wisdom that RPI is flawed, nowhere near as accurate, the committee doesn't use it, they give other ratings more credence, etc. etc. etc., I really thought there would be a significantly greater correlation between the Pomeroy rank and the actual seed received by a team.  I thought the performance of the RPI would be significantly worse than it was.

I guess I shouldn't be surprised anymore that the general response seems to be "don't bother us with the facts."

[Marquette84]

Pretty sure you missed the humor in the "fixed" comment..all he was basically saying was that instead of taking the time to read through your long e-mail...people could get the conclusion and findings of your analysis in 1 sentence.

It would have been a lot shorter if I simply said that everyone was wrong, RPI was actually slightly more accurate at predicting the field and seeds come tournament time for the 2010 season.

But if I didn't include the facts and methodology, someone would take such a non-contoverial subject, regardless of its basis in fact, and launch into some personal attack.