I've written about this topic before, but now have done some deeper analysis.
Ted Glover wrote an article about over and underrated teams back in March. It was based on Chris Stassen's over and under rated matrix based on pre-season polls and the final AP poll, which can be found here. In brief, Stassen uses a cumulative difference between a consensus pre-season ranking and the final AP ranking every year since 1989 (when the AP poll went back to 25 teams) to show which schools are the most overrated. An unranked team is always treated as #26.
I have several issues with his approach, but my main objection is that highly ranked teams can't move up and thus his approach is skewed. The other main issue is that he doesn't normalize for the number of poll appearances a team makes.
The use of #26 for all unranked teams is a problem, but there isn't a great solution so I followed his lead in my analysis. I did switch to solely using the pre-season AP poll, though, as that seems more fair (as well as easier).
Average Finish for Each Poll Position
The best linear fit is y=8.51+0.623x.
As expected, the top spots fall a lot on average while the average is more and more correct as you reach the bottom of the poll.
How Bad is the Pre-Season Poll?
The bottom spots in the average finishes above are skewed by 26 being the lowest possible value. At the lower ranks, the majority of the teams finish unranked (#25 is only 4/24). Strangely, pre-season #3 and 4 are bad places to be while #14 and #17 do well relative to their neighbors.
Number of unranked teams that started at each spot:
As you can see, the last 10-15 spots are pure guesswork.
In case anyone wonders, the number of teams that finish out of the poll hasn’t changed much over the years. The average is 9.2 teams but a linear fit starts at 8.2 teams in 1989.
So How Overrated Are Teams?
Knowing what the expected finish for each spot should be, I went back and calculated the difference between where a team did finish and where they were expected to finish.
Example: 2002 OSU
Average finish for #13 is 17.54
Actual finish was #1
Difference = 16.54
I did this calculation for every team in every pre-season poll and then averaged the results for each school. In a nutshell, positive numbers are good and negative numbers are bad.
1. OR – 4.67
2. AL – 3.80
3. Stanford – 3.75
4. OSU – 3.37
5. VT – 2.68
6. KSU – 2.65
7. UGA – 2.47
8. PSU – 2.09
9. BSU – 1.97
10. Miami – 1.60
I think many will be surprised to see AL and OSU so high on this list. 30 schools averaged a positive value out of 68 schools that were ranked.
OSU – 3.37
PSU – 2.09
NE – 0.37
MI – -0.03
IA – -0.08
PU – -1.01
MSU – -1.15
WI – -1.96
IL – -4.68
Others of note:
ND – -4.21
FSU – 1.13
Miami – 1.60
So as much as many people like to complain about the kings being overrated every year, it’s really not true. They tend to do better than other teams ranked that same way with the notable exception of ND. Instead, what people are noticing is that most teams can’t live up to their preseason ranking and some subset of the kings usually occupy the top spots.