Part 1 - Beyond the Stars: Unpacking the Reality of College Football Recruiting Ratings
Breaking down a useful, though often misunderstood, proxy metric
Chapter 15 of Let Me Be Frank, the 2013 book that Frank Beamer cowrote with Jeff Snook, is titled “Recruiting: How many stars? Who Cares?”
Commenting on the star system of player evaluation, Beamer wrote (pg. 188-189):
I don’t pay much attention to the “star ratings.” Right now, there are many services out there that rank players on a one-to-five-star rating. I hear from fans all the time who ask me, “Why aren’t we signing more five-star guys?”
Well, for one, I don’t put much stock in those ratings. Most aren’t based on scouts who actually watch the player play the game. Sometimes, they are even based on who is recruiting the player. If Notre Fame, Michigan, and Texas happen to be recruiting him, well, then he must be a five-star player…
…I could name dozens of players who have come through Virginia Tech with not many stars attached to their name and then left as all-conference or even All-Americans. We didn’t beat anybody to sign Shyrone Stith. We beat only Wake Forest to sign Lee Suggs. [Current Safeties Coach] Pierson Prioleau was a so-called two-star guy and he left here as an All-American. There were so many more.
Because all recruiting ratings are imperfect, I generally utilize the 247Composite rating, which aggregates bias by averaging out where each player stands across all the ratings services.
However, to better understand what these ratings actually tell us, let’s put 247’s in-house rating under the microscope.
What the 247 rating actually represents
On its rating explanation page, 247 states that their “ultimate goal is to accurately project college success with an element of raw ability baked in. To that end, we have turned to the NFL Draft as the measuring stick and answer key that best indicates, in a quantitative manner, the level of success and ability players showcased during their college careers.”
To be clear, the rating projects likelihood of draft position, which serves as a proxy measure of college success. This is a really important distinction because the college and pro games are very different.
For example, lots of quarterbacks and left tackles get drafted in the first round because they are very valuable - there are precious few players at those positions who can perform at the elite level required by the NFL.
Of the three players Beamer cited in his book when discussing recruiting ratings, here is where they landed in the NFL draft:
Shyrone Stith - 7th round selection in 2000
Lee Suggs - 4th round selection in 2003
Pierson Prioleau - 4th round selection in 1999
247 was not founded until 2010, but in general, were these three Hokie greats underrated coming out of high school? Yes. However, if the ultimate goal is to project NFL draft likelihood (and round) as opposed to college success, then they were not as underrated as many in Hokie Nation might think.
Not only is NFL draft status an imperfect proxy for college success, it is also an imperfect means of projecting future success in the NFL (which 247 is careful to state).
Regarding points of emphasis, 247 explains (emphasis added):
Due to our NFL Draft guiding principles, we take some cues as to trends and points of emphasis related to positional value and geographic origination. By our final Top247 and five-star release, we try to meet some standard thresholds for which regions typically produce the most players and which positions tend to get drafted at a higher rate and are valued higher than others.
Did you catch that? Players from talent rich areas that play high value positions benefit from additional weighting. No doubt, this phenomena has led to overrating certain players from the 757 in recent years who ended up being busts in college. It also really benefits players from Florida, Texas, and California - the hottest of hotbeds when it comes to recruiting.
Hokie fans have long speculated, and complained, about the propensity for recruits to see their ratings take off when they receive offers from a blue-blood school.
247 does not explicitly state whether or not there is a blue blood benefit, but this section of their explainer would appear to cover such a weighting scheme if it existed (again, emphasis added):
Based on exhaustive analytical studies of NFL Draft results and retroactive looks at NFL Draft picks as high school prospects, we have established position-by-position principles that are most predictive of college football success that we use to inform our position rankings.
Based on that text, the formula might apply a multiplier to the rating of quarterback recruits with Big 12 offers (since the Big 12 produces prolific passers who tend to get drafted early in the NFL). For offensive linemen, it might be recruits with Big Ten and SEC offers that get the multiplier boost.
While such a weighting system might improve the accuracy of the ratings regarding who is most likely to get drafted (and drafted early), it would be a self-reinforcing mechanism that distorts comparisons between players and teams.
Perhaps that is how we ended up with Alabama entering the 2023 season with the highest team talent rating in the country. Anyone with eyes could see that was not the case in reality. In fact, this year’s edition of the Crimson Tide was one of Saban’s least talented based on their on-field performance. (My eyes are still reeling from that USF game…)
In sum, 247 combines measurables and on-field performance to project the likelihood of each player getting drafted.
What could go wrong?
(Tune in next week to find out!)