Kerry Byrne can cherry-pick with the best of them

Yesterday we again decried ESPN's attempts to paint their own Total QBR as a revolutionary passing metric. Another day, another gross instance of statistical malfeasance, this time delivered by CHFF's Kerry Byrne in a column for SI. It's pretty bad, quite frankly - virtually a page out of the playbook for how to misinterpret and overstate stats and their meaning.

The spirit of what Byrne and CHFF are trying to do (factor rushing into a QB rating) is excellent, and in full disclosure it's something I've also been working on since last year myself. Yet, the manner in which Byrne is presenting the data for Tebow's 2011 starts is completely self-serving and ignores some crucial context. Let's examine some of the more glaring fallacies of Byrne's column:

Byrne: Tebow's Broncos are winning because he consistently outperforms the opposing quarterback...And clearly, the defense has improved dramatically in recent weeks, either purely as coincidence or as a by-product of the fact that Tebow has helped the team improve in all areas by protecting the football.

I'm going to stick with the traditional QB Rating here just to prove a point, even if it's far from a perfect measure of quarterbacking. As we all know, the Broncos started out the season 1-4 and were frankly getting their asses kicked. There was no mystery as to why this was happening - the team's D was getting annihilated via the pass. During the Broncos' first five games (aka The Kyle Orton Era, 2011 Edition), the team allowed opposing QBs to tally an otherworldly 106.5 QB rating to go along with 10 TD passes, 49 rushing yards and four rushing TDs. Tom Brady's 2011 QB rating (second only to Aaron Rodgers) at this point is a phenomenal 105.1, which is just a hair below what the Broncos were giving up.

Since then (Tim Tebow Era, 2011), the Denver D has allowed opponents an 85.0 QB rating (with nine TD passes, plus 47 rushing yards for zero scores), which is still decent and a touch above the league average of 81.9 - but it's also a full 21.5 points below that earlier figure. To recap, that's 14 QB scores (2.8 per game) and a 106.5 rating in five games pre-Tebow, and nine QB scores (1.5 per game) and an 85.0 rating in six subsequent games. 

In those same six games, Tebow has compiled a 78.9 QB rating along with 418 rushing yards and two touchdowns. Personally, I think it's fair to say that Tebow has outplayed his counterparts over the past six weeks when adding in his impact in the rushing game, Matt Stafford excluded. If you'd like to interpret all of this as Byrne does (This superior play is the No. 1 reason for Denver's sudden success -- now 5-1 with Tebow at QB this year after a dismal 1-4 start.), then be my guest.

Byrne: Here's the Correlation to Victory of several notable indicators through Week 12 of the 2011 season. Real QB Rating -- 156-20 (.886)

To be honest, it's completely clear that Byrne has no clue what correlation is in a statistical sense. If Byrne wanted to do this properly, he would regress winning percentage to his Real QB rating like TJ did here with ESPN's Total QBR and then determine, through correlation and r-squared (known as correlation coefficient and coefficient of determination) if the rating system could describe a real relationship. Instead, Byrne has apparently just taken the Real QB Ratings from each game individually and tallied games as either "Wins" or "Losses" for CHFF's metric. 

Byrne's method of "correlation" appears to give each game equal weight, which is completely wrong when it comes to actually correlating statistics. A true correlation would measure the weight of every pass (or rush, in Real QB Rating's case) and its effect on a team's chances of winning. But with Byrne's faulty method, he's claiming that the eight passes and nine carries Tebow had in Kansas City a few weeks ago (for 112 total yards, two scores) and the 54.8-point Real QB Rating advantage he held over Matt Cassel and Tyler Palko in that game are the reasons the Broncos won at Arrowhead.

Put another way, Byrne is equating the importance of Tebow's 17 touches (pass attempts + sacks + rushes) at KC with the performances of Aaron Rodgers (47 touches, 432 net yards, six TDs) and Stafford (33 touches, 279 net yards, three TDs) in their respective teams' beatdowns of Denver, as far as the weight of their individual contributions go. Tebow's impact on the Broncos winning in KC equals that of Rodgers' effort against Denver, by Byrne's method of "correlating" things. This is mathematically unsound and couldn't be farther from how correlations actually work. Again, if you think Byrne is standing on firm logical ground there, it's a free country.

BTW, did you notice that according to traditional QB rating, Tebow was only severely outplayed by Stafford, and slightly so by Mark Sanchez? What, pray tell, is Real QB Rating really adding to the discussion here? Sure seems like a bit of thinly-veiled self-promotion, Worldwide Leader-style.

Byrne: There are two underlying reasons why Tebow is so effective, two reasons that explain his impressive Real Quarterback Rating week after week. He gets the ball in the end zone more often than any QB in football today...He produces touchdowns at an amazing clip, better than any quarterback in football in his brief career...

Career percentage of touches that result in a TD:
Tim Tebow -- 6.0 percent
Aaron Rodgers -- 5.7 percent
Peyton Manning -- 5.5 percent
Tom Brady -- 5.1 percent
Drew Brees -- 4.7 percent
John Elway -- 3.9 percent

Tim Tebow has started nine games in the NFL. This is a small sample. You know what happens with small samples? They are quite easily distorted and manipulated by outliers and luck. But, that's not even what's happening here. As you might potentially recall, Tebow wasn't the starter for most of last season. But if you're new around here, allow me to briefly explain: while Orton was still the starting QB in 2010, Tim would occasionally come in as part of the Broncos' short-yardage and goal-line set, or "Tebow Package." This happened a total of 13 times - one pass (a touchdown) and 12 rushes (including three TDs). That's four touchdowns in 13 short-yardage attempts, and those four scores covered 10 yards combined. Orton & Co. drove the field, and Tebow punched it in from 5, 1, 1, and 3 yards out. Do you think that does anything to help predict whether Tebow will help the Broncos score more touchdowns in the future? Unless they shrink the field to measure about 10 yards, I'm going to say the answer is no.

Now, if we take out these 13 special plays and simply focus on Tebow's actions as the primary QB, we get a TD rate (passing and rushing combined) of 5.1% over his career, and a rate of 4.7% so far this season. These are still very good numbers (better than Elway's), and they still compare favorably with the esteemed group that Byrne cites. Yet, why didn't Byrne take out those "Tebow Package" plays? Is it because it doesn't serve the impact of his column well, and he wouldn't be able to proclaim that Tebow scores more often than any QB in football? Once again, I'm not going to tell you how to interpret all of this. If you think it makes sense to include Tebow's goal-line plays from 2010 in trying to figure out how often Tim gets the ball into the end zone, have a party.

Byrne: But the Broncos are winning not just because Tebow protects the football, but because he protects it better than any QB in the game today. Here's how he stacks up against some of the more prolific QBs in the game today.

Career interception percentage:
Tim Tebow -- 1.78 percent
Aaron Rodgers -- 1.83 percent
Tom Brady -- 2.2 percent
Drew Brees -- 2.71 percent
Peyton Manning -- 2.75 percent
John Elway -- 3.1 percent

Add in that Tebow has lost just one fumble in his career (with four INT) and his turnover rate is an incredibly miniscule 1.4 percent.

Here we go again - making grand pronouncements about Tebow's effectiveness as it compares to all-time greats and today's undisputed best quarterback. Makes for flashy headlines worth emailing your friends and citing in your conversations during the game on Sunday, right? Again, Byrne is playing fast and loose and jumping at the chance to say something nobody else is willing (wreckless?) enough to write themselves. I agree with Byrne that Tebow has done a great job of keeping the ball away from the opposing team so far. However, it's foolish to point to Tebow's one lost fumble as proof of that - it's been firmly established that fumble recoveries are a matter of pure randomness (by Brian Burke and Football Outsiders, among others), and as Burke writes, "fumbles, and not fumbles lost is a better stat for estimating a team's future likelihood of fumbing."

In other words, Tebow's one lost fumble out of six does not in any way suggest that future fumbles by TT are more likely to be recovered by Denver than with any other player's fumbles. Rather, Tebow and the Broncos have been quite fortunate to this juncture that they've not turned it over more on his fumbles. This luck will turn at some point. But, if you're buying Byrne's claim that Tebow is historically great at playing keepaway after 225 career pass attempts, and his one lost fumble out of six evidences a repeatable skill, then go for it. Mind you, just one more INT would have raised his rate to 2.2% and two more picks would make it 2.7% - I'm only pointing this out to express just how small of a sample we're dealing with here and the danger of assigning historical significance to it.

Finally, Byrne does go on to write,

We're not trying to extrapolate too much out on Tebow's career. Clearly, he's played only a handful of games. The other passers on those lists played over the long haul. A lot can change between here and the end of his career -- whenever and wherever that may come.

Sounds like a nice "small sample size" disclaimer, but did you notice how it comes at the end of the article after multiple proclamations that Tebow is doing things on the football field better than many of the greatest players in NFL history ever did and are doing? The damage has already been done, because if you're buying into the ludicrous assertions Byrne makes throughout his column, no two-liner thrown in at the close is going to bring you back to reality.

Oh, and Kerry - are you out of your @#$%ing tree? Foie gras is delicious.

Doug is IAOFM’s resident newsman and spelling czar. Follow him on Twitter @IAOFM

Stats That Don't Lie

2014 Offseason

Offseason coverage