I long ago lost my faith in the quality of Bon Jovi's work.
Yet I can't recall the exact game I stopped believing in NFL statistics. Perhaps it's best if I tell you when I began to lose my faith.
It was last year's game when the Broncos were pulverized (yet again) by the Baltimore Ravens. You probably remember the statistical headlines from the game. Kyle Orton threw for over 300 yards and Brandon Lloyd went for more than 100 yards receiving. I had already posted my Gut Reaction to the game, which like all Gut Reactions, are usually posted within fifteen minutes of a game's finish. It was well-written, yet there was something that still haunted me.
So I hit the game tape over and over again. It didn't take more than a few series to see the horror that was Haloti Ngata.
Ngata was dominant that day, even though he finished with only three tackles. Play after play he dominated the Broncos' offensive line. Ngata got gap penetration; he used the same swim move what seemed like half a dozen times (which I promptly taught to a pee-wee football team the following week); he pushed the line back into the backfield two or three yards; he was a nightmare that, to this day, probably causes Ryan Harris to wake at night in cold sweats.
I had already known that stats often hid the truth in a game requiring eleven giant men to synchronize their movements against another eleven. Still, it was this game that was the Marcus Nash/Jarvis Moss moment for me. Ray Lewis, Joe Flacco, and Ray Rice were getting the stats.
But Haloti Ngata was the guy beating the living hell out of everyone.
At the same time I reread the Nassim Nicholas Taleb classic Fooled By Randomness, except this time I actually read it. The primary message of the book was quite profound: the more complex a system, the less likely we can describe it with models (i.e., statistics). While reading the book made me realize I might have wasted money getting an MBA in Finance (I had always been slightly annoyed that my financial models failed to predict the movement of securities), I was more than happy to apply Taleb's thoughts to the NFL.
Stats couldn't really account for Haloti Ngata. He was disrupting plays, yet he wasn't credited with a tackle. He was forcing double teams so that others could get a sack. He was eating up space so that Ray Lewis could sell body wash.
I began focusing on tape, which was more interesting anyway, which led to the the popular Playbook Abides series. It opened my eyes and took me back to my playing days. Rather than focus on the numbers, I began to see what the players were reading and what they were doing. If Ryan Clady got beat on a spin move and forced Kyle Orton to throw the ball away, I knew there was an explanation for that incomplete pass. If Orton correctly identified the Mike linebacker on a pre-snap blitz read and Knowshon Moreno still missed his blocking assignment, I had a reason for the sack. In short, I began to at least attempt to put the numbers into context.
Further, I realized all of the biases I had been bringing to my statistical analysis:
- Confirmation Bias - I was using stats to confirm my own beliefs about the Broncos. For example, I could easily quote Kyle Orton's stats on 3rd down rather than watch the tape if I wanted to push my personal view that Orton was a bad option for the Broncos (as a side note, the tape supports the numbers).
- Outcome/Hindsight Bias - I was judging decisions the Broncos made by their outcome rather than their context at the time. This lead to an eventual weekly piece called Huge Decision, in which I tried to evaluate weekly coaching decisions in context rather than their eventual result. So that 4th and 2 that the Broncos failed on? Rather than assume it was a mistake, I attempted to see if the numbers supported the decision at the time it was made.
- Selection Bias - We've all made this mistake. Pick a set of data that supports your view and ignore the rest.
- Illusory Correlation - Here, I'd try to find a correlation coefficient between two events and hint at a strong relationship. I finally stopped doing this when I realized that there was absolutely no correlation between teams coming off a short week and their records the following week. But I so badly wanted to explain what I had mistakenly perceived as a pattern (as a side note, it doesn't take stats to find a correlation between Raiders fans and dropout rates).
The list could go on and on (counting stats bias and sample size bias are two more), but there's simply no reason to point out all of my flaws. I plan to use statistics in the future, after all. Let's hope I use them in a better way.
One of the reasons that I've become a convert of Brian Xanders recently is that he at least has a feel for how to use numbers. From his interviews, he spends a lot of time with tape (which I've made fun of from time to time), but when it came time for the draft, he wisely stockpiled quality picks, which improved the Broncos' chances of landing quality starters. Although this might seem obvious, many teams simply junk their drafts trying to move up to get players. Those teams, as Taleb might say, will benefit from more randomness.
The same sort that had the Broncos move up in the 2006 Draft to get Jay Cutler. One pick later, the Ravens took Haloti Ngata.
Jay Cutler has a higher quarterback rating; Haloti Ngata has more wins.
Bon Jovi? Who cares, I just needed a creative title.
TJ Johnson can be reached through telegraph, ESP, Spanish interpretor, or via email: firstname.lastname@example.org. Follow him on Facebook and Twitter if you want to see him mock "the man." He assumes you are following It’s All Over Fat Man on Facebook and Twitter, but if you are not, that’s nihilistic, man.