In the great “stats vs. scouts” debates we’ve had over these many years, Saberists have often been labeled as arrogant know-it-alls. Some of it was warranted; some statheads were, and still are, too confident in fallible methods of objective measurement. However, it is typically grounded in the scientific method, by definition the least arrogant method of observation out there. For those, like me, who have long been separated from their last science class, the steps to the scientific method are:
- Ask a question. E.g. “Are strikeouts strongly related to a pitcher’s success?”
- Form a hypothesis. E.g. “Strikeouts are strongly correlated to a pitcher’s success in both the short- and long-term.”
- Make a prediction. E.g. “Strikeouts will be strongly correlated to a pitcher’s success at a statistically significant level.”
- Test your hypothesis. E.g. finding the strikeout rates and earned run averages of all pitchers who threw at least 150 innings in a season dating back to 2007 and determining the correlation between the two.
- Analyze the results. E.g. “There is a statistically significant relationship between strikeouts and same-season ERA. r = .87, p < .05.”
Above examples and figures are completely made up.
Adhering to the scientific method is the least arrogant way one can attempt to answer any of life’s questions ranging from the trivial to those of the utmost importance. It is admitting your fallibility as a human being, realizing that your perception and your memory are too unreliable to take any untested claim as truth. Of course, humans are just as fallible using their memory to provide key details as they are running unbiased, methodologically-sound scientific tests, so one should take any single test’s results with a grain of salt as well.
I bring this up because Sam Donnellon wrote an article for the Daily News yesterday that smacked of old school arrogance.
RUBEN AMARO JR. is right. Moving Chase Utley to third base so Freddy Galvis can play second will improve the Phillies in 2013.
He just can’t prove it.
This is the exact opposite of the scientific method: make a claim, realize there is no evidence nor is there any objective way to validate your claim, and then declare it correct anyway. “Kim Kardashian is right. When it rains, it is really just God crying about your sins. She just can’t prove it.” Belief in the absence of facts is not only foolish but incredibly arrogant, to state matter-of-factly that your experience and perception not only should but does match that of everybody else.
Donnellon spends some time in the article talking about the lack of reliable defensive statistics, and he isn’t wrong. Sabermetric defensive statistics are far behind offensive statistics in terms of reliability. Colin Wyers (@cwyers) of Baseball Prospectus fame has done yeoman’s work over the years pointing out the fallacy of relying on subjective classification of batted balls. In other words, one scorer’s line drive is another’s fly ball. Of course Donnellon’s criticism is much less nuanced than Wyers’, but it is easy to see where he is going. Being skeptical of defensive statistics is completely warranted.
What many old school types have done in the past, when confronted with a new statistic that they don’t trust, is to find the biggest outlier and use that as a justification to discard the statistic entirely and return to the older, more familiar (and less illuminating) statistics of the past. For instance, Wins Above Replacement has only recently become an accepted part of the mainstream baseball lexicon. It wasn’t that long ago that writers dismissed it out of hand because Mark Reynolds earned a higher grade, according to the statistic, than Ryan Howard. WAR certainly has its problems (reliance on single-season samples, subjectivity of human scorers, etc.) but just because it isn’t a perfect statistic, you don’t revert back to the problematic methods that got us here in the first place. It’s like saying, “Chemotherapy isn’t guaranteed to cure my cancer, so I’m just going to go back to using mercury.”
Later in the article, Donnellon romanticizes the old school scout types like Pat Gillick. There is certainly nothing wrong with saying that the subjective has a legitimate place in baseball; objective measurements have not and will never come close to encapsulating all of the known variables. However, it is clearly wrong to defer, unquestioningly, to them on all subjects. Pat Gillick is one of the brightest baseball minds the game has ever seen, but I’m always going to be skeptical of anything that he — or anyone else — says until it is tested empirically. To uncritically take a statement as fact is ignorant; to pass that deference off as an admirable quality is arrogance, and a particularly offensive arrogance that some old school types have displayed when their preconceived notions have clashed with Sabermetric revelations.
Towards the end of his article, Donnellon writes:
These are plays not measured in any comprehensible statistics, although those scripting for something called “The Fielding Bible” have given it an honest effort. But all baserunners do not have the same speed, guts or baseball IQ. Strength of arm is not just about putouts, but by a lack of attempts to run on that arm. They have tried to measure range, but these are not dynamic or even easily understandable stats.
Essentially, Donnellon says in the above paragraph that some smart people have attempted to quantify defense, but it is imperfect because it fails to consider the several listed variables. Donnellon takes the absence of evidence as evidence of absence (of quality), and uses that to revert to his preconceived notion that the eye test is still the best method of defensive evaluation. What he should have done, instead of lazily clinging to old ideas, is talk to John Dewan or any of the other smart minds who work on The Fielding Bible. If they weren’t aware of Donnellon’s criticisms already, they likely would have welcomed the criticism and considered it deeply. Or he could have talked to Sean Forman of Baseball Reference, UZR creator Mitchel Lichtman, or any number of other smart minds in the baseball research arena. People like Donnellon are important to the progress of Sabermetrics because many prescribe to the doctrine and thus are biased and less likely to be aware of methodological flaws.
Instead, Donnellon firmly entrenches himself in the soil of old ideas, closing out his article:
The good news for you people is that Amaro, whose resume is still without a championship and thus incomplete, seems to get it, too.
I can’t tell you how many runs will be saved in moving Utley to third and having Galvis at second for a season.
But my eyes can tell he’s right for doing it.
Human perception and memory have been scientifically proven, time and time again, to be unreliable and often simply inaccurate. Furthermore, their knowledge of how the human brain processes and retains information is often lacking. As a result, they give far too much credit to our ability to pick out, process, and remember key information. For instance, in this study, all 16 experts disagreed with the statement “human memory works like a video camera, accurately recording the events we see and hear so that we can review and inspect them later.” 63 percent of the 1,500 regular people polled agreed with the statement.
Rather, statistics and “the eyes” should act as tools in a tool box, ready to be used at any particular moment for a job that they are specifically useful in completing. Statistics, compared to the eyes, will be the best choice when trying to determine exactly how much less impactful Chase Utley’s offensive contributions will be at third base than at second base. On the other hand, the eyes will be the best tool for the job when you’re watching Roy Halladay throw a bullpen and notice something isn’t quite right. Even if Pitch F/X was available for bullpens, we may not be able to notice anything if velocity, movement, and release point remain unchanged, but a trained professional like Rich Dubee will be able to see that Halladay is laboring and shouldn’t make his next start.
As it happens, the more accurate our objective measurements become, the less reliant we are on experts who utilize subjectivity. Thanks to amazing advances in medical knowledge and technology over many years, we can quickly and easily run a test to see if a woman is pregnant. No longer do we have to inject urine into female rabbits to reach a conclusion. The progress of and increased reliance on Sabermetrics to answer questions is emblematic of yet another area in which the scientific method has increased our awareness and knowledge where none existed previously. That progress should be celebrated and anyone that tries to tell you otherwise is attempting to sell you swampland in Florida.
Those who have firmly dug their heels into the ground may learn from Plato’s “Allegory of the Cave“. (Warning: The video below has some repetitive flashing lights, which may be bad for those of you with epilepsy.)