On The Future of Baseball Research

Earlier this morning, Adam Felder and Seth Amitin posted, in part, the results of a much-awaited study on the potential understated bias of the language of baseball television coverage at The Atlantic. When I made my thoughts on the subject clear here a few days ago, I was wishing desperately that this study had already been published, but now that it has, you can go read a little empirical justification for that thesis.

I don’t know Felder at all, and my interactions with Amitin have been limited to trading Dodgers jokes on the internet, so I’m not saying this out of a desire to pump up a friend, but you need to read that article. It’s important not only because of what it says, but because it represents of an underserved portion of baseball writing.

Most of you probably know this about me, but I spent three years as a political science grad student, and in that time I probably learned more about statistics, game theory and research methods than I actually did about politics, but I learned a great deal about what separates actual research from conjecture and speculation.

I think one of the best things about the advanced analytics movement in baseball is that it’s brought the rigor of social science research to sportswriting. It’s not perfect, but the average baseball fan knows way more about how to read statistics than he or she did ten or even three years ago. We’re slowly stamping out falsehoods based on preconceived notions whose factual underpinnings are either obsolete or nonexistent, and the positive effects of this movement cannot be overstated.

As scouting information gets democratized, as we debunk concepts like “clutch” and “small ball,” we’re replacing mythology with empirical study. I think this is, in part, why many former athletes and traditional sports media personalities hate advanced metrics and bloggers–they know the mythology and we’re killing God, so to speak. As someone who believes religion and science can co-exist in the real world, I think that creates a false choice when it comes to baseball, but that’s another story.

So why is this study so important? Because it’s empirical baseball research based on something other than game data. You can find enormous amounts of research based on game statistics, pitch f/x and BIS coding. And as much is out there, and as many conclusions as have been drawn by the public, you can bet that teams have even more.

But where we’re lacking, in my mind, is in qualitative analysis. Felder and Amitin’s study is still qualitative, but it’s based on coding of commentary, not box scores. That’s how we’re going to effect change–if media analysis is backed up with large-sample data from which we can draw meaningful conclusions.

Now, this study isn’t perfect. Even if all the concerns I have about their methodology (which is detailed in the post enough for a magazine article but not for a work of social science) are unfounded, what happens when you expand the sample? Or when you turn your attention to print media? Pre-game and post-game analysis? I buy the basic premise (partially, I fear, because I believed in their conclusions before the study), but it raises more questions than it answers. Which is kind of the point–you want knowledge that’s going to generate more knowledge.

So why don’t we have more work like this? Well, it’s absolutely not cost-effective. Game data leads to research that’s either valuable commercially (to ESPN, FanGraphs, Baseball Prospectus or whoever) or competitively to a team. But the only kind of qualitative data or media data that’s valuable (that we know of) is scouting data, and as much as I respect people who can evaluate young players and write coherently about them, I don’t think we’re drawing any scientifically rigorous conclusions there.

On the other hand, doing this kind of research right is expensive (it took upwards of $3,000 to fund this study) and requires people who know what they’re doing. As often as not, those people are doing real social science instead, or their work is stuck in academic journals and either unavailable to the public or off the beaten path. Make no mistake, it exists, but its effects aren’t showing up in places the average baseball fan is going to see it. I’m not sure what the solution is, but even though baseball produces more and better numbers than any other sport, we shouldn’t restrict serious baseball research to what we can count.

Leave a Reply

*

31 comments

  1. Paul Boye

    August 27, 2012 08:11 PM

    It’s also worth noting that there is a longer edition of the report mailed to some Kickstarter contributors that expounds on what was published in the Atlantic.

  2. Michael Baumann

    August 27, 2012 08:48 PM

    I imagine that’s where my methodology questions would be answered. Nevertheless, we need to do everything we can to encourage more research like this.

  3. Brian

    August 27, 2012 11:58 PM

    Definitely an interesting study. I do have a few things to nitpick. First, “he plays the game the right way”. Yes, the phrase is crappy cliche, and is often a broadcasters way of saying, “I have a super-big man-crush on this player. Isn’t he the best?”. It’s also used when a player demonstrates superior situational awareness and/or intangibles, like leadership, hustle, etc. The author states that “all players play the game the right way”. This is simply untrue. Whatever value one may assign to the statement (and its abuse), there is some truth to it.

    Second, and more important, I wonder how they determine the presence of bias, simply because the non-American players received less ‘intangibles’ praise. Maybe it was justified? If we want to determine bias, on average, do players. We need to know a lot about the players, the announcers, and the games monitored.

    Finally, what if there is a bias, but could it be explained by simple language barrier? Most Americans (which I assume the announcers are) speak just English. Could limited interaction be interfering with the broadcasters’ ability to craft narrative, as they are wont to do? Though, again, we’d need to know more about the announcers and the players.

    More than anything, the study leaves more questions than it answers – mostly about the value of commentary and role story telling in sports broadcasting.

  4. Brian

    August 28, 2012 07:58 AM

    I was really tired when I wrote that last night. Sorry if it’s unintelligible.

  5. Ben

    August 28, 2012 08:29 AM

    While I like that this stuff is being explored and I think there should be more of it, I do not think it was conducted in a very thorough way. Even though they took data from (I believe) 95 games and 200 broadcasts over a 1 week span, this is still too small of a sample as a lot of the praise/criticism relies on how the player performed over that one week stretch.

    Let’s just look at Zambrano since he was the most criticized player according to their study. He pitched one time during their data collection and gave up 8H, 8ER (including 5 HR) in 4.1 innings. He was also ejected that game. I don’t care what race/ethnicity a player is, if he posts that stat line he is going to be criticized heavily. On the other hand, Mat Latos (one of the other most criticized players) pitched 2 games with a combined line of 13 IP, 8H, 5ER. Not great but certainly good and he was still criticized (maybe because his team lost both games).

    I am aware that these are just two examples and I do not mean to say that their finding are definitely false (in fact I am inclined to agree with them). However, this research could have been conducted better. I hope they expand on it at some point.

  6. Jordan

    August 28, 2012 01:17 PM

    Do you know if they established any inter-rater reliability in the coding? At the surface, it sounds much like how we would do qualitative research in psychology.

  7. Michael Baumann

    August 28, 2012 03:22 PM

    The study isn’t perfect, and I never tried to say that it was. Seth at least has been very forthcoming about the limitations under which they operated. So while it’s legitimate to question the methodology, I think it’s churlish to denounce them (and I’m not saying anyone here has) for trying and not getting it completely right without acknowledging the effort they put in and how little obligation they had to do something like this at all.
    And the thing about social science research is that no project is ever perfect. Not even professionals with NSF grants, TAs and tenure on the line produce bulletproof work. Ever.
    My main point is that I appreciate Seth and Adam giving it a shot, and I hope it leads to more and better in that vein.

  8. CJ

    August 28, 2012 03:28 PM

    “MLB Announcers Favor American Players Over Foreign Ones”

    Are the announcers American? Are most of their listeners/viewers American?

    Not being dismissive, but I think that in 2012 we’ve reached a certain Concern Fatigue about issues that most of the rest of the world considers kind of normal.

    Do the foreign broadcasters show a reverse favoritism? Do the American Spanish-speaking broadcasters show a bias toward Spanish-speaking players?

    Are we being set up for “a very special episode” of Big Bang Theory to fix our bias?

  9. Mike

    August 28, 2012 03:37 PM

    I’m sorry, but anyone who interprets the phrase “plays the game the right way” as nonsense because no player runs clockwise around the bases either hasn’t played the game or hasn’t played the game the right way himself so he wouldn’t know what that meant. I don’t think this is a minor point – I think that lack of understanding puts the remainder of the article in question. For someone who is ultra-focused on metrics, whether a player runs around the bases clockwise seems like a strange metric to use to determine whether someone plays the game the right way. Batters who dive into first base (other than in response to errant throws in which the first baseman is trying to make a tag) don’t play the game the right way. Catchers who don’t block balls in the dirt but instead get lucky and catch the ball on the bounce don’t play the game the right way. Hitters that watch their home runs, especially when they are losing, don’t play the game the right way. Outfielders that routinely miss their cutoff men and allow base runners to advance a base don’t play the game the right way. I think you get the idea. I’m sure anyone that is playing baseball in the major leagues is *capable* of being taught how to do these things correctly, but that’s the point – they made it to the major leagues *despite* their inability to do those things and hence the use of the phrase “plays the game the right way” is reserved for those players that do those things correctly.

  10. Phillie697

    August 28, 2012 04:00 PM

    @Mike,

    Don’t know about you, but if “playing the right way” doesn’t end up producing measurable results, what would be the point? Would it matter if that’s the “right way” or not? The focus on statistical analysis removes those tendencies to hold onto old unproven beliefs, and instead focus on results; if there IS a right way to play the game, it ought to be reflected in the statistics, no? Otherwise I don’t care if holding a bat like you hold a hockey stick is the wrong way to play the game, if you holding that bat like a hockey stick results in you hitting 50 HRs; “wrong” way or not, it produced results.

    If you just want to arbitrarily label something as “playing the right way,” then perhaps you can be a judge in a baseball “beauty” contest. Otherwise, what’s the “right” way to me is whatever that helps the team win the most while playing within the rules, and that’s measurable. Otherwise, it’s just a bunch of old men talking about a bunch of younger men because the old men can’t play anymore. I’m sure many things are the “right way” because they DO in fact produce results. If that’s the case, then why just talk about it being the “right” way? How about “doing this saves/drives in X runs?” No one will question if that’s the right way with that kind of results-oriented approach.

  11. CJ

    August 28, 2012 05:28 PM

    “Don’t know about you, but if “playing the right way” doesn’t end up producing measurable results, what would be the point?”

    Phillies607, I appreciate the increased use of stats and dissing older folks as much as the next man, but that’s a pretty broad implication. I don’t any coach would lecture a batter hitting 50s home runs about “the right way” to bat. Coaches have been tolerating odd batting styles and pitching motions for years, as long as they produce results.

    I think most of the “right way” talk (hitting the cut off man, running out hits, etc) is measurable, no?

  12. Mike

    August 28, 2012 06:45 PM

    @Phillie697,

    I guess all I can say is that I would love for my opposing coaches to have your point of view – their batters would dive into first base, their catchers wouldn’t know how to block a ball in the dirt, and their outfielders would miss their cutoff men because these coaches wouldn’t have a study to cite to prove to them with scientific measurable results that those approaches are fundamentally incorrect. I have no clue how many runs are saved by catchers that properly block balls that are in the dirt compared to catchers that don’t, but if your intuition doesn’t tell you that a positive difference exists then I hope that you aren’t coaching baseball.

    But, don’t worry, I wouldn’t run up the score on these coaches because I know how to play the game the right way. I would, however, instruct my pitcher to drill the next batter in the ribs (not in the head) if one of their players stood at home plate too long watching his home run. You seem like a numbers guy – what’s the probability that someone gets drilled with a pitch after a teammate watches a home run too long? That’s measurable isn’t it?

    In any case, if these fundamental baseball concepts aren’t resonating with you (and I used four specific examples, none of which included a batting style, which you tried to use as a counter example) and are being confused with some sort of baseball beauty show then my guess is that you haven’t been coached to play the game the right way, but I don’t have any data to prove that either.

  13. jauer

    August 28, 2012 07:41 PM

    Mike,

    And if I managed against you, I would instruct all my batters to admire their fly balls, so that they get a free HBP their next time up. The right way to play the game: giving the other team a free base when they slightly bother you.

  14. jauer

    August 28, 2012 07:46 PM

    “Catchers who don’t block balls in the dirt but instead get lucky and catch the ball on the bounce don’t play the game the right way. Outfielders that routinely miss their cutoff men and allow base runners to advance a base don’t play the game the right way.”

    These aren’t people who are playing baseball the “wrong” way. These are just bad baseball players.

  15. CJ

    August 28, 2012 08:19 PM

    “These aren’t people who are playing baseball the “wrong” way. These are just bad baseball players.”

    Actually, players who miss cutoffs are often good baseball players. Talented with strong arms. They’re not bad players, they are doing the wrong things.

    Why does it make some fans nervous to note that there is a right way and a wrong way to play baseball, unlike every other endeavor in life?

  16. Brian

    August 29, 2012 08:11 AM

    “Plays the right way” also usually refers to someone that plays with an ounce of stoicism – the guy that doesn’t do a bat flip and admires his home run. The guy that will slide wide to break up a double play, but won’t run over a catcher when unnecessary. AKA – not an asshole. Is there a way to measure that?

  17. Scott G

    August 29, 2012 08:14 AM

    I don’t know about any of you, but I sometimes get the feeling that when commentators (Chris Wheeler), mention a non-white athlete “respecting the game”, it’s an unnecessary remark. I hear him say that Jimmy Rollins respects the game, but not Chase Utley. I bet Chase Utley respects the game. To me, the need to make that statement is a form of negative towards Rollins. Why wouldn’t he “respect the game”? Because he’s non-white?

  18. Phillie697

    August 29, 2012 10:43 AM

    @CJ,

    What makes us nervous isn’t that there is a right way or wrong way to play the game. What makes us nervous is when you label one as right or wrong without any explanation. Some things may be intuitive, others are not. For example, I don’t need any proof that 1 + 1 = 2, but if you’re going to tell me the square root of 489574 is 699.696 (rounded), then sorry, I’m going to have to bust out a calculator and check if you’re correct. My problem with any of the people who likes to call something the “right” way is that they object anyone who wants to challenge their notion of what’s the right way. What you think is right may not 1) actually be right, 2) may not be right for someone else, and 3) people’s opinion may differ. THAT is the problem with calling something the “right” way.

  19. Phillie697

    August 29, 2012 10:50 AM

    @Brian,

    The VERY first HOFer we have in the Baseball Hall of Fame is a dude who was universally acknowledge as an asshole and who didn’t play with even a hint of a notion that he wants to be the “good guy.” Beat that with a stick.

    Not saying it’s the way I would want my sons and daughters to play baseball, but please, don’t call something the “right” way to play when it is clearly just a matter of choice/opinion.

  20. Richard

    August 29, 2012 11:45 AM

    “playing the game the right way” seems to imply, to me, crap like “going the other way”, bunting, “giving yourself up”, putting the ball in play… which themselves are made to imply other actually important stuff like hitting cut off guys and being a good baserunner

    Polanco plays the game the right way, in this calculation, Howard does not. Use of Polanco in that last sentence notwithstanding, those skills all too often are praised in scrappy white bench/utility players. (Or, rather, scrappy utility types have more often been white.)

  21. Brian

    August 29, 2012 01:06 PM

    @Phillie697,

    That’s a good point, but I don’t think anyone’s ever said that Ty Cobb “played the game the right way”. He’s even quoted as saying, “When I began playing the game, baseball was about as gentlemanly as a kick in the crotch.”

    When I hear that phrase uttered, it’s usually in reference to the idea that baseball is, or was (in some vague time past), a “gentleman’s game”.

    Though as this thread shows, there’s plenty of other definitions.

  22. Phillie697

    August 29, 2012 01:16 PM

    @Brian,

    Well, apparently not even back then did they consider it the “wrong” way of playing baseball, as they voted the guy in as the first HOFer of the whole shebang.

    Baseball has always been a mix of pure talent vs. hard work. We like to call certain things “the right way” because we’d like to believe that it’s hustle that makes someone successful, not because he can hit a ball 500 feet without even trying while we can barely muster 3 HRs a season in our high school teams with 320 feet outfield fences. Like jauer said, sometimes, there are just good and bad baseball players. I don’t feel inclined to knock someone who can produce 7 WAR a season but doesn’t feel like legging out a routine ground ball or admires his HRs. Ty Cobb certainly didn’t give a rat’s ass.

  23. CJ

    August 29, 2012 02:16 PM

    Phillie697,

    OK, the “right way” can be hard to measure. And there are exceptions, and disagreements about certain details. Just like in many areas of life. But there is still a right way and a wrong way. I mean, there is nothing wrong with the term. Just how it is sometimes applied, is my point.

  24. CJ

    August 29, 2012 02:21 PM

    “The guy that will slide wide to break up a double play, but won’t run over a catcher when unnecessary. AKA – not an asshole. Is there a way to measure that?”

    Brian,

    Measuring assholery is like defining obscenity, per Justice Potter Stewart:

    “I shall not today attempt further to define the kinds of material I understand to be embraced within that shorthand description ["hard-core pornography"]; and perhaps I could never succeed in intelligibly doing so. But I know it when I see it.”

  25. CJ

    August 29, 2012 02:25 PM

    “I hear him say that Jimmy Rollins respects the game, but not Chase Utley. I bet Chase Utley respects the game. To me, the need to make that statement is a form of negative towards Rollins. Why wouldn’t he “respect the game”? Because he’s non-white?”

    Scott G., I think it’s cultural as much as anything. Maybe generational. Sarge Matthews talks about “the way the game is supposed to be played” all the time. And I’m pretty sure it’s been said about Utley too.

  26. Mike

    August 29, 2012 02:59 PM

    I think the disconnect between the two camps might be from this source: coaching vs. calculating/plotting.

    I’m coming from a coaching perspective – if one of my players hits a home run and watches the ball too long then he’s going to get an earful from me. If Chase Utley hit a home run and watched it (which I don’t think that he would) then that would be fine with the people that are calculating his WAR numbers, but it wouldn’t be fine for me because some kid will be watching him and will want to imitate him.

    With catchers and blocking balls in the dirt, I can evaluate whether a catcher “plays the right way” by how he blocks the ball on pitches in the dirt. For others, they might not have any idea about this skill and have no way to measure success with this skill so, by definition, it makes no sense to them to even consider it to be a part of their “plays the right way” equation. I have often heard announcers say “good catch” about a catcher who caught a ball that bounced in the dirt. The problem is that a catcher *shouldn’t* catch the ball in that situation. The catcher just got lucky and over time will miss more than he catches. The announcers simply don’t know what they are talking about. Now, I need to coach a catcher how to block a ball in the dirt, and I don’t have any data to back me up. Should I just throw up my hands and tell the kid that I don’t have any data so he should just take a random approach?

    Like I always tell my son – If he makes the major leagues and can hit the ball standing on his head then I might not like it but he will have the right to do so, but for now, he is going to “play the right way.” For me, the litmus test for that phrase is whether I would want a kid to imitate the technical, not probabilistic, aspects of the play. If so, then I will have no shame in saying that someone “plays the right way.”

  27. Sean

    August 29, 2012 03:04 PM

    It’s probably way late to get into the whole Jimmy Rollins and groundballs debate, but I feel like contributing my own experience, so if you’ll indulge me for a minute or two. My father (who I have caught engaging in racial stereotyping in the past) complains almost incessantly about a perceived lack of effort from Rollins. He talks about being a bad influence; about having to be ready for an error. He goes on and on.

    This almost always leads to him contrasting Rollins to other players, who he perceives as putting in a “full” effort. For a long time, this was pretty much just Chase Utley, and if the story ended there, this wouldn’t be worth typing about. But this season, my father has found a new guy on the team who he sees as always giving one-hundred percent. Someone he has, for months now, excitedly called his “favorite Philly in a long time,” even above Chase Utley. A man by the name of Juan Pierre. He can’t say enough positive about Juan Pierre’s effort. He knows Juan Pierre is not some kind of team-carrying All-Star, but he’s been completely won over by Pierre’s constant “hustle.” It’s almost funny when he says he “knows [Pierre] can’t really play defense anymore, but he does all the little things right.” He’ll mention at least once a game how Juan Pierre and Chase Utley “play the game right,” and how he’d like the Phillies to go get more people who “play the game right” this off-season. (He’s also mentioned that he likes how Domonic Brown plays on offense. Just not nearly as often as Pierre. And he used to complain about the way Hunter Pence “played the game” too, so hey.)

    So I’m not convinced unconscious racism is the whole picture with Rollins. I’m not saying these biases don’t exist in baseball, but (and maybe I’m being some mix of naive and optimistic) I think for many it’s not a factor in Rollins’s case. They just have somewhat irrational ideas about the importance of things like running out grounders. Which might be more offensive than racism. (Kidding.)

    As a final note, it’s true my father gave Cliff Lee a pass when he didn’t run one out last year because he’s a pitcher, but he sees pitchers as ultra-fragile creatures, and always cringes at the sight of one anywhere on the basepaths. I’m sure he’d give Antonio Bastardo or Jose Contreras the same pass.

  28. CJ

    August 29, 2012 03:20 PM

    Sean,

    Nice input.

  29. Scott G

    August 29, 2012 05:33 PM

    Mike,

    Isn’t this about pro players who CAN practically hit the ball standing on their head, and do pretty much every other skill? Unless missed something, this isn’t about determining the influence on non-pros.

  30. Mike

    August 29, 2012 08:15 PM

    Scott G,

    But the non-pros today will be the pros in the future, and the influences that the non-pros have today will cause people in the future to comment on whether they play the game the the right way or not. The implication of that phrase is that the player was not taught correctly (or didn’t listen well) as a non-pro. A batter who dives head first into first base thinking that he is going to get to the base quicker probably didn’t start doing that as a pro, but rather as a non-pro.

  31. Scott G

    August 29, 2012 08:24 PM

    If you honestly think that what players do have a serious bearing on younger players your crazy. I think I stopped mimicking ken Griffey when I was approximately 8. Also, the pros are pros for a reason, so unless they’re doing something explicitly bad/wrong, who is anyone to say its not the right way to do something? You even said that before. If players are in the pros, they must be doing nearly everything right most of the time.

Next ArticlePhillies Call Up Tyler Cloyd