Sam Harris on Statistical Concepts
Author, philosopher, and neuroscientist (what a combo!) Sam Harris appeared on the Joe Rogan Experience yesterday for a few hours of discussion. Towards the end, Harris and Rogan discussed some statistical concepts such as sampling bias, the hot hand fallacy, and probability. As this blog makes heavy use of Sabermetrics and statistical concepts in general, I felt that part of the discussion was quite enlightening.
You can watch the discussion below, or click the link for the transcript. Be warned that there is some strong language (not much), and a very brief discussion of religion, so use your discretion if you are easily offended.
Harris: A huge component and perhaps the component that explains all of those experiences is what’s called sampling bias, where you notice all of the hits and you don’t notice the non-hits. You don’t notice all the times you pick up the phone and you had no idea who was calling you, which is most of the time.[…]
The hits are salient to us and the failures aren’t. Religious people do this all the time: God always gets credit for the good things he does, but he doesn’t get scored for all the disasters he fails to prevent. A bus crashes and everyone’s dead except one little girl, and it’s God’s miracle that she walked away, but what about all the dead people?
And so the non-hits always outweigh the hits. The classic demonstration of this, which is still shocking to people, is called the hot hand fallacy in basketball: the idea that people get on a shooting streak. Michael Jordan would shoot two outside jumpers, when he goes up for his third, there’s this sense both in him and in the audience that he’s actually more likely to make that third shot having made those prior two because he’s just on a roll. So, that whole feeling of being on a roll in basketball has been studied because what’s amazing about professional sports like basketball is these statistics are every single game that has ever been played, it’s completely broken down. We know every shot, every basket, every rebound, so statisticians can just sit down and analyze: Is there such a thing as the hot hand? Is it actually true that if a guy has made three jumpers in a row, he’s more likely to make the fourth, or is the fourth truly independent of everything that’s gone before.
They’ve found that, despite the personal experience of being on a roll and despite the fact that we feel like we’ve seen someone on a roll, there’s really no such thing as being on a roll in basketball. Now, there may be other sports where it’s different, but there’s so much chaos and so much uncertainty, once that ball leaves your hand, it’s a low-percentage enough phenomenon that it’s actually insensitive to the fact that you feel great and everything’s going well for you and you just sank two baskets and you go up to sink a third, that third — you’re no more likely to sink that third than you were if it was your first one, or you just missed two and then you go up for the third.
Rogan: So, do you completely discount the player’s comfort, like when they feel like they’re in the zone, it doesn’t exist? [overlapping talk] So no one’s ever more likely to make a basket, is that what you’re saying? Even though they’re confident and they make the basket knowing they’re going to make the basket? Statistically, you say.
Harris: […] My understanding is, there’s zero evidence of what’s called the hot hand in basketball. Literally, it is a subjective error everyone feels powerfully. And so, given that, just map that on to all of our so-called psychic experiences where you feel like you knew someone was going to call.
Rogan: It’s just luck. How do you know, though? How do you know it’s just luck? How do you know it’s not either/or?
Harris: It’s testable. It should be testable.
Rogan: By placing the experiment on anything, don’t you automatically alter the results? Especially when you’re talking about something so ethereal as an idea of a friend, and then -boom- there’s an email in your box and you haven’t talked to them in years. I’m just playing devil’s advocate here, because I don’t believe that. But I don’t know that I don’t believe that, either.
Harris: Statistically, it seems completely beyond the pale, but again, there are going to be low probability events that happen every day. I know a guy who went, for instance, his college girlfriend broke up with him and went to Europe and he was convinced the relationship wasn’t over. He decided he was going to go to Europe to find her. He’s like 18, not a crazy guy, just a really romantic guy and it was not a scary break up. So anyway, nice story, they got back together. Only an 18-year-old can get it into his head to do this, he gets on a plane and he doesn’t even know which country she went to, so he just goes to Europe. He goes to Spain because he speaks Spanish. He’s like ten days into his trip, so he’s in Barcelona. Other tourists hand him a camera and say, “will you take our picture?” He picks up the camera and he’s taking their picture and into frame walks his ex-girlfriend. Within ten days he just found her, so what are the chances of that happening?
Rogan: Pretty fucking small.
Harris: But that stuff happens to people. It would be a miracle if low-probability events never happened. It would be a miracle if no one in human history ever had a story like that to tell. We’re in a system where there’s bound to be very low-probability events.
Rogan: Or it’s just not fully evolved and this kid went on his instincts. He tapped into the information of the universe and he knew instinctively where to go. He met his sweetheart like a goddamn Sandra Bullock movie.
Harris: If it’s an ability that anyone should care about, it should be testable. It should be strong enough that we can test it.
Rogan: But how can you test something when it’s a unique event that very rarely takes place, completely uncontrollable and it’s something that’s almost like a natural phenomenon that rarely occurs. How are you going to put something in a testing environment, the environment of a laboratory, and see if it really does make sense that you think about someone before they call you.
Harris: It might be Sheldrake that’s doing this, there’s the test of the phone call phenomenon. People can subscribe, they enlist like the five closest people in their lives to call them at random intervals. I guess they disable their caller I.D. or whatever.
Rogan: That’s still not a true moment because they’re acting on an experiment. How do you know that someone calling you might not just be calling because they tapped into this idea of loving you and this idea of missing you, and that is the tune that breaks through all the way to you and causes you to look at your phone right when they tune in. That’s not replicable. The idea of unique events isn’t disproven by experimenting with it. I don’t know that it’s true that sometimes you know when someone is calling, but I’ve had the experience myself, but I haven’t had it in years. It’s been years and years since I knew who was going to call me and I looked at it, but I do remember having it happen in my life several times. What if it’s something that just, you know, on a fucking summer solstice, the planets are aligned, your own biology, you have a certain amount of biology in your system and boom, there’s a certain amount of love and the thought gets through. Is that possible?
Harris: Sure, it’s possible. But the question is, even if it’s true, given this description, it doesn’t really seem to matter. It’s not a sign of how connected you are with the person, because sometimes you know it’s somebody you don’t even like. It’s not a measure of how crucial it is that they reach you at that moment because sometimes it’s completely trivial, and then other times when someone really had to get a hold of you because your dad was sick, they couldn’t find you, and you were just blissfully ignorant of the fact that the closest person in your life is having a medical emergency.
Rogan: No one is saying that it works all the time and that because it works in a blue moon that it can’t…
Harris: But then that’s not working.
Rogan: It’s not that it’s not working, it just doesn’t work all the time. It’s a rare event. Is it possible that there are rare events like that?
Harris: Given what we’re talking about now, this is all much more likely to be coincidence. You don’t have that many people in your life that you think about, you don’t have that many people in your life who call you. If you actually just run the numbers on this… one problem is that we have very bad intuitions of probability. Have you ever heard of “the birthday party problem”?
Harris: If I put you in a room with just random strangers, how many would I have to have in the room for you to be confident that two people in the room had the same birthday.
Rogan: Jesus. It would have to be thousands for me. And that would just be a guess.
Harris: So that’s basically everyone’s guess. “There’s 365 days in a year, we gotta get two people on the same day, that’s gonna be a lot of people.” The truth is, and this is just a fact, that if you get 23 people in the room, you have crossed the 50 percent threshold so that it’s more likely than not that two of them have the same birthday. 23 people. What’s crucial to understand is that, the crucial piece — and this deflates so many conspiracy theories and other bad ideas — is that it’s absolutely crucial that we didn’t specify which birthday. It’s any two birthdays. Any two people, any single birthday. That changes the game entirely. If I said, “how many people in a room to find one with my birthday?” that changes it. But any two people on any given birthday, it’s 23 people.
Rogan: If you go with just your birthday, that would be thousands. That would have to be thousands.
Harris: I don’t off-hand know what that would be. Our intuitions are bad about probability.
Those points apply to baseball as well. There is a lot of sampling bias involved in justifying belief in clutch hitting, for example. The ninth-inning walk-off will always register more strongly in your memory than a ninth-inning ground out that sent the game into extra innings. When people justify calling a player clutch, they rattle off all of his clutch hits, but don’t put it within the context of total chances nor do they consider other factors that may have been at play.
The hot hand fallacy gets some play in baseball, especially when players go on hitting streaks. Jimmy Rollins went 1-for-6 in the first game of what would become a 38-game-hitting streak in 2005-06. If you had asked anyone within the first 13 games if Rollins was dialed in, they most likely would have affirmed. However, Rollins actually put up a lackluster .254/.318/.356 line. Rollins wasn’t any more likely to get a hit then than he was in any other situation at that time.
Rare events certainly drive baseball narratives. The Red Sox slipped out of post-season contention at the end of the regular season last year, and it was blamed post-hoc on the team’s consumption of beer and fried chicken in the clubhouse. Other teams have thrown away their post-season hopes in worse ways than the Red Sox, and it had nothing to do with beer and fried chicken. They went 7-20 (.259) in September, which is bad. But if you swapped their September with June (16-9, .640), the beer-and-chicken narrative goes away. It’s certainly not impossible for a .557 team to lose 20 of 27 games in a given stretch. Some of them happen early, some of them happen late, but it’s only when it happens late do we really pay attention and assign meaning to the occurrence.