Originally posted 11 Feb 20
Here is an excerpt:
My colleagues and I, we’ve been spending a lot of time looking at medical decision-making. Say you walk into an emergency room, and you might or might not be having a heart attack. If I test you, I learn whether I’m making a good decision or not. But if I say, “It’s unlikely, so I’ll just send her home,” it’s almost the opposite of survivorship bias. I never get to learn if I made a good decision. And this is supercommon, not just in medicine but in every profession.
Similarly, there was a work done that showed that people who had car accidents were also more likely to have cancer. It was kind of a puzzle until you think, “Wait, who do we measure cancer in?” We don’t measure cancer in everybody. We measure cancer in people who have been tested. And who do we test? We test people who are in hospitals. So someone goes to the hospital for a car accident, and then I do an MRI and find a tumor. And now that leads to car accidents appearing to elevate the level of tumors. So anything that gets you into hospitals raises your “cancer rate,” but that’s not your real cancer rate.
That’s one of my favorite examples, because it really illustrates how even with something like cancer, we’re not actually measuring it without selection bias, because we only measure it in a subset of the population.
How can people avoid falling prey to these kinds of biases?
Look at your life and where you get feedback and ask, “Is that feedback selected, or am I getting unvarnished feedback?”
Whatever the claim—it could be “I’m good at blank” or “Wow, we have a high hit rate” or any sort of assessment—then you think about where the data comes from. Maybe it’s your past successes. And this is the key: Think about what the process that generated the data is. What are all the other things that could have happened that might have led me to not measure it? In other words, if I say, “I’m great at interviewing,” you say, “Okay. Well, what data are you basing that on?” “Well, my hires are great.” You can counter with, “Have you considered the people who you have not hired?”
The info is here.