Hello, you (shall we walk in a sample’s shoes?)…Understanding Science Studies in the News

from Family Inequality

 by Philip N. Cohen

I don’t remember who, but someone in my 1980s radio career once advised me to speak as if I were addressing only one person, not the whole audience. So, it’s not, “Hello everybody,” it’s, “Hello, you.”

Today’s science reporters face the challenge of getting through to both editors and audiences, and I don’t envy them (well, I do envy them, but not because their jobs are easy). One tool of the trade is the personalized interpretation of science results. It’s a shame, because it mushes up the science in ways that probably contribute to common problems of scientific illiteracy.

The attention-grabbing “you” in science reporting headlines is the problem I’m concerned with here. It often adds an unrealistic level of certainty to science results.

Case in point is the widely circulated Op-Ed in the New York Times unfortunately titled, “Homophobic? Maybe You’re Gay.” The piece drew from a study that found a positive relationship between “implicit” homosexuality and homophobic attitudes among people who say they are 100% straight. Implicit homosexuality was determined by tripping people up with subliminal messages (“ME”) flashed on the screen

I read it as carefully as I could for a study outside my expertise, and nowhere does it say what proportion of homophobic people are actually driven by such repressed sexual orientations. It’s interesting that it happens at all, and it’s plausible, but does it “cause” any significant amount of the widespread anti-gay attitudes and actions we see? Maybe. For what it’s worth, if I read it right the variation in homophobia explained (R-squared) in the experiments was in the neighborhood of 5%.

Another recent study drew headlines such as, “Why your doctor should be trim.” The actual finding in the study – which is quite dramatic – was this:

physicians with normal BMI were more likely to engage their obese patients in weight loss discussions as compared to overweight/obese physicians (30% vs. 18%, P = 0.010).

But think about it. Weight loss is something that doctors should at least discuss with their obese patients. But the great majority of doctors don’t — 70% or 82% – according to the study, depending on their weight.

This could be a social problem. But should this study be used to advise patients to seek thin doctors? If you want a doctor that discusses weight loss with you, it’s much more efficient to ask a doctor to discuss it with you than it is to shop for a thin doctor (and hope you get one of the few that discusses weight loss). And if your fat doctor won’t discuss weight loss, I recommend finding a new doctor (of any weight), rather than trying to get your doctor to lose weight!

In other words, turning the study results into personal advice ends up turning patients away from what matters — their relationship with their doctor, which they can directly observe and act upon — and toward a superficial feature (weight) that might or might not contribute to it.

Here are some other recent examples:

  • Why your left side is your best side,” from a study showing more positive average reactions to left-sided portraits (even when they’re reversed to look like right-sided portraits) — but the “why” in the analysis is speculative.
  • Why your kid isn’t creative,” about a new book that “synthesizes the latest scientific research into creativity” and concludes that “our education system and social mores discourage creativity.”
  • Back pain on the job? It could be your bad attitude,” from a finding that 1 out of 5 people who went to the doctor for back pain had a “persistent” condition, and those 1-in-5 were more likely to have negative attitudes about it. (The other 4 out of 5 may have just hurt their backs, and their attitudes weren’t discussed.)

Show me the distribution

Part of my frustration is with psychology studies in particular, which often don’t include a simple cross-tabulation or descriptive table that allows the reader to assess the overall pattern and the strength of the relationship. Any statistical association can be pitched as today’s must-read take-home message. But how big a deal is it in individual real life?

Take education and income, which are certainly strongly, causally related. Here is data from 40 random women, drawn from the 2010 American Community Survey:

This variable alone accounts for 17% of the variation in earnings among these 40 women, and the relationship is highly significant statistically. That strong a relationship is unusual in the typical flow of newsworthy social science studies. Yet look how much variation there is around the line.

If you only knew one thing about a woman and had to guess her income, education level would be a good place to start. But when it comes to personal advice — to understanding our own lives — we know so much more than that. Only then — for example, after we have narrowed down the pool to college-educated, professional suburbanites between 40 and 50 years old — do the studies about women’s negotiating abilityshoe style, and so on, make a big difference. Marginal things matter, but for most people they’re not the causal story. Put another way, they matter more for populations than they do for individual readers.

This reminds me of Mitt Romney saying to that college student who was worried about the economy:

What I can promise you is this: when you get out of college, if I’m elected, you will have a job. If President Obama’s re-elected, you will not be able to get a job.

The unemployment rate is 4.9% for people with a BA degree. How much does Romney think it will vary according to the presidential election outcome? Or, put another way: “Unemployed? It could be your president.”

About these ads

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s