When ‘common sense’ programs backfire

When ‘common sense’ programs backfire

What do these two teenagers have in common? Kayla is a junior in high school and is spending several weeks feeding, caring, changing diapers and acting as a mother to Jayden. Kayla’s schedule is upended, and she feels the fatigue and frustration that caring for an inconsolable infant can bring.

But after just a few weeks, Jayden is abruptly taken away from Kayla.

Why? Because Jayden is a robot baby and Kayla participates in a school program that shows teenagers firsthand what it’s like to have a baby. In fact, this program is so popular that 89 countries around the world use it.

when-common-sense-programs-backfire2

 

Now let’s look at Colton. He is also a junior in high school, but he’s considered “at-risk” due to his truancy and shoplifting arrest last year.

Colton is spending the day at a county prison, not for a new crime but as part of a program at his school. He hears about the difficulty and horror of prison life from real convicts. They don’t hold back and don’t sugar coat what would happen if imprisoned. The prisoner's goal is to make prison seem so bad that he is “Scared Straight.”

These programs make intuitive sense. If you realized how difficult life is as a single mom, you'd take steps to avoid teen pregnancy. If you understand how awful prison is, you’ll avoid situations, friends, and activities that get you arrested.

But there’s one problem, and it’s what these teenagers have in common.

Both programs don’t work.

In fact, they lead to increased rates of teen pregnancy((See this Lancet study for more details.)) and increased rates of incarceration.((The scared straight program touted an 80+% success rate for years. But the “success” was based on survey. A voluntary four question survey parents completed shortly after the prison visit. The parents who didn’t respond were missing in the data and not included in the analysis.))

Now, knowing these programs fail, it’s easy to think of reasons why.

Taking care of a robot baby leads Kayla to enjoy being a mom. She realizes she can handle it. Unconsciously or consciously, Kayla may stop taking steps to avoid pregnancy.

Colton is scared by some prisoners, but at the same time respects some of the inmates. He may even view some prisoners as stupid or dumb…no wonder they got caught. Colton believes he’s smarter than the prisoners and won’t make those dumb mistakes. ((A teenager like Colton could be 28% more likely to commit crime after participating in Scared Straight. For more info on the failure of Scared Straight read Matthew Syed's, Black Box Thinkingpg 159-165))

What’s at play here?

As humans, we are great storytellers. Telling stories is fundamental to how we understand the world and how we explain it to each other. We can look at a series of facts and string together logical, intuitive reasons to justify how it is related.

The stronger the narrative, the more we believe it is true and then don’t question its conclusion.

This is called the narrative fallacy. It’s especially seductive when we create stories on seemingly obvious and intuitive explanations.

The stronger the story, the stronger the belief.

But, how can you avoid doing the same thing?

If you really want to discover firsthand how this works, you’ll have to do a little homework and run an experiment. ((If you can easily think of obvious reasons and a narrative why it doesn’t work, you absolutely need to test. You may have just used the wrong obvious narrative at the start.)) Your experiment can be as simple as an A/B test with only two groups. Ideally, done by someone with no vested interest in the outcome.

Randomly assign half to the test, and leave the other half untouched as a control group. The key here is randomization. Note that you must have the same types of people make up each group.

Then measure the difference between your test group and the control.

Collecting the data is the easy part, but it takes courage to accept negative results.

Perhaps your experiment may prove the program or “story” has no effect, and there is no difference with the control.

Or you may get a negative effect, as the robot baby study showed.

A teenager like Kayla who participates in the program was twice as likely to have a child by 20 as someone who never participated. ((Participants were also 50% more likely than to the control to have an abortion by age 20. See this Lancet study for more details.  The study looked at an Australian version of the program called Baby Think It Over. In the US it is called RealityWorks.

Image Source: scimex.org))

Just because something sounds like a good idea, makes obvious common sense, and has a strong narrative doesn’t make it work.

 

The takeaway:

What are you or your organization doing that is based on a strong, compelling narrative but has never been questioned for effectiveness?

The narrative may be wrong. Test it.

When seeing is misleading

When seeing is misleading

When data gives the wrong solution

When data gives the wrong solution