Survivor Bias

Kevin T. Bauder

During a recent conversation, a friend and I were reminiscing about some of the hazards we had faced as children. I’ve heard my parents tell about a couple of occasions when, as a baby, I was dropped on my head (which probably explains some things). I can recall being kept home from school for contracting chicken pox, mumps, and measles. I can also remember falling out of the hay loft onto a disc harrow below (I was saved from being maimed by my BB gun, which landed between me and the blades). Then there was the time that I climbed an old beech tree, slipped near the top, and banged into what seemed like every limb on the way down. Oh, and I can recall being in a car crash, long before seat belts were even an option in most cars.

As I recounted these incidents to my friend and he told me his stories, I commented, “You know, we’ve survived a lot of stuff.” He replied, “Aha! That’s survivor bias!” It really wasn’t, because I do recognize the hazardous nature of at least some of the events I survived. Survivor bias, however, occurs when people discount the seriousness of the hazards because they overlook the victims who did not survive. Survivor bias is widespread for one simple reason: survivors are present to talk about their experiences, while no one speaks for those who did not survive.

Failure to recognize survivor bias can skew the evaluation of events and institutions. For example, imagine that a hospital conducts a study of people who have fallen from various heights. It admits patients who have fallen one story and who show minor injuries. It admits patients who have fallen two stories and who show serious injuries. It admits patients who have fallen three stories and who show grave injuries. Yet the hospital admits no patients at all who have fallen from four stories or more. As the result of this study, it concludes that falling from four stories or more results in no injuries at all. Would anyone find this conclusion persuasive?

The answer is obvious. Those who fell from four stories or higher did not survive. They were not admitted to the hospital as patients because they were taken to mortuaries instead. No one is there to speak for them. Any failure to account for the silence of the dead will result in a seriously flawed perception of the risk of falling.

I wonder whether we do not have something of a survivor bias in our ministries. I believe that I have seen it at work in some churches. For example, I once had the opportunity to observe an independent Baptist church in a major Southern city. The preaching focused on three themes: get saved, get right, and get busy. Biblical exposition was considered offensive. The pastor thought that he should lead by telling people what to do, even in their private choices. Every service ended with an attempt to get all attendees to “go to the altar.” Discipleship consisted of being sent out for bus visitation on Thursday night. Members who wearied of the regimen were told that they needed to get right and get busy.

At one time this church managed to attract large crowds. At its high point, one out of every four homes in the city had someone attending its services. It baptized dozens of people some Sundays. The church prided itself on its success, and it put itself forward as a model for other churches to emulate.

After a time, however, I became aware of another dynamic in the church. The church was crowding people in the front door, but at a certain point it began to lose people out the back door just as rapidly. Most people could not stand up to the weekly barrage of high-pressure tactics. They burnt out and left, and when they left, they didn’t just leave that church. Very often they left Christianity altogether.

The church had a core of people who thrived on its vision of ministry. They tended to be Type-A personalities, the sort of people who loved working in management and in sales. They prided themselves upon the apparent success of the ministry. They tended to see themselves as the cause of its success. They were the epitome of survivor bias.

Over the years, this particular ministry damaged far more people than it helped. In fact, it often did some damage even to the people it did help. I would meet former members and attendees all over the community. When it came to spiritual discussions, they were among the most closed people I have ever met. They carried with them a deep hurt and they associated that hurt with Christianity in general.

The example I have chosen is deliberately extreme. I wonder, though, how often we allow survivor bias to creep into our evaluation of our own ministries. We tend to think that people who flourish are being helped to grow because of the excellence of our ministry. At the same time, we tend to think that people who leave have departed because of some defect in them. But what if it’s the other way around?

True, some people make bad choices, even under good ministries. When they do, they alone are responsible for the consequences. But some people also make good choices, even under very bad ministries. Their good choices are testimony to the power of the Word and the Spirit, and not to the validity of the ministry under which they were made. When we wish to evaluate our own ministries, we should never be content simply to point only to the survivors. We must consider the people whom our ministries have failed, hurt, or even destroyed along the way.

This essay is by Kevin T. Bauder, Research Professor of Systematic Theology at Central Baptist Theological Seminary. Not every one of the professors, students, or alumni of Central Seminary necessarily agrees with every opinion that it expresses.

Psalm 91
Isaac Watts (1674–1748)

Ye sons of men, a feeble race,
Exposed to every snare,
Come, make the Lord your dwelling-place,
And try and trust his care.

No ill shall enter where you dwell;
Or if the plague come nigh,
And sweep the wicked down to hell,
‘Twill raise his saints on high.

He’ll give his angels charge to keep
Your feet in all their ways;
To watch your pillow while you sleep,
And guard your happy days.

Their hands shall bear you, lest you fall
And dash against the stones:
Are they not servants at his call,
And sent t’ attend his sons?

Adders and lions ye shall tread;
The tempter’s wiles defeat;
He that hath broke the serpent’s head
Puts him beneath your feet.

“Because on me they set their love,
I’ll save them,” saith the Lord;
“I’ll bear their joyful souls above
Destruction and the sword.

“My grace shall answer when they call,
In trouble I’ll be nigh;
My power shall help them when they fall,
And raise them when they die.

“Those that on earth my name have known
I’ll honor them in heav’n;
There my salvation shall be shown,
And endless life be giv’n.”