"One-third of adults still sleep with a comfort object."

"The global rate for washing hands after using the toilet is under 20 percent."

This information comes from the results of surveys.

A survey is a method of gathering information from a sample of people, traditionally with the intention of generalizing the results to a larger population .

But is the information reliable?

To know whether we can trust survey data, we need to be aware of any biases.

A laptop with data graphics on the screenPhoto by Lukas Blazek on Unsplash

Survey Terminology

Flaticon Icon

Bias

  • A survey's measurement process might overestimate or underestimate the value of a population's survey responses.

Flaticon IconSample

  • It's usually not practical for researchers to analyze a whole population.

  • A sample is asmaller set of resultsthat the researchers use to represent a larger population.

Flaticon IconValidity

  • The survey results should relate to and support the researchers' conclusion .

  • A valid survey measures what it's supposed to measure.

1. Sampling Bias

Ideally, the sample should be random and follow the criteria of the survey. If the sample is not random, the survey is less valid.

A graphic explaining that sampling bias means that some members of the intended population are more likely to be selected

Example: The Truman-Dewey Presidential Race

In 1948, President Harry Truman ran against Thomas Dewey in the US Presidential election.

Nationwide telephone surveys predicted that Dewey would defeat Truman in a landslide, but Truman won.

What went wrong with the survey?

  • Wealthy people, who mostly supported Dewey, were more likely to have phones in their homes

  • Lower to middle-class families, who mostly preferred Truman were less likely to have phones at home to answer the survey

  • The survey was inaccurate because it was biased towards Dewey voters

Boromir from Lord of the Rings saying, "One does not simply recruit people for a survey without considering sample bias."

2. Non-Response Bias

Non-response bias is when there is a significant difference between those who responded to your survey than those that chose not to.

People might not respond to a survey because:

  • the survey asks for sensitive information that the person doesn't want to answer

  • the target audience doesn't relate to the survey questions

  • the person's email account sends the survey invite to a SPAM folder

Example: Sensitive Information

A survey asking young adults about their adult video watching habits may lead to a non-response bias because it's a very personal topic and respondents do not wish to share this information about themselves.

Peter Parker from Spiderman crying. The text reads: "Non-response bias. When no one responds to your survey."

Quiz

Malina is conducting a survey and trying to avoid non-response bias. Which of the following strategies should she avoid?

Long surveys

Poorly worded questions

SPAM email invitations

All of these

3. Response Bias

Response bias causes survey participants to answer inaccurately, which skews data and makes meaningful analysis difficult.

People might answer a survey inaccurately because:

  • Leading questions with suggestive language push them to answer a certain way.

  • They're embarrassed to give a socially unacceptable answer.

  • They're asked to give "all or nothing" responses — a hard yes or no that doesn't account for their mixed feelings.

  • They're overeager to support the survey's conclusion.

Marge Simpson filling out a survey, saying,

Example: Patient Satisfaction

In this NCBI study of patient satisfaction with their primary physician:

  • Patients who had an overall higher satisfaction score were more likely to participate in satisfaction surveys compared to patients who had an overall low satisfaction score.

  • Physicians were overestimating their patients' satisfaction and healthcare providers were not able to see the big picture.

Quiz

What phrase from this government education survey question leads to response bias: "Do you think our world-leading education system needs improvement, or are you satisfied with the high-quality education your child receives?"

world-leading education

do you think

high-quality education

receives

4. Question Order Bias

The order in which questions are asked can influence how respondents answer the rest of the survey.

Flaticon Icon

Example: List of Qualities

  • A 1984 survey asked participants to choose the top 3 qualities a child should have from a list.

  • 66% chose "honest" when it was higher on the list.

  • Only 48% chose "honest" when it was lower on the list.

5. Answer Order Bias

Answer order bias occurs when the order of your answer options influences the respondent to select a particular answer or a combination of answers.

Example: Pet Peeves

  • SurveyMonkey asked 400 respondents to identify their biggest workplace pet peeves.

  • Half of the participants were given a randomized list of pet peeves to choose from, while the other half's list had a predetermined order.

  • "Rude coworkers" was a popular response with both groups, but the participants in the group with the predetermined list were more likely to choose an answer at the top of the list.

Chart with biggest pet peeves at work

Take Action

Jim Carey saying,

License:

Your feedback matters to us.

This Byte helped me better understand the topic.

I feel confident that I can apply what I learned.

This Byte helped me better understand the topic.

Connect with Rumie learners

Browse more content and join a supportive community!


- Discord Member

This Byte has been authored by

RA

Raheema Afridi

Passionate for science, teaching and coffee.