This logo isn't an ad or affiliate link. It's an organization that shares in our mission, and empowered the authors to share their insights in Byte form.
Rumie vets Bytes for compliance with our
Standards.
The organization is responsible for the completeness and reliability of the content.
Learn more
about how Rumie works with partners.
"One-third of adults still sleep with a comfort object."
"The global rate for washing hands after using the toilet is under 20 percent."
This information comes from the results of surveys.
A survey is a method of gathering information from a sample of people, traditionally with the intention of generalizing the results to a larger population .
But is the information reliable?
To know whether we can trust survey data, we need to be aware of any biases.
Survey Terminology
1. Sampling Bias
Ideally, the sample should be random and follow the criteria of the survey. If the sample is not random, the survey is less valid.
Example: The Truman-Dewey Presidential Race
In 1948, President Harry Truman ran against Thomas Dewey in the US Presidential election.
Nationwide telephone surveys predicted that Dewey would defeat Truman in a landslide, but Truman won.
What went wrong with the survey?
Wealthy people, who mostly supported Dewey, were more likely to have phones in their homes
Lower to middle-class families, who mostly preferred Truman were less likely to have phones at home to answer the survey
The survey was inaccurate because it was biased towards Dewey voters
Did you know?
You can get feedback on your resume from Rumie community members in our Discord Server. Come by, we'll help you get the job!
2. Non-Response Bias
Non-response bias is when there is a significant difference between those who responded to your survey than those that chose not to.
People might not respond to a survey because:
the survey asks for sensitive information that the person doesn't want to answer
the target audience doesn't relate to the survey questions
the person's email account sends the survey invite to a SPAM folder
Quiz
Malina is conducting a survey and trying to avoid non-response bias. Which of the following strategies should she avoid?
Participants might not be motivated to complete a survey if it's too long or the questions aren't related to their lives. If the invitation email ends up in their SPAM folder, they might not even see the survey.
3. Response Bias
Response bias causes survey participants to answer inaccurately, which skews data and makes meaningful analysis difficult.
People might answer a survey inaccurately because:
Leading questions with suggestive language push them to answer a certain way.
They're embarrassed to give a socially unacceptable answer.
They're asked to give "all or nothing" responses — a hard yes or no that doesn't account for their mixed feelings.
They're overeager to support the survey's conclusion.
Example: Patient Satisfaction
In this NCBI study of patient satisfaction with their primary physician:
Patients who had an overall higher satisfaction score were more likely to participate in satisfaction surveys compared to patients who had an overall low satisfaction score.
Physicians were overestimating their patients' satisfaction and healthcare providers were not able to see the big picture.
Quiz
What phrase from this government education survey question leads to response bias: "Do you think our world-leading education system needs improvement, or are you satisfied with the high-quality education your child receives?"
"World-leading education" and "high-quality education" show an underlying assumption that the educational institution is already exceptional. This is a leading question that will bias the results.
4. Question Order Bias
5. Answer Order Bias
Example: Pet Peeves
SurveyMonkey asked 400 respondents to identify their biggest workplace pet peeves.
Half of the participants were given a randomized list of pet peeves to choose from, while the other half's list had a predetermined order.
"Rude coworkers" was a popular response with both groups, but the participants in the group with the predetermined list were more likely to choose an answer at the top of the list.
Take Action
This Byte has been authored by
Raheema Afridi
Passionate for science, teaching and coffee.