"One-third of adults still sleep with a comfort object."

"The global rate for washing hands after using the toilet is under 20 percent."

This information comes from the results of surveys.

A survey is a method of gathering information from a sample of people, traditionally with the intention of generalizing the results to a larger population .

But is the information reliable?

To know whether we can trust survey data, we need to be aware of any biases.

Photo by Lukas Blazek on Unsplash

Bias

• A survey's measurement process might overestimate or underestimate the value of a population's survey responses.

Sample

• It's usually not practical for researchers to analyze a whole population.

• A sample is asmaller set of resultsthat the researchers use to represent a larger population.

Validity

• The survey results should relate to and support the researchers' conclusion .

• A valid survey measures what it's supposed to measure.

Ideally, the sample should be random and follow the criteria of the survey. If the sample is not random, the survey is less valid.

Example: The Truman-Dewey Presidential Race

In 1948, President Harry Truman ran against Thomas Dewey in the US Presidential election.

Nationwide telephone surveys predicted that Dewey would defeat Truman in a landslide, but Truman won.

What went wrong with the survey?

• Wealthy people, who mostly supported Dewey, were more likely to have phones in their homes

• Lower to middle-class families, who mostly preferred Truman were less likely to have phones at home to answer the survey

• The survey was inaccurate because it was biased towards Dewey voters

Non-response bias is when there is a significant difference between those who responded to your survey than those that chose not to.

People might not respond to a survey because:

• the survey asks for sensitive information that the person doesn't want to answer

• the target audience doesn't relate to the survey questions

• the person's email account sends the survey invite to a SPAM folder

Example: Sensitive Information

A survey asking young adults about their adult video watching habits may lead to a non-response bias because it's a very personal topic and respondents do not wish to share this information about themselves.

Response bias causes survey participants to answer inaccurately, which skews data and makes meaningful analysis difficult.

People might answer a survey inaccurately because:

• Leading questions with suggestive language push them to answer a certain way.

• They're embarrassed to give a socially unacceptable answer.

• They're asked to give "all or nothing" responses — a hard yes or no that doesn't account for their mixed feelings.

• They're overeager to support the survey's conclusion.

Example: Patient Satisfaction

In this NCBI study of patient satisfaction with their primary physician:

• Patients who had an overall higher satisfaction score were more likely to participate in satisfaction surveys compared to patients who had an overall low satisfaction score.

• Physicians were overestimating their patients' satisfaction and healthcare providers were not able to see the big picture.

The order in which questions are asked can influence how respondents answer the rest of the survey.

Example: List of Qualities

• A 1984 survey asked participants to choose the top 3 qualities a child should have from a list.

• 66% chose "honest" when it was higher on the list.

• Only 48% chose "honest" when it was lower on the list.

Answer order bias occurs when the order of your answer options influences the respondent to select a particular answer or a combination of answers.

Example: Pet Peeves

• SurveyMonkey asked 400 respondents to identify their biggest workplace pet peeves.

• Half of the participants were given a randomized list of pet peeves to choose from, while the other half's list had a predetermined order.

• "Rude coworkers" was a popular response with both groups, but the participants in the group with the predetermined list were more likely to choose an answer at the top of the list.