How to seamlessly integrate data quality measures into your questionnaire

Please tweet! @PeanutLabsMedia @LoveStats
How to seamlessly
integrate data quality
measures into your
questionnaire
Presented by Annie Pettit
Chief Research Officer at
Peanut Labs
Please tweet! @LoveStats @PeanutLabsMedia

How to distinguish between these people?
But,
incentive getting,
satisficing,
cheaters make lots
of mistakes
When answering
surveys, good
honest people will
naturally make a
mistake or two

Multi-Select Questions
Red Herrings
High/Low Incidence
Over/Under Clicking
Following Instructions
Sed turpis elit, venenatis ut
porttitor sed, venenatis in arcu.
Nam consequat leo ut lorem
viverra, ac commodo ex
ullamcorper?
 Praesent
 Maximus
 Ullamcorper
 Faucibus
 Lacinia
 Tincidunt
 Efficitur

Red Herrings
• Question Type: Multi-selects, Single-selects,
Rating scales
• Concern: Respondents may have misread or
confused a name. It may be a real name!
• Cheaters: Incentive Getting. Responder is
providing the minimum amount of information
required to proceed.
• Application: Choose at least TWO fake names
and Google them to ensure they are extremely
low incidence
• Scoring: Flag answers in sets of two
Which of the following stores
have you visited in the past 3
months? (Please select all that
apply)
 Abercrombie & Fitch
 Aéropostale
 American Apparel
 American Eagle Outfitters
 Anthropologie
 Bellamis New York
 Bloomingdale's
 Brooks Brothers
 Club Forenzo
 Coldwater Creek
 Dillard's
 DKNY
 Other

High Incidence
• Question Type: Multi-selects
• Concern: Respondents are tired or bored.
• Cheaters: Satisficing. Responder is providing the
minimum amount of information required to
proceed.
• Application: Works best on longer lists. Include
at least a few answers that ought to be selected
by everyone
• Scoring: Flag people with the fewest clicks for
very common responses.
Which of the following activities
have you participated in during
the last 3 months? (Please select
all that apply)
 Attended a sporting event
 Attended a music event
 Exercised
 Listened to music
 Used the internet
 Visited a community center
 Visited a library
 Watched TV
 Went to a grocery store
 Went to the movies
 Went to school
 Went to work

Low Incidence
• Concern: Respondents are thinking of people
other than themselves
• Cheaters: Incentive getting. They may be trying
to avoid screening out so they will qualify for an
incentive.
• Application: Incorporate at least 3 extremely low
incidence options. Rare doesn't happen in pairs.
• Scoring: Flag people who select two or more rare
answers.
Which of the following ailments
do you have? (Please select all
that apply)
 Acid Reflux
 Acidosis
 Adrenal Disorders
 AIDS
 Alzheimer's Disease
 Amyloidosis
 Anemia
 Anorexia Nervosa
 Arteriosclerosis
 Autism
 Blood Pressure (High)
 Bronchitis
 Cancer

Underclicking
• Concern: Respondents are tired or bored.
proceed.
• Application: Works best on longer lists. Include
at least a few answers that ought to be selected
by everyone.
• Scoring: Flag people with the fewest clicks across
the whole question.
apply)
 Albertsons
 Big Lots
 Costco
 CVS
 Dollar General
 Family Dollar
 Kroger
 Publix
 Safeway
 Target
 Walgreen
 Walmart

Overclicking
• Concern: They may be thinking of their
household or family members.
• Cheaters: Incentive getting. They may be trying
to qualify for the incentive.
• Application: Works best on longer questions.
Ensure there are options that don’t really go
together.
• Scoring: Flag people with the most clicks across
the whole question.
apply)
 Abercrombie & Fitch
 Aéropostale
 American Apparel
 American Eagle Outfitters
 Anthropologie
 Barneys New York
 Bloomingdale's
 Brooks Brothers
 Club Monaco
 Coldwater Creek
 Dillard's
 DKNY
 Eddie Bauer

Following Instructions
• Question Type: Unvalidated multi-selects
• Concern: Respondents may have misunderstood
the task, or want to provide a ‘more accurate’
answer
• Cheaters: Satisficing. The responder wants to
finish and get their incentive.
• Application: Word questions so it makes sense to
ask for exactly 2 or 3 choices. Don’t validate!
• Scoring: Flag anyone who chooses more or fewer
responses than requested.
Which three of the following
stores did you visit most often?
(Please select only 3)
 Aéropostale
 American Apparel
 Anthropologie
 Bellamis New York
 Bloomingdale's
 Brooks Brothers
 Club Forenzo
 Coldwater Creek
 Dillard's
 DKNY
 Other

Don’t Know
• Question Type: Multi-selects, Single-selects
• Concern: They probably do know if they would
think about it a little bit
proceed.
• Application: Follow good survey design practice
of including DK wherever it is possible
• Scoring: Count how many times they select
‘Don’t Know’ across the survey
apply)
 Aéropostale
 American Apparel
 Anthropologie
 Barneys New York
 Bloomingdale's
 Brooks Brothers
 Club Monaco
 Coldwater Creek
 Dillard's
 DKNY
 Don’t know

Grid Rating Questions
Straightlining
Sed turpis elit, venenatis ut
porttitor sed, venenatis in
arcu?
o o o o o
o o o o o
o o o o o
o o o o o
Praesent maximus
Ullamcorper faucibus
Lacinia tincidunt
Efficitur aliquam

Straightlining
• Question Type: Multi-select ratings
• Concern: Respondents may have only glanced at
the items, not noticed reverse keyed items
• Cheaters: Satisficing. Get in and get out!
• Application: Be sure to include positively and
negatively keyed items.
• Scoring: Measure every grid for patterns -
vertical, diagonal, repetitive. Each straightline
generates a flag.
What is your opinion about
each of these statements?
o o o o o
o o o o o
o o o o o
o o o o o
Type A
X
X
X
X
X
Type B
X
X
X
X
X
Type C
X
X
X
Tastes good
Smells tempting
Feels nice
Looks pretty

“Select the third answer”
• Question Type: Multi-Select Rating
• Concern: It’s confusing.
It creates suspicion.
SQUIRREL!
• Cheaters: Satisficing. Answer quickly and get
your incentive.
• Application: Ask responders to choose a specific
answer, “Select Somewhat Disagree in this row”
• Scoring: Don’t! This is a terrible measure!
What is your opinion about each of
these statements?
o o o o o
o o o o o
o o o o o
o o o o o
o o o o o
Tastes bland
Would recommend
Select Disagree Somewhat
Box is ugly
Smells delicious
Agree
Strongly
Agree
Somewhat
Neutral
Disagree
Somewhat
Disagree
Strongly
Squirrel!

Open-End Verbatims
Pellentesque quis aliquet felis,
sit amet ultricies erat. Etiam vel
metus augue.

dunno NA
I lik that their
crunchy
? none
1) If Congress cannot do its job - pass a
Budget EVERY year (not a CR), they should
NOT get paid AND should NOT get ANY
benefits; 2) An ex government official should
be prohibited form becoming a lobbyist within
2 years of leaving office; 3) Contracting out
should NOT be done if NO REAL costs savings
can be realized; and 4) People making over
$250, 000 should pay a higher percentage of
income tax - minimum of 30%
Verbatim Length
• Question Type: Long verbatims
• Concern: It’s difficult to introspect and self-
evaluate
proceed.
• Application: Create at least one question that
requires a long answer. “Describe three reasons
why…” (And benefit from the tidbits of gold!)
• Scoring: Flag any response under 10 characters.

asdf
You’re stupid
dumbdumbdumb
ass
hjsrthetutjytnse5hyyjn,jiuuyj
ydtpysrgawe.btzergrsebearh
uyllkjhhgcfddwad
Verbatim Gibberish
• Question Type: Short verbatims, Long verbatims
• Concern: It’s difficult to introspect and self-evaluate
proceed.
• Application: Ensure there is at least one open-end
question for a text response.
• Scoring: Look for no spaces, improbable letter
combinations (hh, kk, yy, hj, bt, js). Assign points
based on the severity (e.g., crude words=3, ‘ass’=2,
‘asdf’=1

Other Question Types
Contradictions; Don’t Know;
Sums and Ranks; Speeding

Contradictions
• Question Type: Any two questions
• Concern: Respondents may misread one of the
questions. The question was poorly worded.
• Cheaters: Satisficing.
• Application: Ensure the questions are far apart.
Focus on related, contextual questions, not
identical questions. Allow improbable answers.
• Scoring: Flag cases where the answers don’t
match
Q3. What pets do you have in
your household?
 Dog
 Cat
 Fish
 Bird
 Small mammal
 Other
 None
Q19. How often does your household
buy pet food?
o At least once per week
o About once per month
o Several times per year
o About once per year
o Less often
o Never

Rank Orders
• Question Type: Unvalidated fill-in-the-blank rank
orders
• Concern: They didn’t read carefully, They weren’t
thinking carefully. They didn’t understand what they
were being asked for.
• Cheaters: Satisficing. Responder is incentive getting.
• Application: Create a question with 5 to 8 options.
Don’t validate!
• Scoring: Flag cases with any numbers less than or
greater than the minimum/maximum, or if there
are any duplicate numbers
Please rank the importance of
these store features from 1 to 5.
1 = Most Important
5 = Least Important
(Please use each number only once.)
 Price
 Location
 Style
 Service
 Selection

Sums
• Question Type: Unvalidated fill-in-the-blank
sums
• Concern: They didn’t understand the question.
Their math doesn’t reflect reality. They aren’t
good with math.
• Cheaters: Satisficing.
• Application: Create a question with 4 to 6
options so the math is reasonable. Don’t
validate!
• Scoring: Flag any response that doesn’t add to
100.
What percentage of your
monthly income is spent in these
areas? (Please make sure the
numbers add up to 100%.)
 Housing
 Food
 Utilities (e.g., heat, electricity)
 Transportation
 Entertainment

Speeding
• Question Type: The entire survey
• Concern: Some people are fast readers. Some
have a lot of survey experience. Some got all the
skips and short paths.
• Cheaters: Satisficing. Incentive getting.
• Application: Wait until all the completes are in.
Fastest 5% is ideal. Fastest 2% is impossible!
• Scoring: Give one point to the fastest 5%. Give
two points to the fastest 2%.
Seconds Cumulative
Frequency
0 - 40 0
41 - 50 0
51 - 60 0.2
61 - 70 0.5
71 - 80 0.9
81 - 90 1.5
91 - 100 2.4
101 - 110 3.5
111 - 120 4.7
121 - 130 6.4
131 - 140 8.3
141 - 150 10.5
151 - 160 13.6
161 - 170 16.6
171 - 180 19.7
181 - 190 23.6
191 - 200 27.6

Creating Cut-Scores
Question Level
Survey Level

Question Level Cut-Scores
Red
Herring
Percent of
people failing
0 78.9
1 15.4
2 2.9
3 2.8
Over
Clicking
Percent of
people failing
0 76.3
1 13.5
2 9.4
3 0.8
Rank
Order
Percent of
people failing
0 77.7
1 9.6
2 2.9
3 9.8
Ideal Scenario Too Few Fail Too Many Fail
•Check and improve
the scoring
•Use it anyways
•Don’t use it at all
•Check and improve
the scoring
•Use it anyways
• Anything
between 3
and 7 is ideal

Question Level Scores of Individual People
Person 1
(Regular Person)
Person 2
(Satisficer)
Person 3
(Incentive Getter)
Overclick 0 0 1
Underclick 0 1 0
Speeding 0 2 0
Red Herring 1 1 2
Low Incidence 0 0 1
High Incidence 0 1 0
Rank Order 0 0 0
Sums 0 0 0
TOTAL 1 5 4
This is not a problem THESE are problems!

Survey Level Cut-Score
• Goal: Fail 5% of respondents
• Cut-Score: 3
• Result: 4.1% of respondents failed
• Why?
– Survey was poorly written
– Cheaters, liars, meanies
– Respondents couldn’t understand the
questions, perhaps ESL or low reading level
Flags Cumulative
Frequency
0 64.0
1 25.0
2 6.9
3 2.5
4 0.8
5 0.5
6 0.1
7 0.1
8 0.1
9 0.0
10 0.0

Tips
• Try to incorporate at least 4 different measures into every
survey
• Don’t cram too many tests into one question
– You COULD put overclicking, underclicking, high incidence, low
incidence, and red herrings in the same question. But if that’s
the only question a good respondent had trouble with, you’ve
excluded good data.
– Use each question a maximum of two times
• Spread the tests throughout the survey - beginning, middle,
and end.
• This doesn’t have to be a lot of work. Create SPSS/Excel syntax
that you can use repeatedly.
• Remember that human beings make mistakes. You makes lots
of mistakes everyday. Don’t expect other people to be better
than you.

Thank you!
Annie Pettit
Chief Research Officer
Peanut Labs
annie@peanutlabs.com
twitter.com/LoveStats
ca.linkedin.com/in/anniepettit/
Questions about our Sample Services?
Jonathan Cheriff
Director of Sales & Marketing
jonathan.cheriff@peanutlabs.com
twitter.com/paperbackdad
www.linkedin.com/in/jonathancheriff

How to seamlessly integrate data quality measures into your questionnaire

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to How to seamlessly integrate data quality measures into your questionnaire

Similar to How to seamlessly integrate data quality measures into your questionnaire (20)

More from Annie Pettit, Research Methodologist

More from Annie Pettit, Research Methodologist (20)

Recently uploaded

Recently uploaded (20)

How to seamlessly integrate data quality measures into your questionnaire