The document summarizes research into the effects of splitting long surveys into two shorter surveys. It finds that data quality is maintained or improved with shorter surveys, as respondents make fewer errors and provide more detailed open-ended responses. Results from different question sets are largely equivalent between survey lengths. Respondent satisfaction may also be higher for shorter surveys, as indicated by more positive survey comments. While not a definitive solution, splitting long surveys appears to address issues like respondent fatigue without harming data quality or equivalence.
5. Problem:
Nothing is directly Comparable
• People choose whether to participate
• Each version is different
– Different length
– Different topics
– Different leading/priming questions
Tweet! #AAPOR @LoveStats
6. Tweet! #AAPOR @LoveStats
Agenda
1. Data quality
2. Data equivalence
3. Responder satisfaction
11/6/2014 6
12. Speeding differences not discernable
Tweet! #AAPOR @LoveStats
Long
Short A
Short B
Standardized Number of Seconds
m=0, sd=1, ceiling outliers removed
13. Drop the worst 5%
from each dataset
6.1% of Long
6.3% of Short A
5.2% of Short B
Tweet! #AAPOR @LoveStats
15. Survey Middle: Trips Taken
Tweet! #AAPOR @LoveStats
80%
60%
40%
20%
0%
1 to 3 trips 4 or more trips
Long Short A Short B
16. Survey Middle: Airline Aware
P<.05 P<.05
Tweet! #AAPOR @LoveStats
P<.05
P<.05
100%
75%
50%
25%
0%
Long Short A Short B
17. Survey End: Like Advertising
Tweet! #AAPOR @LoveStats
60%
50%
40%
30%
20%
10%
0%
Youtube Facebook Flyers Billboards
Long Short A Short B
P<.05
21. Same Completion Trend
Optional
watch a
commercial
Optional
watch a
commercial
Tweet! #AAPOR @LoveStats
100%
75%
50%
25%
Long Short A Short B
Grid
22. More Words in Verbatim
Tweet! #AAPOR @LoveStats
60
40
20
0
Verbatim 1 Verbatim 2 Verbatim 3 Verbatim 4 Verbatim 5 Verbatim 6
Long ShortA ShortB
Optional
P<.05
23. Possibly Higher Satisfaction
P<.05
P<.05
Tweet! #AAPOR @LoveStats
100%
75%
50%
25%
0%
Easy Boring Long Reco
Long Short A Short B
P<.05
24. More Positive Survey Comments
Tweet! #AAPOR @LoveStats
75%
86%
93%
Long Short A Short B
Percentage of optional survey comments that were positive
25. Conclusions
• Not a slam-dunk win
• Data quality is as good or better
• Results are equivalent
• Survey satisfaction is as good or
better
Tweet! #AAPOR @LoveStats
26. Final Thoughts
• Responders self-select for a 10 or 20 minute
Tweet! #AAPOR @LoveStats
experience
• Differences are in line with sampling error
• Large portions of most surveys do
not need to be answered by everyone.
Plan ahead.
27. Thank you!
Annie Pettit
Chief Research Officer
Peanut Labs
annie@peanutlabs.com
@LoveStats on Twitter
www.peanutlabs.com
Tweet! #AAPOR @LoveStats
: )