A comparison of some leading sample sources including river sample and router sample.

1. Does the Sample Source Matter?
A comparison of some leading sample sources
October 3, 2011

2. Sample is the soil
and water of
research.
The quality of it
determines the
value of what you
produce.

3. We compared SBA to four other sample
sources that VC regularly uses
We conducted a five minute survey Nov 4 to 7, 2011
with a sample of about 500 people each, using sample
from Vision Critical’s Springboard America, and four of
the leading sample providers that we have anonymised
as “Panel B”, “ Panel C”, “Panel D” and “River E”.
The questionnaire included items from the US 2010
General Social Survey, as well as a questions about
snack consumption and a concept test for a snack food.
We counted the number of statistically significant
differences between the panels (95% CI)

4. Sample Quality: Dropout rate during the survey
Dropouts as a percent of all who started the
survey
20%
16%
15%
10%
10% 8%
7%
5% 4%
0%
SBA Panel B River C Panel D River E

5. Data quality: straight-lining
% who gave the same answer for all 13 trust ratings
10%
9%
7.9% 7.8%
8%
7% 6.5%
6%
5%
3.9%
4%
3%
2%
1% 0.5%
0%
SBA Panel B River C Panel D River E

6. Data quality: use of “don’t know”
Average % "Don't Know"
8%
7%
6% 6% 6%
6%
5%
4% 4%
4%
3%
2%
1%
0%
SBA Panel B River C Panel D River E
Average percent who choose the option "don’t know", "can't choose", "not
sure", "prefer not to say".

7. Differences between panels in how people
answered all questions
Percent of times panel differed from the others
15%
13%
11%
11%
9%
7% 7%
5% 6%
Expected
5% 4% by chance
3%
1%
-1% SBA Lightspeed SSI
Panel C Research Now
Panel D Fulcrum
River
Panel B
Count of the number of statistically significant differences (95% CI) as a percent
of the total number of differences possible

8. Differences between panels in how people
answered snack purchase questions
Purchase behaviour: Percent of times panel differed
from the others
15%
13% They were
13%
less likely to
11% buy snacks
9%
7% 6% 6%
5%
3% 2% 2%
1%
-1% SBA Panel B River C Panel D River E
Count of the number of statistically significant differences (95% CI) as a percent
of the total number of differences possible

9. Differences between panels in how people
answered concept test questions
Concept Test: Percent of times panel differed from the
others
30%
26%
25%
20%
16%
15% 13%
11%
10%
6%
5%
0%
SBA Lightspeed
Panel B SSI
Panel C Research Now
Panel D Fulcrum
River
Count of the number of statistically significant differences (95% CI) as a percent
of the total number of differences possible

10. Differences between panels in how people
answered the purchase intent question
Definitely will Probably will
45%
40%
35%
30%
25% 28%
21%
20% 23%
21%
15%
15%
10%
5% 13% 12%
9% 8%
6%
0%
SBA Panel B River C Panel D River E
Count of the number of statistically significant differences (95% CI) as a percent
of the total number of differences possible

11. Conclusions
Sample matters. A lot.
Springboard America has fewer drop-outs, notably
better data quality, and is “right down the middle”
when it comes to comparison to the other panels on
the findings, unlike some panels which show real
variation—particularly on the concept test.