Multimode Global Scale Usage
âą
1 gefĂ€llt mirâą593 views
Three studies examined scale usage across different data collection modes. Study 1 found differences in extreme, midpoint, and acquiescence response styles between mobile and web surveys, with mobile sliders producing higher extreme responding than grids. Study 2 found response styles varied by age and country but not gender, and using more scale points reduced midpoint responding. Study 3 found on social media that women, older people, and Australians responded more positively than others.
Empfohlen
Empfohlen
Weitere Àhnliche Inhalte
Ăhnlich wie Multimode Global Scale Usage
Ăhnlich wie Multimode Global Scale Usage (20)
Mehr von Annie Pettit, Research Methodologist
Mehr von Annie Pettit, Research Methodologist (20)
KĂŒrzlich hochgeladen
KĂŒrzlich hochgeladen (20)
Multimode Global Scale Usage
- 1. Multimode, Global Scale Usage Understanding respondent scale usage across devices, borders & modes Presenters: Melanie Courtright Annie Pettit Co-authors: Kartik Pashupati Roddy Knowles
- 2. Battle Of The Scales
- 3. Battle Of The Scales Scales are a cornerstone of market research 49% 42%
- 4. Measurement Scales & Response Styles A personâs tendency to systematically respond to questionnaire items in a certain way regardless of content Paulhus 1991 ARS MRS ERS Substance meaning of the question Factors unrelated to content
- 5. Why Response Styles Matter Lack of measurement equivalence in data across countries implies that there is no common basis to compare data across countries: In such cases, observed mean differences on relevant constructsâŠmight result from measurement artifactsâŠrather than from true differences across countries Alain De Beuckelaer et al., 2007
- 6. Three studies to examine scale usage across three modes of data collection Scope Of This Paper Social media Mobile surveys Web surveys
- 7. Scope Of This Paper What is the effect of using 3 alternative data collection options on response styles in mobile surveys? Compare differences in responses to 5-point Likert scales across 3 data collection options: Standard grid questions on mobile web browser Slider scales on mobile web browser Slider scales on native app
- 8. Three studies to examine scale usage across three modes of data collection Scope Of This Paper Social media Mobile surveys Web surveys
- 9. 10-point scales Web Surveys Scope Of This Paper 4-point scales 5-point unlabeled scales 5-point labeled scales 7-point scales
- 10. Three studies to examine scale usage across three modes of data collection Scope Of This Paper Social media Mobile surveys Web surveys
- 11. Scope Of This Paper Gender Social Media Country Age
- 12. Overall Research Plan For Web & Mobile Surveys (Studies 1 & 2) Globally relevant topics Include behavioral statements that should correlate to attitudinal questions 7-minute attitudinal survey Mix of positive & negative wording Scales with published measures of reliability Fielded in multiple countries to analyze cross-cultural differences
- 13. Overall Research Plan For Social Media (Study 3) Collect verbatim responses from thousands of sites Data collected only in English-speaking countries Score responses into a 4-point scale
- 14. Overall Research Plan For Web & Mobile Surveys (Studies 1 & 2) 4-point scales 5-point unlabeled scales 10-point scales 5-point labeled scales 7-point scales Used only 5-point Likert scales for all constructs Standard grid questions on mobile web browser Slider scales on mobile web browser Slider scales on native app Used only 5-point Likert scales for all constructs Fielded in 10 countries
- 15. Overall Research Plan Research Plan Globally relevant topics Include behavioral statements that should correlate to attitudinal questions 7-minute attitudinal survey Mix of positive & negative wording Scales with published measures of reliability Gender Country Age Simultaneously field across 10 countries VOP Research Nowâs proprietary Valued Opinions Panel Simultaneously compare the effect of multiple response options (4-, 5-, 7- & 10-point scales) Using a large, census-balanced, multi-country sample
- 16. Battle of the Scales Analysis Plan for Studies 1 & 2 1 1 1 Extreme Response Style Index (ERSI) Acquiescence Response Style Index (ARSI) Medium Response Style Index (MRSI) Indices were averaged for respondents Value of each index ranges from a minimum of 0 to a maximum of 1.0.
- 17. Battle of the Scales Research Questions: Study 1 RQ1 RQ2 Grid Slider Native APP Questions?
- 18. Results 1: Study 1 Differences in Extreme Response Style Across Modes Extreme response style (ERS) index Health environment sensitivity Personal health responsibility Attitude toward helping others Material values scale 70 60 40 30 20 0 Attitude towards advertising in general (AAG) Online piracy concern Lie tolerance scale 50 Online web labeled Online web unlabeled Mobile native app Mobile web grid Mobile web slider
- 19. Results 2: Study 1 Midpoint Response Style (MRS) Index Health environment sensitivity Personal health responsibility Attitude toward helping others Material values scale 50 30 20 10 0 Differences in Midpoint Response Style Across Modes Attitude towards advertising in general (AAG) Online piracy concern Lie tolerance scale 40 Online web labeled Online web unlabeled Mobile native app Mobile web grid Mobile web slider
- 20. Results 3: Study 1 Acquiescence Response Style (ARS) Index Health environment sensitivity Personal health responsibility Online web labeled Online web unlabeled Mobile native app Attitude toward helping others Mobile web grid Mobile web slider Material values scale 80 60 40 20 0 Differences in Acquiescence Response Style Across Modes Attitude towards advertising in general (AAG) Online piracy concern Lie tolerance scale
- 21. Research Questions & Hypotheses: Study 2 Differences in Response Styles Across Countries, Genders & Ages Acquiescent Midpoint Extreme H1: There will be no differences between male & female respondents with respect to ERS & MRS H2: ERS & MRS will differ across age categories, but the relationship between age & ERS or MRS will not be monotonic H3: There will be significant differences across countries with respect to ERS, MRS & ARS
- 22. Results 1: Study 2 Impact of Gender on Response Styles Hypothesis 1: There will be no meaningful differences in ERS, MRS or ARS indices between male & female respondents ERSI ARSI MRSI .43 .32 .28 .43 .33 .28
- 23. Results 2: Study 2 Impact of Age on Response Styles MRS (Medium) ARS (Acquiescent) ERS (Extreme) 15-17 18-24 25-34 35-44 50 45 30 25 20 0 45-54 55-64 65-74 35 Hypothesis 2: ERS & MRS will differ across age categories, but the relationship between age & ERS or MRS will not be monotonic 40 75-84 85+
- 24. Results 3: Study 2 Differences in Response Styles Across Countries H3: There will be significant differences across countries with respect to ERS, MRS & ARS 60 50 40 30 20 10 0 ERS (Extreme) ARS (Acquiescent)
- 25. Results 4: Study 2 Impact of Response Scales on Response Styles RQ3: Does the use of 4-point, 5-point, 7-point & 10-point scales make any difference to 4 Point 5 Point Labelled 50 45 30 25 20 0 5 Point Unlabelled 7 Point 35 the level of extreme, acquiescent or midpoint responding? 40 10 Point MRS (Medium) ARS (Acquiescent) ERS (Extreme)
- 26. Battle Of The Scales Scales in Social Media Awesome Wicked Bomb Good Nice Cool Huh Dunno Whatevs Crap Yuck Dumb Hideous Disgusting Abhor Scales are a cornerstone of market research
- 27. Research Questions: Study 3 Impact of Gender, Age & Region on Social Media Scaling RQ4: How do gender differences affect scaling in social media? For example, do men/women make word choices that are more/less extreme in sentiment resulting in more/less extreme responding? RQ5: How does age of a social media participant affect the scaling of social media sentiment? RQ6: Are there systematic differences across geographic regions or nationalities in terms of scaling social media sentiment?
- 28. Results 1: Study 3 Impact of Gender on Response Styles RQ6: How do gender differences affect scaling in social media? For example, do men (women) make word choices that are more (less) extreme in sentiment resulting in more (less) extreme responding? 5.0 = Strong/Negative Moderate/Negative Moderate/Positive Strong/Positive 34.1 51.0 9.9 3.8 29.2 53.7 13.2 -1.1 -4.9 2.7 3.3 %
- 29. Results 2: Study 3 Impact of Age on Response Styles RQ7: How does age of a social media participant affect the scaling of social media sentiment? 1930 Decade of Birth Sample Size Strong/Negative Moderate/Negative Moderate/Positive Strong/Positive 1940 1950 1960 1970 1980 1990 488 2676 4337 6492 8307 21644 20156 2.7% 1.1% 1.2% 1.4% 1.1% 1.5% 1.7% 34.4% 33.0% 30.0% 31.6% 33.0% 33.6% 35.1% 58.0% 58.7% 61.5% 60.0% 59.9% 59.6% 58.8% 4.9% 7.1% 7.3% 7.0% 6.0% 5.3% 4.4%
- 30. Results 3: Study 3 Impact of Country on Response Styles RQ7: Are there differences in the scaling of social media sentiment across different (English-speaking) countries? AUSTRALLIeAss CnAeNgAaDtivAity aIRmEoLnAgN DAustralUiaKns US Sample Size 29,372 54,505 155,068 962,270 Strong/Negative Moderate/Negative More extreme responding among United Kingdom authors Moderate/Positive Strong/Positive 9,323 More negativity among Americans 6.6% 7.7% 7.9% 9.5% 9.5% 29.5% 31.0% 29.6% 28.3% 30.7% 50.0% 47.1% 47.2% 45.0% 45.9% 13.9% 14.1% 15.3% 17.1% 13.9%
- 31. Mobile Surveys ï§ There are significant differences in ERS, ARS and MRS between mobile and computer responses, but differences are not consistent. ï§ Not enough evidence to conclude if mobile is better/worse than computer-based data collection. ï§ Compared to mobile grids, mobile sliders seem to produce slightly higher levels of ERS. ï§ Compared to mobile grids, mobile sliders seem to produce lower levels of MRS. ï§ There are no significant differences between mobile sliders and grids in ARS. Conclusions
- 32. Web Surveys ï§ ERS, MRS and ARS do not differ by gender, but do differ as age increases. ï§ Response styles vary from one country to another. ï§ Varying the number of response options does help to reduce MRS. ï§ Scale labeling did not impact scale reliability, but did impact ERS, MRS and ARS. Conclusions
- 33. Social Media ï§ Women are more positive. ï§ Older people are more positive. ï§ Australians are more positive. Conclusions
- 34. Multimode, Global Scale Usage Understanding respondent scale usage across devices, borders & modes Melanie Courtright Kartik Pashupati Annie Pettit Roddy Knowles
Hinweis der Redaktion
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Lighten up bullet points â way too much being communicated on this slide. Convert from 6 different bullet points to 6 different slides.
- Although there are significant differences in ERSI between the different measurement modes, the differences are not systematically attributable to a single mode of measurement. The mobile native app produced the highest levels of extreme responding for 4 of the scales but the ERSI was only significantly and meaningfully higher for 2 of the scales. The only observable systematic pattern is that the labeled version of the online web survey generally produced lower extreme responding than the unlabeled version.
- The results of the multi-mode comparison for midpoint responding style (MRS) are presented in this graph. There is at least one systematic trend evident here, namely, that the mobile web slider consistently produces the lowest MRSI for all of the scales â although the difference is statistically significant only for 3 out of 6 scales. This trend is presented visually in Exhibit D.
- When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
- Convert bullet points into something more graphic (i.e. a flow chart illustrating: 1) survey requirements, 2) plan, 3) objective)
- When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
- See notes for slide 8
- See notes for slide 8
- See notes for slide 8
- Include a more fitting image than what is provided to the right.
- Convert bullet points into something more graphic (i.e. a flow chart illustrating: 1) survey requirements, 2) plan, 3) objective)
- When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
- When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
- When we examine the differences in acquiescence response styles (ARS), the patterns are similar to those observed for ERSI. There are significant differences in ARS between different data collection modes, but the differences are not systematic. Therefore, it is not possible to confidently state that one particular mode or scale type is likely to result in greater or lesser acquiescence bias. Again, the data in Exhibit E indicate that the mobile web slider produces lower levels of ARS than the other methods in 4 out of 7 scales, but these differences are statistically significant only for two scales (lie tolerance and AHO).
- do we have more users?
- do we have more users?
- do we have more users?