SlideShare ist ein Scribd-Unternehmen logo
1 von 27
Downloaden Sie, um offline zu lesen
Online Communities in
    Citizen Science
           Andrea Wiggins
              Postdoctoral Fellow
University of New Mexico & Cornell University

           8 November, 2012
What is citizen science?
Members of the public engaging in real-world
scientific research
 Crowdsourcing
 Collaboration
 Community
What is citizen science?

               public
            participation
             in science
           cro                                 r
         so w d -                           ee g
                                          nt in
           urc
               ing                     olu itor
                                      v n
                                       mo

 online
 communities
                        *
                     infrastructure
                         cyber-
                                              scientific
                                         collaboration




           = citizen science
What is citizen science?
                                intelligent
                               mobile sensor
                                 net work


                        ?
crowdsourcing science

    human
computation      Public Participation
                in Scientific Research
Typ(ologi)es of citizen science


Goals and place
Participation in scientific tasks
Contribution types
Participant experiences
and more...
Participation in scientific tasks
PPSR$models:                 Contributory*   Collabora1ve* CoACreated*


Define*a*ques1on/issue*
Gather*informa1on*
Develop*explana1ons*
Design*data*collec1on*methods*
Collect*samples*
Analyze*samples*
Analyze*data*
Interpret*data/conclude*
Disseminate*conclusions*
Discuss*results/inquire*further*
                                    CAISE Report, 2009
Goals and the role of place
   Type         Primary Goals     Physicality
                   Action &
   Action                             ✓
                 Intervention
                Conservation &
Conservation                          ✓
                 Stewardship

Investigation      Science            ✓

   Virtual         Science

                 Education &
 Education                            ✓
                  Outreach
                       Wiggins & Crowston, 2011
Contribution types
 Data collection
• Most common
• Observations & measurements


 Data processing
• On the rise
• Entirely virtual
• Image recognition & puzzle solving


 Data transcription
• On the rise
• Mostly virtual
Participant experience
Sharing my data/experiences
• Fits into daily life

• People like to share their passions

Working on their/our tasks
• Novel tasks

• Can reinforce hierarchy

Playing games & solving puzzles
• Fits into daily life

• Explicit symbolic rewards, entertaining
What does it accomplish?
engage%cri)cal%thinking%
(Trumbull%et%al%2000)%


science%learning,%bonding%
(Kountoupes%and%Oberhauser%2008)%


environmental%ac)on;%social%networks%
(Overdevest%et%al.%2004)%


social%capital%
(Ballard%2008)%


improved%policy%
(Wing%et%al.%2008)%
What does it accomplish?
documen(ng*range*shi0s*
(Bonter*et*al.*unpublished*data)*


iden(fying*poten(al*mismatches*
(Batalden*et*al.*2007)*


iden(fying*vulnerable*species*
(Crimmins*et*al*2008,*2009)*


health*planning*
(Leve(n*and*Van*de*Water*2008)*


an(cipa(ng*effects*on*water*sources*
(e.g.,*CoCoRaHS)*


processing	
  large	
  image	
  data	
  sets
(e.g.,	
  Zooniverse	
  projects)

applying	
  human	
  computa:on	
  skills
(e.g.,	
  Foldit)
Galaxy Zoo
 Classifying images of galaxies
 Participation involves:
 Looking at pictures of galaxies online
 Answering a few questions about them

Started in 2007 by a team of academic astronomers
 Instant success and exciting new discoveries
 Galaxy Zoo 1, Year 1: 50M classifications, 150K volunteers
 Galaxy Zoo 2, Year 2: 60M classifications in 14 months
 Hanny’s Voorwerp
 Green Pea galaxies
eBird
   Collecting bird abundance and distribution data
   Participation involves:
    Choosing observation methods
    Recording bird observations (analog or digital)
    Entering observations and metadata online

Launched in 2002 by Cornell Lab of Ornithology
(with National Audubon Society)

World’s largest biodiversity data set: 100M records
Currently receives about 3M observations/month
Data used in research and decision-making for land
management, policy (and recreation)
Crowds vs Communities


Is citizen science crowdsourcing?
When is it crowdsourcing, and when is it not?
What separates a crowd from a community?
Crowdsourcing versus communitysourcing?
Motivations

 Galaxy Zoo
Motivations: similar to other participant surveys
Forums: evidence of shared interests & practices

 Alice
Reader to Leader...
...to Scientist!

               http://blog.galaxyzoo.org/2009/07/01/shes-an-astronomer-alice-sheppard/
More than just motivation

Motivations
 Intrinsic (altruism)
 Extrinsic (money)
 Dynamic
Personal Values (domain, science)
Individual Goals (contributing)
               http://www.flickr.com/photos/verbeeldingskr8/4875710270/
Implications for Design

Who will participate?
Why will they
participate?
How will they be
rewarded?
How can experiences be
expanded?
                   http://www.flickr.com/photos/andymangold/4335799638/
Which Came First?

Science-first project design
  Tech focuses on data entry
  Experiences are usually
  simplified science
Citizen-first project design
  Tech focuses on ease of use
  Experience adapts existing leisure practices
    Self-rewarded & socially rewarded
                    http://www.flickr.com/photos/hammer51012/495218105/
ICTs for Citizen Science

 Web-based data entry & social tools
“Vanilla” website
CMS
Custom platform

 Mobile
 Games
CitizenSort         image courtesy
                    of Eric Graham
Images courtesy of Nathan Prestopnik and the
   CitizenSort team at Syracuse University
            US NSF Grant # 09-68470
Online Communities

We don’t know much!
Primary social & communication tool in virtual (place-
independent) projects
But what about...
   Place-dependent projects

   Pre-existing communities

   Scale of participation

Cohesive systems vs system assemblages
What makes it a community?

How did you judge whether a citizen science
project you reviewed has an online
community or not?
 Blogs + forums + social media
 Leaderboards, visibility of others’ data, “shared
 checklists”

Were there signs of offline community?
 What are the implications of that?
Communities in Citizen Science

Primary recruitment method (in place-based)
 Not feasible for entirely virtual projects

Important for retention
 Come for the coffee, stay for the conversation

Main community engagement strategies
 Traditional: tapping into existing communities
 Virtual: creating a new community
Unrealized potential value

Retention tool
Recruitment tool
Capture serendipity
Promote inquiry
Gather evaluation data
More...
Challenges

Strategic implementation required with
existing communities
$$$
Active management
Technical resources

Zooniverse research underway to learn more
Yay, Sloan Foundation!
Thanks!

andrea.wiggins@cornell.edu
@AndreaWiggins


dataone.org
citizenscience.org
andreawiggins.com
Typologies
• Lawrence,  A. (2006). “No Personal Motive?” Volunteers, Biodiversity, and the False Dichotomies of
 Participation. Ethics,	
  Place	
  &	
  Environment, 9(3), 279-298.

• Bonney,  R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., et al. (2009). Public Participation in
 Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. A
 CAISE Inquiry Group Report (Tech. Rep.).

• Danielsen, F., Burgess, N., Balmford, A., Donald, P., Funder, M., Jones, J., et al. (2009). Local
 participation in natural resource monitoring: a characterization of approaches. Conserva4on	
  Biology,
 23(1), 31–42.

• Cooper,C. B., Dickinson, J., Phillips, T., & Bonney, R. (2007). Citizen Science as a Tool for Conservation
 in Residential Ecosystems.	
  Ecology	
  and	
  Society, 12(2).

• Wilderman,  C. C. (2007). Models of community science: design lessons from the field. Proceedings of
 Citizen Science Toolkit Conference.

• Wiggins,A. & Crowston, K. (2011). From Conservation to Crowdsourcing: A Typology of Citizen
 Science. Proceedings of the 44th Annual Hawaii International Conference on System Sciences.

• Wiggins,
         A. & Crowston, K. (2012). Goals and Tasks: Two Typologies of Citizen Science Projects.
 Proceedings of the 45th Annual Hawaii International Conference on Systems Sciences.
                                                                                                                        16

Weitere ähnliche Inhalte

Was ist angesagt?

Citizen Science Phenotypes
Citizen Science PhenotypesCitizen Science Phenotypes
Citizen Science PhenotypesAndrea Wiggins
 
Citizen science
Citizen scienceCitizen science
Citizen sciencesamar1407
 
Data Intensive Collaboration in Science and Engineering: CSCW workshop themes
Data Intensive Collaboration in Science and Engineering: CSCW workshop themesData Intensive Collaboration in Science and Engineering: CSCW workshop themes
Data Intensive Collaboration in Science and Engineering: CSCW workshop themesAndrea Wiggins
 
What's up at Kno.e.sis?
What's up at Kno.e.sis? What's up at Kno.e.sis?
What's up at Kno.e.sis? Amit Sheth
 
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...Artificial Intelligence Institute at UofSC
 
EPA 2013 Air Sensors Meeting Big Data Talk
EPA 2013 Air Sensors Meeting Big Data TalkEPA 2013 Air Sensors Meeting Big Data Talk
EPA 2013 Air Sensors Meeting Big Data TalkAdina Chuang Howe
 
What is eScience, and where does it go from here?
What is eScience, and where does it go from here?What is eScience, and where does it go from here?
What is eScience, and where does it go from here?Daniel S. Katz
 
Maintainable Software Practices for e-Science - Introduction to Workshop
Maintainable Software Practices for e-Science - Introduction to WorkshopMaintainable Software Practices for e-Science - Introduction to Workshop
Maintainable Software Practices for e-Science - Introduction to WorkshopSoftwarePractice
 
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...EarthCube
 
Digital Monitoring of societal Discussions in online Social Networks
Digital Monitoring of societal Discussions in online Social NetworksDigital Monitoring of societal Discussions in online Social Networks
Digital Monitoring of societal Discussions in online Social NetworksTimo Wandhoefer
 
Supporting Libraries in Leading the Way in Research Data Management
Supporting Libraries in Leading the Way in Research Data ManagementSupporting Libraries in Leading the Way in Research Data Management
Supporting Libraries in Leading the Way in Research Data ManagementMarieke Guy
 
Social Machines of Science and Scholarship
Social Machines of Science and ScholarshipSocial Machines of Science and Scholarship
Social Machines of Science and ScholarshipDavid De Roure
 
Scholarship in the Digital World
Scholarship in the Digital WorldScholarship in the Digital World
Scholarship in the Digital WorldDavid De Roure
 
Ed Fox on Learning Technologies
Ed Fox on Learning TechnologiesEd Fox on Learning Technologies
Ed Fox on Learning TechnologiesGardner Campbell
 

Was ist angesagt? (20)

Citizen Science and Inquiry
Citizen Science and InquiryCitizen Science and Inquiry
Citizen Science and Inquiry
 
Citizen Science Phenotypes
Citizen Science PhenotypesCitizen Science Phenotypes
Citizen Science Phenotypes
 
Little eScience
Little eScienceLittle eScience
Little eScience
 
Crowdsourcing Science
Crowdsourcing ScienceCrowdsourcing Science
Crowdsourcing Science
 
Citizen science
Citizen scienceCitizen science
Citizen science
 
Data Intensive Collaboration in Science and Engineering: CSCW workshop themes
Data Intensive Collaboration in Science and Engineering: CSCW workshop themesData Intensive Collaboration in Science and Engineering: CSCW workshop themes
Data Intensive Collaboration in Science and Engineering: CSCW workshop themes
 
Knoesis Student Achievement
Knoesis Student AchievementKnoesis Student Achievement
Knoesis Student Achievement
 
Engaging the software in research community
Engaging the software in research communityEngaging the software in research community
Engaging the software in research community
 
What's up at Kno.e.sis?
What's up at Kno.e.sis? What's up at Kno.e.sis?
What's up at Kno.e.sis?
 
Oess NCRM Festival
Oess NCRM FestivalOess NCRM Festival
Oess NCRM Festival
 
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...
Data Processing and Semantics for Advanced Internet of Things (IoT) Applicati...
 
EPA 2013 Air Sensors Meeting Big Data Talk
EPA 2013 Air Sensors Meeting Big Data TalkEPA 2013 Air Sensors Meeting Big Data Talk
EPA 2013 Air Sensors Meeting Big Data Talk
 
What is eScience, and where does it go from here?
What is eScience, and where does it go from here?What is eScience, and where does it go from here?
What is eScience, and where does it go from here?
 
Maintainable Software Practices for e-Science - Introduction to Workshop
Maintainable Software Practices for e-Science - Introduction to WorkshopMaintainable Software Practices for e-Science - Introduction to Workshop
Maintainable Software Practices for e-Science - Introduction to Workshop
 
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...
AHM 2014: The iPlant Collaborative, Community Cyberinfrastructure for Life Sc...
 
Digital Monitoring of societal Discussions in online Social Networks
Digital Monitoring of societal Discussions in online Social NetworksDigital Monitoring of societal Discussions in online Social Networks
Digital Monitoring of societal Discussions in online Social Networks
 
Supporting Libraries in Leading the Way in Research Data Management
Supporting Libraries in Leading the Way in Research Data ManagementSupporting Libraries in Leading the Way in Research Data Management
Supporting Libraries in Leading the Way in Research Data Management
 
Social Machines of Science and Scholarship
Social Machines of Science and ScholarshipSocial Machines of Science and Scholarship
Social Machines of Science and Scholarship
 
Scholarship in the Digital World
Scholarship in the Digital WorldScholarship in the Digital World
Scholarship in the Digital World
 
Ed Fox on Learning Technologies
Ed Fox on Learning TechnologiesEd Fox on Learning Technologies
Ed Fox on Learning Technologies
 

Ähnlich wie Online Communities in Citizen Science

The role of learning in citizen science
The role of learning in citizen scienceThe role of learning in citizen science
The role of learning in citizen scienceMuki Haklay
 
The role of learning in community science and citizen science
The role of learning in community science and citizen scienceThe role of learning in community science and citizen science
The role of learning in community science and citizen scienceMuki Haklay
 
Extreme Citizen Science: the socio-political potential of citizen science
Extreme Citizen Science: the socio-political potential of citizen scienceExtreme Citizen Science: the socio-political potential of citizen science
Extreme Citizen Science: the socio-political potential of citizen scienceMuki Haklay
 
Summer Social Webshop: Technology-Mediated Social Participation
Summer Social Webshop: Technology-Mediated Social ParticipationSummer Social Webshop: Technology-Mediated Social Participation
Summer Social Webshop: Technology-Mediated Social ParticipationUniversity of Maryland
 
Citizen science presentation
Citizen science presentationCitizen science presentation
Citizen science presentationHessah Alajmi
 
Enrique RCODI presentation symposium 2017
Enrique RCODI presentation symposium 2017Enrique RCODI presentation symposium 2017
Enrique RCODI presentation symposium 2017Jesus Enrique Aldana S.
 
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...Cindy Regalado
 
The ECSA Characteristics of Citizen Science
The ECSA Characteristics of Citizen ScienceThe ECSA Characteristics of Citizen Science
The ECSA Characteristics of Citizen ScienceMargaret Gold
 
Netnography webinar
Netnography webinarNetnography webinar
Netnography webinarsuresh sood
 
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...Ramine Tinati
 
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?Todd Suomela
 
Citizen Science & Geographical Technologies: creativity, learning, and engage...
Citizen Science & Geographical Technologies: creativity, learning, and engage...Citizen Science & Geographical Technologies: creativity, learning, and engage...
Citizen Science & Geographical Technologies: creativity, learning, and engage...Muki Haklay
 
Oxford Martin School talk - May 2014
Oxford Martin School talk - May 2014Oxford Martin School talk - May 2014
Oxford Martin School talk - May 2014Muki Haklay
 
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen Science
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen ScienceECSA, the ECSA principles, and the ECSA Characteristics of Citizen Science
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen ScienceMargaret Gold
 
Cal academy convenings presentation rc
Cal academy convenings presentation rcCal academy convenings presentation rc
Cal academy convenings presentation rcrhiannoncrain
 
Citizen-led perspectives: bridging the gap
Citizen-led perspectives: bridging the gapCitizen-led perspectives: bridging the gap
Citizen-led perspectives: bridging the gapErinma Ochu
 
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"Darlene Cavalier
 
British columbia september2014_final
British columbia september2014_finalBritish columbia september2014_final
British columbia september2014_finalcarolbumd
 

Ähnlich wie Online Communities in Citizen Science (20)

The role of learning in citizen science
The role of learning in citizen scienceThe role of learning in citizen science
The role of learning in citizen science
 
The role of learning in community science and citizen science
The role of learning in community science and citizen scienceThe role of learning in community science and citizen science
The role of learning in community science and citizen science
 
Extreme Citizen Science: the socio-political potential of citizen science
Extreme Citizen Science: the socio-political potential of citizen scienceExtreme Citizen Science: the socio-political potential of citizen science
Extreme Citizen Science: the socio-political potential of citizen science
 
Summer Social Webshop: Technology-Mediated Social Participation
Summer Social Webshop: Technology-Mediated Social ParticipationSummer Social Webshop: Technology-Mediated Social Participation
Summer Social Webshop: Technology-Mediated Social Participation
 
Citizen science presentation
Citizen science presentationCitizen science presentation
Citizen science presentation
 
Enrique RCODI presentation symposium 2017
Enrique RCODI presentation symposium 2017Enrique RCODI presentation symposium 2017
Enrique RCODI presentation symposium 2017
 
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...
What is Extreme Citizen Science? Volunteerism & Publicly Initiated Scientific...
 
The ECSA Characteristics of Citizen Science
The ECSA Characteristics of Citizen ScienceThe ECSA Characteristics of Citizen Science
The ECSA Characteristics of Citizen Science
 
Netnography webinar
Netnography webinarNetnography webinar
Netnography webinar
 
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...
WSI Stimulus Project: Centre for longitudinal studies of online citizen parti...
 
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?
DigiCCurr 2013 PhD Workshop - Citizen Science and Data Curation: Who needs what?
 
Citizen Science & Geographical Technologies: creativity, learning, and engage...
Citizen Science & Geographical Technologies: creativity, learning, and engage...Citizen Science & Geographical Technologies: creativity, learning, and engage...
Citizen Science & Geographical Technologies: creativity, learning, and engage...
 
Cooper "Simplicity is the Ultimate Sophistication: Accessible, Ubiquitous Tec...
Cooper "Simplicity is the Ultimate Sophistication: Accessible, Ubiquitous Tec...Cooper "Simplicity is the Ultimate Sophistication: Accessible, Ubiquitous Tec...
Cooper "Simplicity is the Ultimate Sophistication: Accessible, Ubiquitous Tec...
 
Oxford Martin School talk - May 2014
Oxford Martin School talk - May 2014Oxford Martin School talk - May 2014
Oxford Martin School talk - May 2014
 
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen Science
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen ScienceECSA, the ECSA principles, and the ECSA Characteristics of Citizen Science
ECSA, the ECSA principles, and the ECSA Characteristics of Citizen Science
 
Cal academy convenings presentation rc
Cal academy convenings presentation rcCal academy convenings presentation rc
Cal academy convenings presentation rc
 
Crowdsourcing: A Geographic Approach to Identifying Policy Opportunities and ...
Crowdsourcing: A Geographic Approach to Identifying Policy Opportunities and ...Crowdsourcing: A Geographic Approach to Identifying Policy Opportunities and ...
Crowdsourcing: A Geographic Approach to Identifying Policy Opportunities and ...
 
Citizen-led perspectives: bridging the gap
Citizen-led perspectives: bridging the gapCitizen-led perspectives: bridging the gap
Citizen-led perspectives: bridging the gap
 
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"
Citizen Science overview for ASU HSD598 graduate course, "Citizen Science"
 
British columbia september2014_final
British columbia september2014_finalBritish columbia september2014_final
British columbia september2014_final
 

Mehr von Andrea Wiggins

Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...
Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...
Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...Andrea Wiggins
 
With Great Data Comes Great Responsibility
With Great Data Comes Great ResponsibilityWith Great Data Comes Great Responsibility
With Great Data Comes Great ResponsibilityAndrea Wiggins
 
Mechanisms for Data Quality and Validation in Citizen Science
Mechanisms for Data Quality and Validation in Citizen ScienceMechanisms for Data Quality and Validation in Citizen Science
Mechanisms for Data Quality and Validation in Citizen ScienceAndrea Wiggins
 
Open Source & Citizen Science
Open Source & Citizen ScienceOpen Source & Citizen Science
Open Source & Citizen ScienceAndrea Wiggins
 
From Conservation to Crowdsourcing: A Typology of Citizen Science
From Conservation to Crowdsourcing: A Typology of Citizen ScienceFrom Conservation to Crowdsourcing: A Typology of Citizen Science
From Conservation to Crowdsourcing: A Typology of Citizen ScienceAndrea Wiggins
 
Motivation by Design: Technologies, Experiences, and Incentives
Motivation by Design: Technologies, Experiences, and IncentivesMotivation by Design: Technologies, Experiences, and Incentives
Motivation by Design: Technologies, Experiences, and IncentivesAndrea Wiggins
 
Secondary data analysis with digital trace data
Secondary data analysis with digital trace dataSecondary data analysis with digital trace data
Secondary data analysis with digital trace dataAndrea Wiggins
 
Reclassifying Success and Tragedy in FLOSS Projects
Reclassifying Success and Tragedy in FLOSS ProjectsReclassifying Success and Tragedy in FLOSS Projects
Reclassifying Success and Tragedy in FLOSS ProjectsAndrea Wiggins
 
Intellectual Diversity in the iSchools: Past, Present and Future
Intellectual Diversity in the iSchools: Past, Present and FutureIntellectual Diversity in the iSchools: Past, Present and Future
Intellectual Diversity in the iSchools: Past, Present and FutureAndrea Wiggins
 
Distributed Scientific Collaboration: Research Opportunities in Citizen Science
Distributed Scientific Collaboration: Research Opportunities in Citizen ScienceDistributed Scientific Collaboration: Research Opportunities in Citizen Science
Distributed Scientific Collaboration: Research Opportunities in Citizen ScienceAndrea Wiggins
 
Designing Virtual Organizations for Citizen Science
Designing Virtual Organizations for Citizen ScienceDesigning Virtual Organizations for Citizen Science
Designing Virtual Organizations for Citizen ScienceAndrea Wiggins
 
National Park System Property Designations
National Park System Property DesignationsNational Park System Property Designations
National Park System Property DesignationsAndrea Wiggins
 
Collaborative Data Analysis with Taverna Workflows
Collaborative Data Analysis with Taverna WorkflowsCollaborative Data Analysis with Taverna Workflows
Collaborative Data Analysis with Taverna WorkflowsAndrea Wiggins
 
Tales of the Field: Building Small Science Cyberinfrastructure
Tales of the Field: Building Small Science CyberinfrastructureTales of the Field: Building Small Science Cyberinfrastructure
Tales of the Field: Building Small Science CyberinfrastructureAndrea Wiggins
 
Coordination Dynamics in Free/Libre and Open Source Software
Coordination Dynamics in Free/Libre and Open Source SoftwareCoordination Dynamics in Free/Libre and Open Source Software
Coordination Dynamics in Free/Libre and Open Source SoftwareAndrea Wiggins
 
Heartbeat: Measuring Active User Base and Potential User Interest
Heartbeat: Measuring Active User Base and Potential User InterestHeartbeat: Measuring Active User Base and Potential User Interest
Heartbeat: Measuring Active User Base and Potential User InterestAndrea Wiggins
 
Replicating FLOSS Research as eResearch
Replicating FLOSS Research as eResearchReplicating FLOSS Research as eResearch
Replicating FLOSS Research as eResearchAndrea Wiggins
 
Social dynamics of FLOSS team communication across channels
Social dynamics of FLOSS team communication across channelsSocial dynamics of FLOSS team communication across channels
Social dynamics of FLOSS team communication across channelsAndrea Wiggins
 
eResearch workflows for studying free and open source software development
eResearch workflows for studying free and open source software developmenteResearch workflows for studying free and open source software development
eResearch workflows for studying free and open source software developmentAndrea Wiggins
 

Mehr von Andrea Wiggins (19)

Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...
Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...
Crowdsourcing Citizen Science Data Quality with a Human-Computer Learning Net...
 
With Great Data Comes Great Responsibility
With Great Data Comes Great ResponsibilityWith Great Data Comes Great Responsibility
With Great Data Comes Great Responsibility
 
Mechanisms for Data Quality and Validation in Citizen Science
Mechanisms for Data Quality and Validation in Citizen ScienceMechanisms for Data Quality and Validation in Citizen Science
Mechanisms for Data Quality and Validation in Citizen Science
 
Open Source & Citizen Science
Open Source & Citizen ScienceOpen Source & Citizen Science
Open Source & Citizen Science
 
From Conservation to Crowdsourcing: A Typology of Citizen Science
From Conservation to Crowdsourcing: A Typology of Citizen ScienceFrom Conservation to Crowdsourcing: A Typology of Citizen Science
From Conservation to Crowdsourcing: A Typology of Citizen Science
 
Motivation by Design: Technologies, Experiences, and Incentives
Motivation by Design: Technologies, Experiences, and IncentivesMotivation by Design: Technologies, Experiences, and Incentives
Motivation by Design: Technologies, Experiences, and Incentives
 
Secondary data analysis with digital trace data
Secondary data analysis with digital trace dataSecondary data analysis with digital trace data
Secondary data analysis with digital trace data
 
Reclassifying Success and Tragedy in FLOSS Projects
Reclassifying Success and Tragedy in FLOSS ProjectsReclassifying Success and Tragedy in FLOSS Projects
Reclassifying Success and Tragedy in FLOSS Projects
 
Intellectual Diversity in the iSchools: Past, Present and Future
Intellectual Diversity in the iSchools: Past, Present and FutureIntellectual Diversity in the iSchools: Past, Present and Future
Intellectual Diversity in the iSchools: Past, Present and Future
 
Distributed Scientific Collaboration: Research Opportunities in Citizen Science
Distributed Scientific Collaboration: Research Opportunities in Citizen ScienceDistributed Scientific Collaboration: Research Opportunities in Citizen Science
Distributed Scientific Collaboration: Research Opportunities in Citizen Science
 
Designing Virtual Organizations for Citizen Science
Designing Virtual Organizations for Citizen ScienceDesigning Virtual Organizations for Citizen Science
Designing Virtual Organizations for Citizen Science
 
National Park System Property Designations
National Park System Property DesignationsNational Park System Property Designations
National Park System Property Designations
 
Collaborative Data Analysis with Taverna Workflows
Collaborative Data Analysis with Taverna WorkflowsCollaborative Data Analysis with Taverna Workflows
Collaborative Data Analysis with Taverna Workflows
 
Tales of the Field: Building Small Science Cyberinfrastructure
Tales of the Field: Building Small Science CyberinfrastructureTales of the Field: Building Small Science Cyberinfrastructure
Tales of the Field: Building Small Science Cyberinfrastructure
 
Coordination Dynamics in Free/Libre and Open Source Software
Coordination Dynamics in Free/Libre and Open Source SoftwareCoordination Dynamics in Free/Libre and Open Source Software
Coordination Dynamics in Free/Libre and Open Source Software
 
Heartbeat: Measuring Active User Base and Potential User Interest
Heartbeat: Measuring Active User Base and Potential User InterestHeartbeat: Measuring Active User Base and Potential User Interest
Heartbeat: Measuring Active User Base and Potential User Interest
 
Replicating FLOSS Research as eResearch
Replicating FLOSS Research as eResearchReplicating FLOSS Research as eResearch
Replicating FLOSS Research as eResearch
 
Social dynamics of FLOSS team communication across channels
Social dynamics of FLOSS team communication across channelsSocial dynamics of FLOSS team communication across channels
Social dynamics of FLOSS team communication across channels
 
eResearch workflows for studying free and open source software development
eResearch workflows for studying free and open source software developmenteResearch workflows for studying free and open source software development
eResearch workflows for studying free and open source software development
 

Kürzlich hochgeladen

Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationKnoldus Inc.
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Mark Simos
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxfnnc6jmgwh
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Strongerpanagenda
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesBernd Ruecker
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 

Kürzlich hochgeladen (20)

Data governance with Unity Catalog Presentation
Data governance with Unity Catalog PresentationData governance with Unity Catalog Presentation
Data governance with Unity Catalog Presentation
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptxGenerative AI - Gitex v1Generative AI - Gitex v1.pptx
Generative AI - Gitex v1Generative AI - Gitex v1.pptx
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better StrongerModern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
Modern Roaming for Notes and Nomad – Cheaper Faster Better Stronger
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
QCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architecturesQCon London: Mastering long-running processes in modern architectures
QCon London: Mastering long-running processes in modern architectures
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 

Online Communities in Citizen Science

  • 1. Online Communities in Citizen Science Andrea Wiggins Postdoctoral Fellow University of New Mexico & Cornell University 8 November, 2012
  • 2. What is citizen science? Members of the public engaging in real-world scientific research Crowdsourcing Collaboration Community
  • 3. What is citizen science? public participation in science cro r so w d - ee g nt in urc ing olu itor v n mo online communities * infrastructure cyber- scientific collaboration = citizen science
  • 4. What is citizen science? intelligent mobile sensor net work ? crowdsourcing science human computation Public Participation in Scientific Research
  • 5. Typ(ologi)es of citizen science Goals and place Participation in scientific tasks Contribution types Participant experiences and more...
  • 6. Participation in scientific tasks PPSR$models: Contributory* Collabora1ve* CoACreated* Define*a*ques1on/issue* Gather*informa1on* Develop*explana1ons* Design*data*collec1on*methods* Collect*samples* Analyze*samples* Analyze*data* Interpret*data/conclude* Disseminate*conclusions* Discuss*results/inquire*further* CAISE Report, 2009
  • 7. Goals and the role of place Type Primary Goals Physicality Action & Action ✓ Intervention Conservation & Conservation ✓ Stewardship Investigation Science ✓ Virtual Science Education & Education ✓ Outreach Wiggins & Crowston, 2011
  • 8. Contribution types Data collection • Most common • Observations & measurements Data processing • On the rise • Entirely virtual • Image recognition & puzzle solving Data transcription • On the rise • Mostly virtual
  • 9. Participant experience Sharing my data/experiences • Fits into daily life • People like to share their passions Working on their/our tasks • Novel tasks • Can reinforce hierarchy Playing games & solving puzzles • Fits into daily life • Explicit symbolic rewards, entertaining
  • 10. What does it accomplish? engage%cri)cal%thinking% (Trumbull%et%al%2000)% science%learning,%bonding% (Kountoupes%and%Oberhauser%2008)% environmental%ac)on;%social%networks% (Overdevest%et%al.%2004)% social%capital% (Ballard%2008)% improved%policy% (Wing%et%al.%2008)%
  • 11. What does it accomplish? documen(ng*range*shi0s* (Bonter*et*al.*unpublished*data)* iden(fying*poten(al*mismatches* (Batalden*et*al.*2007)* iden(fying*vulnerable*species* (Crimmins*et*al*2008,*2009)* health*planning* (Leve(n*and*Van*de*Water*2008)* an(cipa(ng*effects*on*water*sources* (e.g.,*CoCoRaHS)* processing  large  image  data  sets (e.g.,  Zooniverse  projects) applying  human  computa:on  skills (e.g.,  Foldit)
  • 12. Galaxy Zoo Classifying images of galaxies Participation involves: Looking at pictures of galaxies online Answering a few questions about them Started in 2007 by a team of academic astronomers Instant success and exciting new discoveries Galaxy Zoo 1, Year 1: 50M classifications, 150K volunteers Galaxy Zoo 2, Year 2: 60M classifications in 14 months Hanny’s Voorwerp Green Pea galaxies
  • 13. eBird Collecting bird abundance and distribution data Participation involves: Choosing observation methods Recording bird observations (analog or digital) Entering observations and metadata online Launched in 2002 by Cornell Lab of Ornithology (with National Audubon Society) World’s largest biodiversity data set: 100M records Currently receives about 3M observations/month Data used in research and decision-making for land management, policy (and recreation)
  • 14. Crowds vs Communities Is citizen science crowdsourcing? When is it crowdsourcing, and when is it not? What separates a crowd from a community? Crowdsourcing versus communitysourcing?
  • 15. Motivations Galaxy Zoo Motivations: similar to other participant surveys Forums: evidence of shared interests & practices Alice Reader to Leader... ...to Scientist! http://blog.galaxyzoo.org/2009/07/01/shes-an-astronomer-alice-sheppard/
  • 16. More than just motivation Motivations Intrinsic (altruism) Extrinsic (money) Dynamic Personal Values (domain, science) Individual Goals (contributing) http://www.flickr.com/photos/verbeeldingskr8/4875710270/
  • 17. Implications for Design Who will participate? Why will they participate? How will they be rewarded? How can experiences be expanded? http://www.flickr.com/photos/andymangold/4335799638/
  • 18. Which Came First? Science-first project design Tech focuses on data entry Experiences are usually simplified science Citizen-first project design Tech focuses on ease of use Experience adapts existing leisure practices Self-rewarded & socially rewarded http://www.flickr.com/photos/hammer51012/495218105/
  • 19. ICTs for Citizen Science Web-based data entry & social tools “Vanilla” website CMS Custom platform Mobile Games CitizenSort image courtesy of Eric Graham
  • 20. Images courtesy of Nathan Prestopnik and the CitizenSort team at Syracuse University US NSF Grant # 09-68470
  • 21. Online Communities We don’t know much! Primary social & communication tool in virtual (place- independent) projects But what about... Place-dependent projects Pre-existing communities Scale of participation Cohesive systems vs system assemblages
  • 22. What makes it a community? How did you judge whether a citizen science project you reviewed has an online community or not? Blogs + forums + social media Leaderboards, visibility of others’ data, “shared checklists” Were there signs of offline community? What are the implications of that?
  • 23. Communities in Citizen Science Primary recruitment method (in place-based) Not feasible for entirely virtual projects Important for retention Come for the coffee, stay for the conversation Main community engagement strategies Traditional: tapping into existing communities Virtual: creating a new community
  • 24. Unrealized potential value Retention tool Recruitment tool Capture serendipity Promote inquiry Gather evaluation data More...
  • 25. Challenges Strategic implementation required with existing communities $$$ Active management Technical resources Zooniverse research underway to learn more Yay, Sloan Foundation!
  • 27. Typologies • Lawrence, A. (2006). “No Personal Motive?” Volunteers, Biodiversity, and the False Dichotomies of Participation. Ethics,  Place  &  Environment, 9(3), 279-298. • Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J., et al. (2009). Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education. A CAISE Inquiry Group Report (Tech. Rep.). • Danielsen, F., Burgess, N., Balmford, A., Donald, P., Funder, M., Jones, J., et al. (2009). Local participation in natural resource monitoring: a characterization of approaches. Conserva4on  Biology, 23(1), 31–42. • Cooper,C. B., Dickinson, J., Phillips, T., & Bonney, R. (2007). Citizen Science as a Tool for Conservation in Residential Ecosystems.  Ecology  and  Society, 12(2). • Wilderman, C. C. (2007). Models of community science: design lessons from the field. Proceedings of Citizen Science Toolkit Conference. • Wiggins,A. & Crowston, K. (2011). From Conservation to Crowdsourcing: A Typology of Citizen Science. Proceedings of the 44th Annual Hawaii International Conference on System Sciences. • Wiggins, A. & Crowston, K. (2012). Goals and Tasks: Two Typologies of Citizen Science Projects. Proceedings of the 45th Annual Hawaii International Conference on Systems Sciences. 16