A presentation for Open Source Bridge that summarizes the 6 weeks of research and development that I just wrapped up while at the Banff Centre for a residency on Arts and Sciences for Saul and Charles. See: http://opensourcebridge.org/sessions/246 We live in dense urban landscapes that have crisis at many different scales. A crisis may be a large event such as a tsunami, an earthquake, a terrorist attack or other large scale disaster. A crisis may be something smaller such as a house fire, or a pileup on the freeway, a gas leak or a road out. A crisis may be something relatively minor, a crisis only to one person, such as being locked out of ones car, or trying to find a lost pet, or needing help sharing babysitting duties. For the purpose of this talk I am interested in the far end of the long tail of crisis response – and what open source technology we can strap together to build tactical systems for helping tackle such crisis. I have been developing an approach using off the shelf tools that can help resolve local crisis – the smaller more personal crisis that interrupts peoples lives. If we watch messages over a city, and provide analytic views for interested parties, then a role for “angels” to participate emerges. Interested individuals can act in a matchmaking capacity to connect complementary interests together. We can collaboratively up-score or down-score public comments aggregated from a variety of geo local sources. We can filter away noise so that responders can more easily decide what to respond to. In sum we can make it possible for people on the ground, in the area, to volunteer to help out by providing clarity. I consider these class of services to be a kind of ubiquitous angel. They are not something that you have to even know exist until they help you. The tools that we can leverage today to build such services are worth knowing – having many applications. In particular solr ( an enterprise search engine ) and carrot2 ( a topic clustering engine ) can play a powerful role in helping filter noise. Other related projects in particular Ushahidi and Swift have a high degree of overlap and those technologies will be discussed as well. Beyond today there’s a role for such technology to help us deal with new kinds of crisis in the future. Over the next decades we may be facing economic and environmental concerns that are granular, evenly distributed and problematic in resolution. These kinds of tools and approaches may be useful there as well.

1. Ubiquitous Angels
Open Source Bridge Portland Oregon June 2009
http://opensourcebridge.org/sessions/246
twitter.com/anselm
blog.makerlab.org
anselm hook

2. What I’ll cover
• A look at other similar sensor efforts
• A little bit of technicals
• Critical perspectives

4. If
• we could see our neighborhoods better
• There might be an opportunity to participate
• We could better help people in need
• We could connect people together with
similar needs
• We could crowd-source social problems
• We all know this.

6. A big use case
• #iranelection
• where are the protests?
• are there specific instances of vote fraud?
• who are the key twitterers?
• are there specific things we can do?
• are there local people who can help?

8. A smaller use case
• Restaurants often dump food every night
• Gleaners try to collect food
• How can we connect dumpers with
gleaners?

9. Other use cases
• My child is missing.
• Somebody just stole my bike.
• Can somebody give me a ride?
• Does anybody have a place where I can stay?
• Car pooling?
• I’d like somebody to share my war stories
with.

10. Crisis at many levels

11. Where is the real time
map of us?

12. Hard to Automate
• ‘Automated maps of feeds look like
database barf on a map’ - Chris Blow

14. Ushahidi Swift Initiative
• blog.ushahidi.com/index.php/tag/swift/
• a crisis listening tool
• the Mumbai Crisis for example
• electoral fraud events such as in Iran
• the upcoming quake in San Francisco
• crowd source filtering to eliminate noise
• crisis editors mark posts up with metadata
• first responders can see filtered views

15. Ushahidi Swift

16. Swift Control Panel

17. Code and Praxis

18. A software stack for a
civic lens
• search - to help restrict the firehose that is twitter and co.
• aggregation - as tuned by search
• reaping - throwing away old data as an ongoing process
• actual fetching content from twitter and co.
• geolocation of content by any means possible
• natural language parsing for semantic content and meaning
• language translation from foreign languages to english
• presentation layer for the sit-at-home-view and in-the-field views
• curatorial role in marking data up or changing metadata
• first responder views, feedback and loop closure

19. Roles
• Participant - somebody involved in an issue
• Reporter - as an on the ground witness
• Curator - acting as an editor to filter reports
• Matchmaker - matching reports with
potential responders
• Responder - going into the field to help
resolve a situation.

20. John Nunemaker’s
Twitter Gem
• http://github.com/jnunemaker/twitter
• http://apiwiki.twitter.com
• h = Twitter::HTTPAuth.new(name,pass)
• twitter = Twitter::Base.new(h)
• results = twitter.user_timeline(:id =>‘joi’)
• results.each { |twit| puts twit.location }

21. Social Graph is easy
• twitter.friends(:id => ‘joi’ )
• Try to rate limit queries
• Build your own model of the social graph
• Easy to do, crucial to provide subjectivity
• An alternative is to go through YQL
• Social graph issues underpin many projects

22. Fetching social graph

23. YQL for the win
• yql = “http://query.yahooapis.com/v1/public/yql?q=”
• query = “select * from party where id = ‘joi’ or id =
‘meedan’ or id = ‘bob’” ...
• schema = “use ‘http://angel.makerlab.org/yql/
twitter.user.timeline.xml’ as party;”
• url = “#{yql}#{url_escape(schema+query)}
• url = url + “format=json”;
• data = JSON.parse(open(url).read)
• data[“query”][“results”].each { |i| puts i.title }
• a higher rate limit ~may~ allow more data access

24. Metacarta Geolocation
• request = Net::HTTP::Get.new(‘/webservices/
GeoTagger/JSON/basic?version=1.0.0&doc=banff!’)
• request.basic_auth name,pass
• h = Net::HTTP.start(‘ondemand.metacarta.com’)
• response = h.request(request)
• data = JSON.parse(response.body)
• latitude = data[“locations”][0][“Centroid”][“Latitude”]

25. Yahoo Placemaker
• geolocating posts by brute force
• url = URI.parse(‘http://wherein.yahooapis.com/v1/document’)
• args = {‘documentContent’ => ‘banff’,
‘documentType’ => ‘text/plain’,
‘appid’=>mysecretappkey }
• r = Net::HTTP.post_form(url,args)
• doc = Hpricot::XML(response.body)
• latitude = (node/:latitude).innerHTML
• longitude = (node/:longitude).innerHTML

27. acts_as_solr for search
• picky and brittle
• doesn’t play well with others
• can geo-constrain searches more or less
• query = “volunteer lat:[42 TO 60]”
• Model.rebuild_solr_index()
• results = Model.find_by_solr(query)
• results.docs { |data| puts data.name }

28. engtagger for meaning
extraction
• engtagger.rubyforge.org
• phrase = “is anyone awake out there? i just got
mugged in front of my house and could really use
help calling the police”
• tagger = EngTagger.new
• tagged = tgr.add_tags(phrase)
• nouns = tagger.get_nouns(tagged)
• {“police”=>1,”help”=>1, ... }

29. Clustering Approaches
to clump activity
• Carrot2
• Amazon Turk
• Human ‘volunteer turk’ approach
• Contextual Network Graphs
• Latent Semantic Indexing

30. Carrot2
• project.carrot2.org
• Submit your requests directly via http
• Can only cluster 1000 documents at a time
• Used as a way to get even more metadata
• Forms clusters dynamically

32. Brief Comments on
Visualization
• Views are political
• Subjectivity is crucial
• Speed of user interface; probably gamelike
• Tactical; showing immediate context
• http://map.cartagen.org for example

34. Still looking for similar
efforts - any advice?
• I call this class of services ‘real time
brokerages’. But I don’t see many of them.
• Craigslist?
• Shizzow?
• IceCondor?
• Other?

35. Criticism
• Side effects often dominate over intended consequences of any project or endeavor.
• Abstract views may make us callous or may badly reflect ground truth.
• Ignorance is bliss. The world is filled with sob stories. Best to not dwell?
• Such services may be used solely for the most banal aspirations and goals.
• Struggle may be important - making things too easy may harm fitness and lower
diversity of skills and abilities over long term.
• What about poor people who are outside of any implied technological social
network?
• Why not just help people around you?
• Any technology should go hand in hand with day to day personal practice that is
unmediated?
• Feedback loops may be created that accelerate and disrupt society.
• Virtual and visual only for curators; not tactile; uses only one sense. A concern?

36. links
http://swiftapp.org/
http://crisismapping.ning.com/
http://ushahidi.com
http://ncfoodnet.com/
http://www.flickr.com/photos/unthinkingly/3507131912/
http://cartagen.net
http://blog.twittervotereport.com/
http://blog.ushahidi.com/?s=swift
http://www.qapture.net/lists/
http://www.marketingpilgrim.com/2008/12/social-media-monitoring-tools.html
http://tinker.com/users/find/?findName=&findEmail=&findLocation=banff&profType=
http://www.crunchgear.com/2007/04/02/patent-monkey-geographic-cell-messaging-with-squareloop/
http://www.dantynan.com/2009/04/02/twitter-added-to-federal-emergency-response-network/ ( April Fools Gag )
http://stephensonstrategies.com/directory-of-major-blog-posts/boy-did-i-underestimate-twitters-value-in-a-disaster/
http://www.comminit.com/en/node/290987/307
http://yvoschaap.com/twitterthoughts/?map=1
http://mashable.com/2009/03/16/twitter-visualizations/
http://theeword.blogspot.com/2009/06/best-twitter-applications-for-social.html
http://monitter.com/
http://maphawk.blogspot.com/2009/01/geographic-search-engine-for-news.html
http://muti.co.za/static/newsmap.html
http://www.planetizen.com/node/38227
http://www.localtweeps.com/
http://keithhopper.com/taxonomy/term/28
http://keithhopper.com/blog/finding-twitterstars
http://pipes.yahoo.com/pipes/pipe.run?Location=portland%2C+oregon&radius=15&_render=rss&_id=2fdf06974be88f13a63937fb4d36e499
http://delicious.com/tag/twitter+map
http://www.twitip.com/twellowhood-find-local-twitter-users/
http://www.tweepz.com/feed?q=loc%3Abanff&C=
http://www.twitterlocal.net/ ( old )
http://www.frontlinesms.com/

37. thanks
http://github.com/
anselm/angel
http://github.com/
anselm/SpinnyGlobe