This document discusses expectations and challenges when visualizing data. The key points are: 1. Expect to find the real need by understanding the audience and goals better than the client. Expect to clean data, which can take a significant amount of time due to multiple sources and formats. 2. Prepare to iterate as the initial visualization may not meet needs or deadlines. Celebrate failures as learning opportunities. 3. Visualization projects include storytelling projects with strict deadlines and analytical tools to support data exploration by technical teams over the long term. The project lifecycle involves identifying needs, prototyping, refining, and maintaining the visualization.

1. Krist Wongsuphasawat / @kristw
6 THINGS TO EXPECT
WHEN YOU ARE
VISUALIZING

2. 6 THINGS TO EXPECT
WHEN YOU ARE
VISUALIZING
Krist Wongsuphasawat / @kristw

3. Computer Engineer
Bangkok, Thailand
Chulalongkorn University
Krist Wongsuphasawat / @kristw

4. Programming + Soccer
Computer Engineer
Bangkok, Thailand
Krist Wongsuphasawat / @kristw

5. Programming + Soccer
Computer Engineer
Bangkok, Thailand
Krist Wongsuphasawat / @kristw

6. (P.S. These are actually not my robots, but our competitors’.)
Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand

7. Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand
PhD in Computer Science
Information Visualization
Univ. of Maryland

8. Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand
IBM
Microsoft
PhD in Computer Science
Information Visualization
Univ. of Maryland

9. PhD in Computer Science
Information Visualization
Univ. of Maryland
IBM
Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand
Data Scientist
Analytics, Experiment
Twitter
Microsoft

10. PhD in Computer Science
Information Visualization
Univ. of Maryland
IBM
Krist Wongsuphasawat / @kristw
Computer Engineer
Bangkok, Thailand
Engineering Manager
Data Experience
Airbnb
Microsoft
Twitter

11. #interactive visualizations
Open-source projects
Visual Analytics Tools
interactive.twitter.com
Apache Superset committer
labella.js (3000+ stars)
react-vega
Internal tools
Academic paperskristw.yellowpigz.com

12. DATA =ME+ VIS

13. Data, I’m ready!

14. Data, I’m ready!
Here I come!

15. WHAT TO EXPECT?

16. 1. EXPECT TO FIND THE REAL NEED

17. INPUT (DATA)
What clients think they have

18. INPUT (DATA)
What clients think they have What they usually have

19. YOU
What clients think you are

20. YOU
What clients think you are What they will get

21. OUTPUT (VIS)
What clients ask for

22. OUTPUT (VIS)
What clients ask for What they really need

23. COMMUNICATE

24. GOALS
Present data
Communicate information effectively
Analyze data
Exploratory data analysis
Tools to analyze data
Reusable tools for exploration
Enjoy
Combination of above

25. GOALS
Present data
Communicate information effectively
Analyze data
Exploratory data analysis
Tools to analyze data
Reusable tools for exploration
Enjoy
Combination of above
Who are the audience?
What do you want to tell?
What are the questions?
Who will use this?
What would they use this for?
Who are the audience?

26. I need this. Take this.

27. I need this. Here you are.
I need this. Take this.

28. & COMPROMISE

29. 2. EXPECT TO CLEAN DATA

30. 2. EXPECT TO CLEAN DATA A LOT

31. 70-80% of time cleaning data
“DATA JANITOR”

32. Collect + Clean + Transform
DATA WRANGLING

33. WHY DOES IT TAKE SO MUCH TIME?

34. 2.1 Many sources and data format

35. DATA SOURCES
Open data
Publicly available
Internal data
Private, owned by clients’ organization
Self-collected data
Manual, site scraping, etc.
Combine the above

36. DATA FORMAT
Standalone files
txt, csv, tsv, json, Google Docs, …, pdf*
Databases
doesn’t necessary mean they are organized
API
better quality with more overhead
Website
Big data*

37. NEED TO…
Change format
e.g. tsv => json
Combine data
Resolve multiple sources of truth

38. 2.2 Data transformation is needed.

39. EXAMPLES
Convert latitude/longitude into zip code
Change country code from 3-letter (USA) to 2-letter (US)
Correct time of day based on users’ timezone
etc.

40. 2.3 Data collection issues

41. EXAMPLES
Typos
Incorrect values
Incorrect timestamps
Missing data

42. 2.4 Definition of “clean” data

43. IS THIS CLEAN?
USER RESTAURANT RATING
========================
A MCDONALD’S 3
B MCDONALDS 3
C MCDONALD 4
D MCDONALDS 5
E IHOP 4
F SUBWAY 4

44. IS THIS CLEAN?
USER RESTAURANT RATING
========================
A MCDONALD’S 3
B MCDONALDS 3
C MCDONALD 4
D MCDONALDS 5
E IHOP 4
F SUBWAY 4
How many reviews are there?
Clean.
How many restaurants are there?
Not clean.
McDonald, McDonald’s, McDonalds

45. 2.5 Bigger data, bigger problems

46. HAVING ALL TWEETS
How people think I feel.

47. How people think I feel. How I really feel.
HAVING ALL TWEETS

48. Hadoop Cluster
GETTING BIG DATA
Data Storage

49. Scalding (slow)
GETTING BIG DATA
Hadoop Cluster
Data Storage
Tool

50. Scalding (slow)
GETTING BIG DATA
Hadoop Cluster
Data Storage
Tool
Your laptop Smaller dataset

51. Hadoop Cluster
Scalding (slow)
Data Storage
Tool
Final dataset
Tool node.js / python / excel (fast)
Your laptop
GETTING BIG DATA
Smaller dataset

52. CHALLENGES
Slow
Long processing time (hours)
Get relevant Tweets
hashtag: #oscars
keywords: “parasite” (movie name)
Too big
Need to aggregate & reduce size
Harder to spot problems

53. CHALLENGES
Slow
Long processing time (hours)
Get relevant Tweets
hashtag: #oscars
keywords: “parasite” (movie name)
Too big
Need to aggregate & reduce size
Harder to spot problems

54. RAMSAY & RAMSEY

55. 2.6 New issues can show up any time.

57. RECOMMENDATIONS
Always think that you will have to do it again
document the process, automation
Reusable scripts
break a gigantic do-it-all function into smaller ones
Reusable data
keep for future project

58. 3. PREPARE TO ITERATE

59. It was a great idea … until I actually tried it.

60. Celebrate your failures
#D3BrokeAndMadeArt

61. TIPS
Don’t give up.
If stuck, look for inspirations.
The vis that gives you insights may or may not be the best vis for sharing.
Exploration vs. Communication
Keep it as simple as possible
but not simpler.

62. “Necessity is the mother of invention.”
— English Proverb

63. “Necessity is the mother of invention.”
— English Proverb
DEADLINE

64. TIPS
Don’t give up.
If stuck, look for inspirations.
The vis that gives you insights may or may not be the best vis for sharing.
Exploration vs. Communication
Keep it as simple as possible
but not simpler.
Set milestones and deadline.

65. PROJECTS

66. STORYTELLING PROJECTS
timely
Deadline is strict. Also can be unexpected events.
wide audience
easy to explain and understand, multi-device support
one-off project
scope
analyze data to find stories and find best way to present them

67. HAPPY NEW YEAR AROUND THE WORLD
[ PROJECT ]

73. HAPPY NEW YEAR 2013
twitter.github.io/interactive/newyear2014/

74. BOBA SCIENCE
[ PROJECT ]

75. https://medium.com/s/story/boba-science-
how-can-i-drink-a-bubble-tea-to-ensure-that-
i-dont-finish-the-tea-before-the-
bobas-7fc5fd0e442d

76. GAME OF THRONES
[ PROJECT ]

77. from fans’ conversations
Reveal the talking points
of every episode of

78. Problem is coming.

79. Problem
Want to know what the audience
talk about a TV show
from Tweets

80. HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.

81. Brief Story

82. A King dies.
A lot of contenders wage a war
to reclaim the throne.

83. Minor characters with no claim to the throne
set their own plans in action to gain power
when all the major characters end up killing each other.

84. Brave/Honest/Honorable characters die.
Intelligent but shady characters
and characters who know nothing
continue to live.

85. While humans are busy killing each other,
ice zombies “White walkers” are invading from the North.
The only group who seems to care about this
is neutral group called the Night’s Watch.

86. HBO’s Game of Thrones
Based on a book series “A Song of Ice and Fire”
Medieval Fantasy. Knights, magic and dragons.
Many characters.
Anybody can die.
8 seasons
Multiple storylines in each episode

87. Problem
Want to know what the audience
talk about a TV show
from Tweets

88. Ideas
Common words
Too much noise

89. Ideas
Common words
Too much noise
Characters
How o!en each character were mentioned?

90. Prototyping
Pull sample data
from Twitter API
Entity recognition and counting
naive approach

91. List of names
Daenerys Targaryen,Khaleesi
Jon Snow
Sansa Stark
Tyrion Lannister
Arya Stark
Cersei Lannister
Khal Drogo
Gregor Clegane,Mountain
Margaery Tyrell
Joffrey Baratheon
Bran Stark
Theon Greyjoy
Jaime Lannister
Brienne
Eddard Stark,Ned Stark
Ramsay Bolton
Sandor Clegane,Hound
Ygritte
Stannis Baratheon
Petyr Baelish,Little Finger
Robb Stark
Bronn
Varys
Catelyn Stark
Oberyn Martell
Daario Naharis
Davos Seaworth
Jorah Mormont
Melisandre
Myrcella Baratheon
Tywin Lannister
Tommen Baratheon
Grey Worm
Tyene Sand
Rickon Stark
Missandei
Roose Bolton
Robert Baratheon
Jojen Reed
Jeor Mormont
Tormund Giantsbane
Lysa Arryn
Yara Greyjoy,Asha Greyjoy
Samwell Tarly,Sam
Hodor
Victarion Greyjoy
High Sparrow
Dragon
Winter
Dothraki

92. Sample Tweet

93. Sample Tweet

94. Sample data
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
Bran Stark 3000
… …
*These numbers are made up for presentation, not real data.

95. Where to go from here?

96. + episodes
The Guardian & Google Trends
http://www.theguardian.com/news/datablog/ng-interactive/2016/apr/22/game-of-thrones-the-most-googled-characters-episode-by-episode

97. + emotion

98. + connections

99. + connections

100. Gain insights from a single episode
emotion & connections

101. Sample data
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 10000
Jon Snow 5000
Daenerys 4000
… …
INDIVIDUALS CONNECTIONS
+ top emojis + top emojis
*These numbers are made up for presentation, not real data.

102. Graph
NODES LINKS
+ top emojis + top emojis
Character Count
Jon Snow+Sansa 1000
Tormund+Brienne 500
Bran Stark+Hodor 300
… …
Character Count
Hodor 1000
Jon Snow 500
Daenerys 400
… …
*These numbers are made up for presentation, not real data.

103. Network Visualization
Node-link diagram
Force-directed layout
http://blockbuilder.org/kristw/762b680690e4b2b2666dfec15838a384

104. Issue: Hairball

105. Issue: Occlusions

106. Tried: Fixed positions

107. + Collision Detection
http://blockbuilder.org/kristw/2850f65d6329c5fef6d5c9118f1de6e6

108. + Community Detection
https://github.com/upphiminn/jLouvain

109. + Collision Detection (with clusters)
https://bl.ocks.org/mbostock/7881887

110. Tormund + Brienne

111. Let’s get other episodes.

112. More data
Hadoop
Rewrite the scripts in Scalding
to get archived data

113. How much data do we need?
Whole week?
5 days?
2 days?
A day?
etc.

114. How much data do we need?

115. Transitions

116. Changing episode

117. Community transition
t=0 t=1

118. Smoother
t=0 t=1t=0.5 t=0.51

119. Colors
Default: D3 category10
Distinct but nothing about the context
Custom palette
Colors related to the groups/houses.
Black = Night’s Watch
Blue = North
Red = Daenerys
Gold = Lannister
…

120. The vis is not enough.

122. Legend

123. Navigation

124. Top 3

125. Adjust threshold

126. Recap

127. Filtered Recap
Tooltip

128. Demo
https://interactive.twitter.com/game-of-thrones

129. Mobile Support

130. Self & Peer
Does it solve the problem?

131. Google Analytics
Pageviews
Visitors
Actions
Referrals
Sites/Social

132. Feedback

133. Feedback

134. ANALYTICS TOOLS

135. VISUAL ANALYTICS TOOL PROJECTS
richer, more features
to support exploration of complex data
more technical audience
product managers, engineers, data scientists
accuracy
designed for dynamic input
long-term projects

136. PROJECT LIFECYCLE
Identify needs
Design and prototype
Make it work for sample dataset
Refine, generalize and productionize
Make it work for other cases
Document and release
Maintain and support
Keep it running, Feature requests & Bugs fix

137. VISUAL ANALYTICS FOR LOG EVENTS
[ PROJECT ]

138. USER ACTIVITY LOGS

139. UsersUseTwitter

140. UsersUse
Product Managers
Curious
Twitter

141. UsersUse
Curious
Engineers
Log data
in Hadoop
Write Twitter
Instrument
Product Managers

142. WHAT ARE BEING LOGGED?
tweet
Activities

143. WHAT ARE BEING LOGGED?
tweet from home timeline on twitter.com
tweet from search page on iPhone
Activities

144. WHAT ARE BEING LOGGED?
tweet from home timeline on twitter.com
tweet from search page on iPhone
sign up
log in
retweet
etc.
Activities

145. ORGANIZE?

146. LOG EVENT A.K.A. “CLIENT EVENT”
[Lee et al. 2012]

147. LOG EVENT A.K.A. “CLIENT EVENT”
client : page : section : component : element : action
web : home : timeline : tweet_box : button : tweet
1) User ID
2) Timestamp
3) Event name
4) Event detail
[Lee et al. 2012]

148. LOG DATA

149. UsersUse
Curious
Engineers
Log data
in Hadoop
Twitter
Instrument
Write
Product Managers
bigger than
Tweet data

150. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Ask
Twitter
Instrument
Write
Product Managers

151. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find
Ask
Twitter
Instrument
Write
Product Managers

152. LOG DATA

153. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Twitter
Instrument
Write
Product Managers

154. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean
Ask
Monitor
Twitter
Instrument
Write
Product Managers

155. UsersUse
Curious
Engineers
Log data
in Hadoop
Data Scientists
Find, Clean, Analyze
Ask
Monitor
Twitter
Instrument
Write
Product Managers

156. Log data
EngineersData Scientists
Usersin Hadoop
Find, Clean, Analyze
Use
Monitor
Ask
Curious
1 2
Twitter
Instrument
Write
Product Managers

157. client page section component element action
Event
50,000+ event types

158. client page section component element action
Event
50,000+ event types
one graph / event
x 50,000

159. DESIGN

160. CLIENT EVENT HIERARCHY
iphone home -
- - impression
tweet tweet click
iphone:home:-:-:-:impression
iphone:home:-:tweet:tweet:click

161. DETECT CHANGES
iphone home -
- - impression
tweet tweet click
iphone home -
- - impression
tweet tweet click
TODAY
7 DAYS AGO
compared to

162. CALCULATE CHANGES
+5% +5% +5%
+10% +10% +10%
-5% -5% -5%
DIFF

163. DISPLAY CHANGES
iphone home -
- - impression
tweet tweet click
Map of the Market [Wattenberg 1999], StemView [Guerra-Gomez et al. 2013]

164. DISPLAY CHANGES
home -
- - impression
tweet tweet click
iphone

165. Demo Demo Demo
Demo / Scribe Radar

166. Twitter for Banana

168. Details separate good and great work
4. RESERVE TIME FOR REFINEMENT

169. “The first 90% of the code
accounts for the first 90% of the development time.
The remaining 10% of the code
accounts for the other 90% of the development time.”
— Tom Cargill, Bell Labs

170. REFINE & POLISH
UX / UI + Mobile Support
Color
Animation / Transition
Metadata for SEO
Social media preview images
Performance
Loading time, Data file size
“The little of visualisation design” by Andy Kirk
http://www.visualisingdata.com/2016/03/little-visualisation-design/

171. Example

172. Issue: Convex hull
http://bl.ocks.org/mbostock/4341699

173. x & y only, no radius

174. Fix it

175. Fix it

176. Flatten the curve
https://www.fastcompany.com/90476143/the-story-behind-flatten-the-curve-the-defining-chart-of-the-coronavirus

177. THE ORIGIN
From a paper “Interim pre-
pandemic planning guidance:
community strategy for pandemic
influenza mitigation in the United
States: early, targeted, layered use
of nonpharmaceutical
interventions”
published in 2007 by the CDC
https://stacks.cdc.gov/view/cdc/11425

178. REVIVAL
Rosamund Pearce, a data journalist
at The Economist, rebuild it for a
piece about COVID-19.
Changed the labeling scheme to
assist colorblind readers.
https://www.economist.com/briefing/2020/02/29/covid-19-is-now-in-50-countries-and-things-will-get-worse

179. THE LINE
Drew Harris, an assistant
professor at the Thomas
Jefferson University, came across
the graphic in The Economist.
He recalled using it a decade
earlier as a pandemic
preparedness trainer.
So he added the dotted line
“healthcare system capacity”
https://www.nytimes.com/article/flatten-curve-coronavirus.html

180. THEN IT WENT VIRAL

184. or find ways to get some
5. PLAN FOR FEEDBACK

185. “Feedback is the breakfast of champion.”
— Ken Blanchard

186. FEEDBACK
During development
Feedback sessions with clients/potential users
After release
Logging
User study
Forum, User group
Office hours

187. 6. LOOK BACK FOR IMPROVEMENT

188. HOW TO BE BETTER?
Retrospective
What could have been better?
Wishlist
Expand skillset
Learning opportunities
Get help
Grow the team
Improve tooling
Solve a problem once and for all
Automate repetitive tasks

189. REUSABLE WORK

190. LABELLA.JS
[ PROJECT ]

192. GRID MAP
[ PROJECT ]

196. COVID-19 Situation in Thailand
by province

197. VX = REACT + D3
[ PROJECT ]

199. SUMMARY

200. 6 STEPS
1.
2.
3.
4.
5.
6.
Krist Wongsuphasawat / @kristw
kristw.yellowpigz.com
Expect to find the real need
Expect to clean data a lot
Prepare to iterate
Reserve time for refinement
Plan for feedback
Look back for improvement

201. My former and current colleagues at Twitter and Airbnb
for their collaboration and support in these projects;
and my wife for taking care of our two kids
while I make these slides.
ACKNOWLEDGEMENT

202. THANK YOU

203. QUESTIONS?