video available here: https://blueprintdigital.com/ia-summit-2017/jenny-benevento-giovanni-fernandez-kincade-jill-fruchter/
This was a talk given at IA Summit 2017 in Vancouver, BC by Jenny Benevento, Gio Fernandez-Kincade, and Jill Fruchter.
Etsy is a marketplace where people around the world connect, both online and offline, to make, sell and buy unique goods. Etsy is also a tech company that invests in the craft of coding and data-driven product development as a strategic priority. Etsy has employed AI and machine learning to tackle personalization, recommendations, image understanding, item similarity, search relevance, spelling correction, and many other tasks. We’ll talk through several examples of how Etsy leverages data, where it’s excelled, and where this hammer hasn’t quite hit the nail on the head.
We will be asking ourselves hard questions, recognizing the limitations of decisions driven purely by big data:
- Who are we satisfying? Our customers or our mathematical models?
- Are those models even an accurate reflection of the outcomes we want?
- In a dual marketplace, where complex changes depend on interactions between both sides of the market, can one metric or measure of success tell the full story?
- How do we consider the impact our models are having on our users?
- Are we even addressing real human needs and motivations in the first place?
- How do we inform and enrich AI with expert created & applied taxonomy & metadata?
3. Etsy is a global marketplace where people around the world connect,
both online and offline, to make, sell and buy unique goods.
3
4. By the Numbers
1.7M
active sellers
AS OF DECEMBER 31, 2016
28M
active buyers
AS OF DECEMBER 31, 2016
$2.84B
annual GMS
IN 2016
45+M
items for sale
AS OF DECEMBER 31, 2016
Photo by Kirsty-Lyn Jameson
5.
6. SELLER
Pursues craft,
grows business
ETSY
Invests in the
platform and delivers
a global base of
buyers
ETSY
Facilitates the
transaction
BUYER
Finds unique
goods that are
hard to find
elsewhere
ETSY EMPOWERMENT LOOP
ETSY EMPOWERMENT LOOP
6
7. SELLER
Pursues craft,
grows business
ETSY
Invests in the
platform and delivers
a global base of
buyers
ETSY
Facilitates the
transaction
BUYER
Finds unique
goods that are
hard to find
elsewhere
ETSY EMPOWERMENT LOOP
ETSY EMPOWERMENT LOOP
7
SEARCH
17. People shop for ideas on Etsy
17
GIFTS LIFE EVENTS
SEASONAL
CELEBRATIONS
STYLE/EMOTION
POP CULTURE/
TRENDS
NOT “known item” searches
18. People shop for ideas on Etsy
18
GIFTS LIFE EVENTS STYLE/EMOTION
NOT “known item” searches
Often, using broad, thematic search terms
SEASONAL
CELEBRATIONS
POP CULTURE/
TRENDS
20. Existing “browse”pages
were low performers, i.e.
not converting.
PROBLEM HOW MIGHT WE…
Make finding on Etsy
easier with a combination
of search & browse
improvements?
20
21. Customers are doing
broad searches and
getting “overwhelmed”
with search results.
PROBLEM HOW MIGHT WE…
Turn broad search
queries into satisfying
experiences of finding
and discovery?
21
22. UX research as the new kid in town
FEB
2014
MARCH
2014 MARCH
2015
JUNE
2015
JULY
2014
AUGUST
2014
● Etsy was founded in 2005
● No UX Research until 2014 (Thick data)
● Since 2010, data-driven design and evaluation (Big data)
RESEARCH
BRAND
DESIGN
PRODUCT
DESIGN
23. Etsy almost solely
relied on AI and
experimentation to frame
and solve problems.
PROBLEM HOW MIGHT WE…
Use this new research
capacity to help frame the
problem and build
a better product?
23
26. GOAL:
Observe “jewelry” search
behavior on- and off- Etsy
to introduce product teams
to shoppers’ expectations
and needs
TEST STIMULI:
- The Internet
- Self-guided searches
Contextual observation (2x)
27.
28. Goals
1 2 3
UX Insights
Searching on Etsy is conceptually different
29. Goals
1 2 3
Highly visual
UX Insights
Searching on Etsy is conceptually different
“I’m not on track”
“I’m looking for something
to catch my eye”
30. Goals
1 2 3
Highly visual Use search to browse
UX Insights
Searching on Etsy is conceptually different
“I’m not on track”
“I’m looking for something
to catch my eye”
“6PM is like running into
the drugstore. On Etsy, I’m
always finding surprises.”
“It’s like watching Reality TV.
I like the process.”
31. Goals
1 2 3
Highly visual Use search to browse
UX Insights
Searching on Etsy is conceptually different
“I’m not on track”
“I’m looking for something
to catch my eye”
“6PM is like running into
the drugstore. On Etsy, I’m
always finding surprises.”
“It’s like watching Reality TV.
I like the process.”
FOMO
“Even if it were $10,000.
I want to know about it.”
“Patience on Etsy
is rewarded.”
34. Goals
1 2 3
UX Insights
Until items feel more similar than different, the more categorization the better
35. Goals
1 2 3
Less FOMO
UX Insights
"It lets me see how it's
organized and go on.
It guides you."
"It's nice the categories are right
here. It prompts me to narrow
down an extremely broad search."
Until items feel more similar than different, the more categorization the better
36. Goals
1 2 3
Less FOMO
UX Insights
"It lets me see how it's
organized and go on.
It guides you."
"It's nice the categories are right
here. It prompts me to narrow
down an extremely broad search."
Until items feel more similar than different, the more categorization the better
More intuitive
“I’m making more progress
with minimal effort.”
37. Goals
1 2 3
Less FOMO
UX Insights
"It lets me see how it's
organized and go on.
It guides you."
"It's nice the categories are right
here. It prompts me to narrow
down an extremely broad search."
Until items feel more similar than different, the more categorization the better
More intuitive
“I’m making more progress
with minimal effort.”
More “Etsy”
“Expands for me what is
on Etsy because I haven't
thought of that being
there.”
41. Berrypicking
“the query is satisfied...by
a series of selections of individual
references and bits of
information at each stage of the
ever-modifying search...retrieval
of this sort is here called
berrypicking.”
Source: Marcia Bates (1989). "The Design of Browsing and Berrypicking
Techniques for the Online Search Interface."