To enable at scale access to critique we present MATT, a chatbot that micro-guides experts to critique in short bursts of time with mediated communication to address experts' time and privacy concerns.

1. Enabling Expert Critique with
Chatbots and Micro Guidance
Carlos Toxtli, Joel Chan, Walter Lasecki, Saiph Savage

2. Problem statement
● Critique is important to improve creative work and help learners grow.
● Experts traditionally provided critique within physical studios. But, many never
received critique as they were not co-located with experts.
● Online alternatives, such as forums, rarely lead people to receive critique
● Scaling online critique is difficult as:
○ People don’t have the knowledge or expertise to provide critique.
○ The experts with the knowledge to critique have limited time and privacy concerns.
● Dannels and Martin 2008
● Luther and Bruckman 2008
● Xu and Bailey 2012

3. Feedback typology [Dannels and Martin 2008]
Feedback type Description Examples
Reactive Emotional or visceral feedback
that does not provide information
on how to improve the work.
“That’s wonderful!”, “Great work!”
or “Horrible!”
Direction Tries to bring the design more in
line with her own expectations of
what the solution should be. The
feedback provides direction but
no reasoning behind it.
“I would have...” or “I wish...”
Critique This type of feedback focuses on
identifying decisions made in the
creative work; relating that
decision to an objective or best
practice.
“According to … the best way to
… is ...”

4. MATT (Micro-Advicing Through Tutorials)
To enable at scale access to critique we present MATT, a chatbot that micro-
guides experts to critique in short bursts of time with mediated communication to
address experts' time and privacy concerns.

6. 0- User starts conversation 2- Experts receive the learners work and give feedback
1- MATT deliver tutorials and asks for work 3- Learners receive the expert’s feedback

7. MATT components
MATT consists of two main components:
1. Learner Helper module: Collects learners’ creative work, distributes the work
to experts, and then shares experts’ feedback to learners.
2. Expert Micro-Guidance module: Orchestrates experts to volunteer quality
micro-feedback – which resembles online critique – to help learners at scale.

8. Learner Helper module
Main functions:
● Allow learners to easily submit their creative work
● Find relevant experts who can critique their work
● Present back to the learner the feedback from experts.

9. Expert Micro-Guidance module features
● Critique in Short Bursts of Time. MATT guides experts to provide critique to
creative work by leveraging task decomposition from crowdsourcing.
● Critique Anywhere. MATT communicates via Facebook Messenger with
experts. This design facilitates portability and on-the-go experiences.

10. ● Privacy. Our design builds upon privacy research that showcases that with
anonymity higher quality feedback is produced. [Ruiling Lu and Linda Bol.
2007]
● Conversational. MATT guides experts to produce critique within a
conversational setting through chatbots. The conversational aspect of MATT
might also help experts to not feel that MATT’s guidance is too dictatorial.
Expert Micro-Guidance module features

11. ● Research Question: Do chatbots micro-guiding experts enable a better
approximation of the gold standard of studio design feedback?
● We conducted a field experiment to compare the feedback experts generated
on MATT to alternative interfaces.
Evaluation

12. Online forum with guidanceChatbot lacking micro-guidanceChatbot with micro-guidance, MATT

13. ● We recruited 548 learners, and 76 experts primarily using social media
(Facebook, Linkedin). All experts volunteered their time.
● Learners submitted 153 creative work pieces to the MATT condition, 213 to
the online forum, and 128 to the chatbot without guidance conditions.
Participants

14. Learners submitted real world creative work pieces from one of these types:
website design; poster design for an NGO; t-shirt design for an organization, and
entrepreneur product design.
Submitted work

15. Results
Three coders classified each of the 548 messages into the category that
represented the message the most (either critique, reactive, or direction).

16. Results - Quantitative analysis
Feedback was significantly more likely to be classified as critique when it came
from MATT, compared to feedback from the online forum condition (p < .01) or
the Bot No Guidance condition (p < .01).
MATT facilitated more critique production in experts.

17. Results - Qualitative analysis
● Experts stated they enjoyed moderately the chatbot interfaces (mean=4.85
for MATT and for the chatbot without guidance).
● Forum interface was also enjoyed, but slightly less (mean=4.77).
● Experts considered all interfaces to be moderately easy to use (mean=4.8)
● Experts felt that MATT addressed their privacy concerns (median=5).

18. Results - Opinions
● Some experts felt that MATT helped them to produce meaningful
feedback by directing the communication into what mattered.
● Some experts expressed that the automated aspect of MATT made its
guidance not feel imposing.
● MATT’s automation also seemed to help experts accept its guidance, as
they felt that machines were made to help humans in their daily work.

19. Conclusion
● We introduced MATT a chatbot that guides experts to critique the creative
work of learners at scale.
● MATT embodies the vision that chatbots facilitate orchestrating experts to
critique while addressing experts’ privacy concerns and without creating an
imposing environment on specialists.
● A field deployment provided evidence that MATT could guide experts to
critique the creative work of hundreds of learners.

20. Thanks
@ctoxtli
carlos.toxtli@mail.wvu.edu
http://www.carlostoxtli.com