Human-in-the-loop machine learning is an incredibly valuable design pattern used in real world machine learning deployments across many types of applications. Taking the best of what humans can do and combining that with the best computers is practical and powerful. Not only that, but it mimics a strategy called active learning, forcing the training data collection process to become very efficient so the algorithm gets better and better.
2. The Effect of Better Algorithms
0%
5%
10%
15%
20%
25%
Naïve Bayes Maximum
Entropy
SVM
Classifier Error Rate
Active Semi-Supervised Learning for Improving
Word Alignment
(Vamshi ACL ’10)
Real World Data
2
3. The Effect of Better Features
0%
5%
10%
15%
20%
25%
30%
Unigrams Bigrams Unigrams+Bigrams
Classifier Error Rate
3
4. The Effect of More Data
Active Semi-Supervised Learning for
Improving Word Alignment
(Vamshi ACL ’10)
Real World Data
0%
2%
4%
6%
8%
10%
12%
14%
N 2N 4N
Classifier Error Rate
4
5. The Effect of Cleaner Data
0%
2%
4%
6%
8%
10%
12%
14%
90% Accurate Data 95% Accurate Data 100% Accurate Data
Classifier Error Rate
5
6. Where Do Data Scientists Spend Their Time?
6
Source: CrowdFlower Data
Science Report 2015
Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence
- Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t
- Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough
I think we can and should apply this to every business process
We start with a machine learning classifier.
Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence
- Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t
- Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough
A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time.
We didn’t invent any of this, lot’s of people are talking about this and thinking about this, including many people in the room. But looking at the industry we see a lot more people talking about it than actually doing it.
We are going to make this setup so easy that you will have no excuse for not doing it.
I think we can and should apply this to every business process
We start with a machine learning classifier.
Unlike humans, artificial intelligence has no ego, so it can make an unbiased estimate of its confidence
- Where it’s confident we use its answer, because hardware CPUs get cheaper+faster every year and human CPUs don’t
- Where it’s not confident we use a human because in real business applications 80% accuracy isn’t good enough
A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time.
We didn’t invent any of this, lot’s of people are talking about this and thinking about this, including many people in the room. But looking at the industry we see a lot more people talking about it than actually doing it.
We are going to make this setup so easy that you will have no excuse for not doing it.
A huge side benefit is that the human labels can be reused used to improve the machine learning classifier over time.
handed control to the driver 272 times and a test driver felt compelled to intervene 69 times
In the field of chess computers passed humans a long time ago.
But if you really want to make a great chess playing algorithm you would still use a human and computer together.
There is a subculture of folks who still play “Advanced Chess” and this is actually where the highest quality chess games take place.
- Still situations where humans are better