SlideShare ist ein Scribd-Unternehmen logo
1 von 33
Downloaden Sie, um offline zu lesen
Talent Search and Recommendation
Systems at LinkedIn
Practical Challenges and Lessons Learned
Qi Guo, Sahin Cem Geyik, Bo Hu, Cagri Ozcaglar,
Ketan Thakkar, Xianren Wu, Krishnaram Kenthapadi
AI @ LinkedIn
+SIGIR 2018
+
The Team
Qi Guo Sahin Cem Geyik Bo Hu Cagri Ozcaglar
Ketan Thakkar Xianren Wu Krishnaram Kenthapadi
Contents
• Introduction
• Ranking Models for Talent Search
• Personalization
• Talent Search Architecture
• Summary
Introduction
LinkedIn Talent Solution:
~65% of LinkedIn’s Annual Revenue
A H I R I N G E C O S Y S T E M
LinkedIn Recruiter
MAJOR PRODUCT
A Talent Search and
Recommendation System
Recruiter Search
• Criteria-Based Search
• A recruiter has specific requisitions to fill
• Candidate Recommendation System
• A recruiter may want many qualified candidates, goes through pages
• Considers Both Sides of the Talent Marketplace
• Talents are limited resources
# of InMail Accepts
OPTIMIZATION OBJECTIVE:
3. Accept
2. Send
InMail
Recruiter
Candidate
1. Search
Ranking Models for Talent Search
Number of InMail Accepts Per Seat: 30% YoY
O V E R A L L I M P R O V E M E N T
Go Non-Linear with Tree Model
• Before: Linear Model optimized for NDCG with Coordinate Ascent
• After: XGBoost Tree Model
• Captures feature interactions
• XGBoost: gradient boosting tree models for richer model complexity
• Online Results:
METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT
Lift +7.5% +7.4% +5.1%
P-Value 2.1e-4 4.8e-4 0.01
Search for “Dentist”, a Software Engineer ranks high
P R O B L E M O B S E R V E D
Search for “Dentist”, a Software Engineer ranks high
P R O B L E M O B S E R V E D
• Focused too much on promoting active job-seeking candidate
• We want our ranking to be more context-aware
f( , , ) => Accept?
Reject?
Recruiter
Context
Query
Context
Context-Aware Ranking – Pairwise Training
f( , , )1
- f( , , )2Recruiter
Context
Query
Context
{
Shared Context
=>
• Pair up two candidates from the same search request:
Accept?
1
Accept?
2?>
Context-Aware Ranking
• Before: Pointwise XGBoost
• After: Pairwise XGBoost with Context-Aware Features
• Recruiter Context: Personalization features
• Query Context: Query-Candidate matching features
• Online Results:
METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT
Lift +18.2% +13.7% +8%
P-Value 1e-16 1.1e-11 9.6e-4
Search for “Machine Learning Engineer”,
desirable to include some Data Scientists
P R O B L E M O B S E R V E D
Representation Learning
• Fuzzy semantic match on title ids, skill ids, company ids etc.
• Unsupervised Graph Embedding
• Co-Occurrence Graph based on profile data
Representation Learning
• Before: XGBoost
• After: XGBoost with Title Similarity Feature
• Based on unsupervised graph embedding
• Online Results:
METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT
Lift +2% +1.8% +3%
P-Value 0.2 0.25 0.11
Deep Learning?
• Differentiable Programming with TensorFlow
• Flexible for model engineering
• Offline result does not justify the effort yet.
• Offline Results (Pairwise NN v.s. Pointwise XGBoost):
METRIC PRECISION@1 PRECISION@5 PRECISION@25
Lift +5.3% +2.8% +1.7%
Personalization for
Talent Search
Entity-Level Personalization with GLMix
• GLMix: Generalized Linear Mixed Models
• GLMix: global model + per-entity models
• We added per-recruiter model and per-contract/company model
Entity-Level Personalization with GLMix
• Model Ensemble
• Nonlinearity via tree interaction features
• Each leaf node is a feature
• Offline Results (GLMix vs. Pairwise XGBoost):
METRIC PRECISION@1 PRECISION@5 PRECISION@25
Lift +8.5% +4.7% +2.0%
Using Recruiter Search requires a lot of skills.
P R O B L E M O B S E R V E D
A Stream of
Recommended Candidates
Recommended Matches
SIMPLIFIED EXPERIENCE
In-Session Personalization
• Step 1: Segment the Space
• Query Intent Clustering
• Step 2: Evaluate each segment
• Multi-Armed Bandits
• Step 3: Modify each segment
• Term Weight Updates
In-Session Personalization: Results
Talent Search Architecture
Search and Retrieval Architecture
• LinkedIn’s Galene is built on top of Lucene.
• Three main components:
• Search index on searcher
• The fanout queries through broker, and
• Live updates to the index using live-updater.
• Query language is similar to Lucene with OR, AND, NOT.
• The search index contains two types of fields:
• Inverted Fields
• Forward Fields
Search and Retrieval Architecture
• Static Rank
• An auxiliary rank for members to help with retrieving at scale
• Based on member profile and activity
• Early termination
• Index partitioned into N-shards, each retrieves and scores candidates
• Not all members in a shard can be retrieved, so query is early terminated on the basis of
static rank.
• Galene Facet Counting:
• Galene supports facet counting (such as region, titles, etc) for any given query.
• Uses statistical counting approximation based on sample in each shard
Layered Ranking Architecture
• L1: Better to scoop into the talent pool and score/rank more candidates.
• L2: Refines the short-listed talent to apply more dynamic features using external cache.
Summary
Summary
• Talent Search
• Criteria Search, Recommendation System, Marketplace
• Talent Search Ranking
• Context-Aware Pairwise Training
• Representation Learning & Deep Learning
• GLMix Personalization
• In-Session Personalization
+
Thank You
Qi Guo Sahin Cem Geyik Bo Hu Cagri Ozcaglar
Ketan Thakkar Xianren Wu Krishnaram Kenthapadi

Weitere ähnliche Inhalte

Was ist angesagt?

Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...
Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...
Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...Lucidworks
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixJaya Kawale
 
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...Balázs Hidasi
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systemsinovex GmbH
 
Session-based recommendations with recurrent neural networks
Session-based recommendations with recurrent neural networksSession-based recommendations with recurrent neural networks
Session-based recommendations with recurrent neural networksZimin Park
 
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Universitat Politècnica de Catalunya
 
Tutorial on sequence aware recommender systems - UMAP 2018
Tutorial on sequence aware recommender systems - UMAP 2018Tutorial on sequence aware recommender systems - UMAP 2018
Tutorial on sequence aware recommender systems - UMAP 2018Paolo Cremonesi
 
Recommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking SystemRecommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking Systemivaderivader
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Faisal Siddiqi
 
Anatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur DatarAnatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur DatarNaresh Jain
 
Simple Matrix Factorization for Recommendation in Mahout
Simple Matrix Factorization for Recommendation in MahoutSimple Matrix Factorization for Recommendation in Mahout
Simple Matrix Factorization for Recommendation in MahoutData Science London
 
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems -  ACM RecSys 2013 tutorialLearning to Rank for Recommender Systems -  ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorialAlexandros Karatzoglou
 
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018Taehoon Kim
 
A brief overview of Reinforcement Learning applied to games
A brief overview of Reinforcement Learning applied to gamesA brief overview of Reinforcement Learning applied to games
A brief overview of Reinforcement Learning applied to gamesThomas da Silva Paula
 
An introduction to reinforcement learning
An introduction to reinforcement learningAn introduction to reinforcement learning
An introduction to reinforcement learningSubrat Panda, PhD
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized HomepageJustin Basilico
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareJustin Basilico
 
Recommendation System Explained
Recommendation System ExplainedRecommendation System Explained
Recommendation System ExplainedCrossing Minds
 

Was ist angesagt? (20)

Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...
Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...
Learning to Rank: From Theory to Production - Malvina Josephidou & Diego Cecc...
 
A Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at NetflixA Multi-Armed Bandit Framework For Recommendations at Netflix
A Multi-Armed Bandit Framework For Recommendations at Netflix
 
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...
GRU4Rec v2 - Recurrent Neural Networks with Top-k Gains for Session-based Rec...
 
Deep Learning for Recommender Systems
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
 
Session-based recommendations with recurrent neural networks
Session-based recommendations with recurrent neural networksSession-based recommendations with recurrent neural networks
Session-based recommendations with recurrent neural networks
 
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
Image-to-Image Translation with Conditional Adversarial Nets (UPC Reading Group)
 
Tutorial on sequence aware recommender systems - UMAP 2018
Tutorial on sequence aware recommender systems - UMAP 2018Tutorial on sequence aware recommender systems - UMAP 2018
Tutorial on sequence aware recommender systems - UMAP 2018
 
Recommender Systems
Recommender SystemsRecommender Systems
Recommender Systems
 
Recommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking SystemRecommending What Video to Watch Next: A Multitask Ranking System
Recommending What Video to Watch Next: A Multitask Ranking System
 
Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019Netflix talk at ML Platform meetup Sep 2019
Netflix talk at ML Platform meetup Sep 2019
 
Anatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur DatarAnatomy of an eCommerce Search Engine by Mayur Datar
Anatomy of an eCommerce Search Engine by Mayur Datar
 
Simple Matrix Factorization for Recommendation in Mahout
Simple Matrix Factorization for Recommendation in MahoutSimple Matrix Factorization for Recommendation in Mahout
Simple Matrix Factorization for Recommendation in Mahout
 
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems -  ACM RecSys 2013 tutorialLearning to Rank for Recommender Systems -  ACM RecSys 2013 tutorial
Learning to Rank for Recommender Systems - ACM RecSys 2013 tutorial
 
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018
머신러닝 해외 취업 준비: 닳고 닳은 이력서와 고통스러웠던 면접을 돌아보며 SNU 2018
 
Learn to Rank search results
Learn to Rank search resultsLearn to Rank search results
Learn to Rank search results
 
A brief overview of Reinforcement Learning applied to games
A brief overview of Reinforcement Learning applied to gamesA brief overview of Reinforcement Learning applied to games
A brief overview of Reinforcement Learning applied to games
 
An introduction to reinforcement learning
An introduction to reinforcement learningAn introduction to reinforcement learning
An introduction to reinforcement learning
 
Learning a Personalized Homepage
Learning a Personalized HomepageLearning a Personalized Homepage
Learning a Personalized Homepage
 
Recommendations for Building Machine Learning Software
Recommendations for Building Machine Learning SoftwareRecommendations for Building Machine Learning Software
Recommendations for Building Machine Learning Software
 
Recommendation System Explained
Recommendation System ExplainedRecommendation System Explained
Recommendation System Explained
 

Ähnlich wie Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learned

Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016
Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016
Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016Glenn Gutmacher
 
Building Search and Personalization at Nordstrom Rack | Hautelook
Building Search and Personalization at Nordstrom Rack | HautelookBuilding Search and Personalization at Nordstrom Rack | Hautelook
Building Search and Personalization at Nordstrom Rack | HautelookLucidworks
 
An analysis of employee performance evaluation and employee motivation
An analysis of employee performance evaluation and employee motivationAn analysis of employee performance evaluation and employee motivation
An analysis of employee performance evaluation and employee motivationchrisnava
 
How to Leverage Marketing Analytics to Source Better Talent
How to Leverage Marketing Analytics to Source Better TalentHow to Leverage Marketing Analytics to Source Better Talent
How to Leverage Marketing Analytics to Source Better TalentData Con LA
 
SEO Training Course on Wecommerce.pk web
SEO Training Course on Wecommerce.pk webSEO Training Course on Wecommerce.pk web
SEO Training Course on Wecommerce.pk webUsamaRiazTutorial
 
About Youth4work - Integrated Talent Solutions
About Youth4work - Integrated Talent SolutionsAbout Youth4work - Integrated Talent Solutions
About Youth4work - Integrated Talent SolutionsYouth4work.com
 
Times Jobs _Times Internet Limited
Times Jobs _Times Internet Limited Times Jobs _Times Internet Limited
Times Jobs _Times Internet Limited Souvik Bhaumik
 
Info session - sourcing & training certification
Info session - sourcing & training certification Info session - sourcing & training certification
Info session - sourcing & training certification Irina Shamaeva
 
Spcua 2013 Alexey Kozhemiakin Enterprise Search
Spcua 2013 Alexey Kozhemiakin Enterprise SearchSpcua 2013 Alexey Kozhemiakin Enterprise Search
Spcua 2013 Alexey Kozhemiakin Enterprise SearchAlex Kozhemiakin
 
Sullivan and Cogliano Capabilties 1.22.16
Sullivan and Cogliano Capabilties 1.22.16Sullivan and Cogliano Capabilties 1.22.16
Sullivan and Cogliano Capabilties 1.22.16Herb Cogliano
 
SmartRecruiters Webinar - Sourcing Tab
SmartRecruiters Webinar - Sourcing TabSmartRecruiters Webinar - Sourcing Tab
SmartRecruiters Webinar - Sourcing TabPaul Andre de Vera
 
Personalizing Search at LinkedIn
Personalizing Search at LinkedInPersonalizing Search at LinkedIn
Personalizing Search at LinkedInViet Ha-Thuc
 
Pull_Request_PAW_Shared_Rohit.pptx
Pull_Request_PAW_Shared_Rohit.pptxPull_Request_PAW_Shared_Rohit.pptx
Pull_Request_PAW_Shared_Rohit.pptxrohitagarwal24
 

Ähnlich wie Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learned (20)

Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016
Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016
Gutmacher In-House Sourcing Model Offshore and Onshore Nov. 2016
 
Building Search and Personalization at Nordstrom Rack | Hautelook
Building Search and Personalization at Nordstrom Rack | HautelookBuilding Search and Personalization at Nordstrom Rack | Hautelook
Building Search and Personalization at Nordstrom Rack | Hautelook
 
Timberhorn ppt
Timberhorn pptTimberhorn ppt
Timberhorn ppt
 
Hcm file-wp
Hcm file-wpHcm file-wp
Hcm file-wp
 
How Google works
How Google worksHow Google works
How Google works
 
An analysis of employee performance evaluation and employee motivation
An analysis of employee performance evaluation and employee motivationAn analysis of employee performance evaluation and employee motivation
An analysis of employee performance evaluation and employee motivation
 
Accendo's TalentPulse Package
Accendo's TalentPulse PackageAccendo's TalentPulse Package
Accendo's TalentPulse Package
 
How to Leverage Marketing Analytics to Source Better Talent
How to Leverage Marketing Analytics to Source Better TalentHow to Leverage Marketing Analytics to Source Better Talent
How to Leverage Marketing Analytics to Source Better Talent
 
SEO Training Course on Wecommerce.pk web
SEO Training Course on Wecommerce.pk webSEO Training Course on Wecommerce.pk web
SEO Training Course on Wecommerce.pk web
 
DTEK Talent Management System
DTEK Talent Management SystemDTEK Talent Management System
DTEK Talent Management System
 
About Youth4work - Integrated Talent Solutions
About Youth4work - Integrated Talent SolutionsAbout Youth4work - Integrated Talent Solutions
About Youth4work - Integrated Talent Solutions
 
REsume Screening
REsume ScreeningREsume Screening
REsume Screening
 
Times Jobs _Times Internet Limited
Times Jobs _Times Internet Limited Times Jobs _Times Internet Limited
Times Jobs _Times Internet Limited
 
Info session - sourcing & training certification
Info session - sourcing & training certification Info session - sourcing & training certification
Info session - sourcing & training certification
 
Spcua 2013 Alexey Kozhemiakin Enterprise Search
Spcua 2013 Alexey Kozhemiakin Enterprise SearchSpcua 2013 Alexey Kozhemiakin Enterprise Search
Spcua 2013 Alexey Kozhemiakin Enterprise Search
 
Twinkle
TwinkleTwinkle
Twinkle
 
Sullivan and Cogliano Capabilties 1.22.16
Sullivan and Cogliano Capabilties 1.22.16Sullivan and Cogliano Capabilties 1.22.16
Sullivan and Cogliano Capabilties 1.22.16
 
SmartRecruiters Webinar - Sourcing Tab
SmartRecruiters Webinar - Sourcing TabSmartRecruiters Webinar - Sourcing Tab
SmartRecruiters Webinar - Sourcing Tab
 
Personalizing Search at LinkedIn
Personalizing Search at LinkedInPersonalizing Search at LinkedIn
Personalizing Search at LinkedIn
 
Pull_Request_PAW_Shared_Rohit.pptx
Pull_Request_PAW_Shared_Rohit.pptxPull_Request_PAW_Shared_Rohit.pptx
Pull_Request_PAW_Shared_Rohit.pptx
 

Kürzlich hochgeladen

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 

Kürzlich hochgeladen (20)

How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 

Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learned

  • 1. Talent Search and Recommendation Systems at LinkedIn Practical Challenges and Lessons Learned Qi Guo, Sahin Cem Geyik, Bo Hu, Cagri Ozcaglar, Ketan Thakkar, Xianren Wu, Krishnaram Kenthapadi AI @ LinkedIn +SIGIR 2018
  • 2. + The Team Qi Guo Sahin Cem Geyik Bo Hu Cagri Ozcaglar Ketan Thakkar Xianren Wu Krishnaram Kenthapadi
  • 3. Contents • Introduction • Ranking Models for Talent Search • Personalization • Talent Search Architecture • Summary
  • 5. LinkedIn Talent Solution: ~65% of LinkedIn’s Annual Revenue A H I R I N G E C O S Y S T E M
  • 6. LinkedIn Recruiter MAJOR PRODUCT A Talent Search and Recommendation System
  • 7. Recruiter Search • Criteria-Based Search • A recruiter has specific requisitions to fill • Candidate Recommendation System • A recruiter may want many qualified candidates, goes through pages • Considers Both Sides of the Talent Marketplace • Talents are limited resources
  • 8. # of InMail Accepts OPTIMIZATION OBJECTIVE: 3. Accept 2. Send InMail Recruiter Candidate 1. Search
  • 9. Ranking Models for Talent Search
  • 10. Number of InMail Accepts Per Seat: 30% YoY O V E R A L L I M P R O V E M E N T
  • 11. Go Non-Linear with Tree Model • Before: Linear Model optimized for NDCG with Coordinate Ascent • After: XGBoost Tree Model • Captures feature interactions • XGBoost: gradient boosting tree models for richer model complexity • Online Results: METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT Lift +7.5% +7.4% +5.1% P-Value 2.1e-4 4.8e-4 0.01
  • 12. Search for “Dentist”, a Software Engineer ranks high P R O B L E M O B S E R V E D
  • 13. Search for “Dentist”, a Software Engineer ranks high P R O B L E M O B S E R V E D • Focused too much on promoting active job-seeking candidate • We want our ranking to be more context-aware f( , , ) => Accept? Reject? Recruiter Context Query Context
  • 14. Context-Aware Ranking – Pairwise Training f( , , )1 - f( , , )2Recruiter Context Query Context { Shared Context => • Pair up two candidates from the same search request: Accept? 1 Accept? 2?>
  • 15. Context-Aware Ranking • Before: Pointwise XGBoost • After: Pairwise XGBoost with Context-Aware Features • Recruiter Context: Personalization features • Query Context: Query-Candidate matching features • Online Results: METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT Lift +18.2% +13.7% +8% P-Value 1e-16 1.1e-11 9.6e-4
  • 16. Search for “Machine Learning Engineer”, desirable to include some Data Scientists P R O B L E M O B S E R V E D
  • 17. Representation Learning • Fuzzy semantic match on title ids, skill ids, company ids etc. • Unsupervised Graph Embedding • Co-Occurrence Graph based on profile data
  • 18. Representation Learning • Before: XGBoost • After: XGBoost with Title Similarity Feature • Based on unsupervised graph embedding • Online Results: METRIC PRECISION@5 PRECISION@25 OVERALL ACCEPT Lift +2% +1.8% +3% P-Value 0.2 0.25 0.11
  • 19. Deep Learning? • Differentiable Programming with TensorFlow • Flexible for model engineering • Offline result does not justify the effort yet. • Offline Results (Pairwise NN v.s. Pointwise XGBoost): METRIC PRECISION@1 PRECISION@5 PRECISION@25 Lift +5.3% +2.8% +1.7%
  • 21. Entity-Level Personalization with GLMix • GLMix: Generalized Linear Mixed Models • GLMix: global model + per-entity models • We added per-recruiter model and per-contract/company model
  • 22. Entity-Level Personalization with GLMix • Model Ensemble • Nonlinearity via tree interaction features • Each leaf node is a feature • Offline Results (GLMix vs. Pairwise XGBoost): METRIC PRECISION@1 PRECISION@5 PRECISION@25 Lift +8.5% +4.7% +2.0%
  • 23. Using Recruiter Search requires a lot of skills. P R O B L E M O B S E R V E D
  • 24. A Stream of Recommended Candidates Recommended Matches SIMPLIFIED EXPERIENCE
  • 25. In-Session Personalization • Step 1: Segment the Space • Query Intent Clustering • Step 2: Evaluate each segment • Multi-Armed Bandits • Step 3: Modify each segment • Term Weight Updates
  • 28. Search and Retrieval Architecture • LinkedIn’s Galene is built on top of Lucene. • Three main components: • Search index on searcher • The fanout queries through broker, and • Live updates to the index using live-updater. • Query language is similar to Lucene with OR, AND, NOT. • The search index contains two types of fields: • Inverted Fields • Forward Fields
  • 29. Search and Retrieval Architecture • Static Rank • An auxiliary rank for members to help with retrieving at scale • Based on member profile and activity • Early termination • Index partitioned into N-shards, each retrieves and scores candidates • Not all members in a shard can be retrieved, so query is early terminated on the basis of static rank. • Galene Facet Counting: • Galene supports facet counting (such as region, titles, etc) for any given query. • Uses statistical counting approximation based on sample in each shard
  • 30. Layered Ranking Architecture • L1: Better to scoop into the talent pool and score/rank more candidates. • L2: Refines the short-listed talent to apply more dynamic features using external cache.
  • 32. Summary • Talent Search • Criteria Search, Recommendation System, Marketplace • Talent Search Ranking • Context-Aware Pairwise Training • Representation Learning & Deep Learning • GLMix Personalization • In-Session Personalization
  • 33. + Thank You Qi Guo Sahin Cem Geyik Bo Hu Cagri Ozcaglar Ketan Thakkar Xianren Wu Krishnaram Kenthapadi