Speaker: Charlie Swanson, Software Engineer, MongoDB Level: 200 (Intermediate) Track: How We Build MongoDB Learn how MongoDB answers your queries from a query system engineer. If you've ever had a performance problem with a query but didn't know how to find the cause, or if you've ever needed to confirm that your shiny new index is being put to work, the explain command is an excellent place to start. MongoDB's explain system is a powerful tool for solving this type of problem, but can be intimidating and unwieldy to use. In this talk, we will discuss how the explain command works and break down its output into digestible pieces. What You Will Learn: - Exactly how indexes are used during your queries and aggregations - How to diagnose your poorly performing operations - How to tune your most important operations to ensure that they scale seamlessly

1. #MDBW17
DECIPHERING EXPLAIN OUTPUT
Charlie Swanson

2. Understand how
MongoDB answers
queries
KNOWLEDGE
Figure out what's
going on
DEBUGGING
Learn some tricks to
optimize your queries &
aggregations
BEST PRACTICES
GOALS OF THIS TALK

3. #MDBW17
CHARLIE
SWANSON
SOFTWARE ENGINEER - QUERY TEAM

4. #MDBW17
OVERVIEW
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

5. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

6. #MDBW17
TARGET QUESTIONS
• Is your query using an index? Which one? 🤔

7. #MDBW17
TARGET QUESTIONS
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
🤔

8. #MDBW17
TARGET QUESTIONS
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
🤔

9. #MDBW17
TARGET QUESTIONS
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
• Why did the server choose to answer the query the way it did?
🤔

10. #MDBW17
TARGET QUESTIONS
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
• Why was your winning plan chosen?
• Can my queries go faster?
🤔

11. #MDBW17
EXAMPLE QUERY

12. #MDBW17
HOW CAN THE SERVER ANSWER THIS QUERY?

13. #MDBW17
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅
OPTION 1: COLLECTION SCAN

14. #MDBW17
COLLECTION SCAN: TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅
204587
190587
SORT
01.
02.

15. #MDBW17
COLLECTION SCAN: TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅
01.
02.
SORT
PROJECTION
02
01
{nFavorites: 204587,
username: '@TaylorSwift'}
{nFavorites: 109587,
username: '@Charlie'}

16. #MDBW17
01.
02.
SORT
PROJECTION
COLLECTION SCAN
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅

17. #MDBW17
OPTION 1: COLLECTION SCAN
SORT
PROJECT
COLLECTION SCAN

18. #MDBW17
OPTION 2: INDEX SCAN TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…

19. #MDBW17
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…
Stop
when
entry is
smaller
than
100,000

20. #MDBW17
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…
SORTED!

21. #MDBW17
TWITTER.TWEETS
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN
SORT

22. #MDBW17
TWITTER.TWEETS
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN
FETCH
204587
190587
{ _id: 400000,
createdAt: ISODate(…),
username: "@TaylorSwift", …
{ _id: 400001,
createdAt: ISODate(…),
username: "@Charlie", … }

23. #MDBW17
TWITTER.TWEETS
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN
FETCH
204
190
{ _id: 400000,
{ _id: 400001,
PROJECTION
{nFavorites: 204587,
username: '@TaylorSwift'}
{nFavorites: 109587,
username: '@Charlie'}
{ _id: 400000,
createdAt: ISODate(…),
{ _id: 400001,
createdAt: ISODate(…),
username: "@Charlie", }

24. #MDBW17
FETCH
204
190
{ _id: 400000,
{ _id: 400001,
PROJECTION
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN

25. #MDBW17
OPTION 2: INDEX SCAN
FETCH
PROJECT
INDEX SCAN

26. #MDBW17
MANY WAYS TO ANSWER A QUERY… WHICH WAS IT?
SORT
PROJECT
COLLECTION
FETCH
PROJECT
INDEX SCAN
…

27. #MDBW17
WHAT IS EXPLAIN?
1. Command to explain execution of various other commands
2. Helper on shell cursor object

28. WHAT IS EXPLAIN? - COMMAND

29. WHAT IS EXPLAIN? - SHELL HELPER

30. WHAT IS EXPLAIN? - SHELL HELPER
???

31. #MDBW17
QUERY PLANS
SORT
PROJECT
COLLECTION
FETCH
PROJECT
INDEX SCAN
…OR
FETCH
INDEX SCAN INDEX SCAN

32. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

33. #MDBW17
EXPLAIN OUTPUT
> db.collection.find(…).explain("queryPlanner")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
…
}
Optional, this is the
default

34. #MDBW17
EXPLAIN OUTPUT
> db.collection.find(…).explain()
{queryPlanner: {
winningPlan: {
stage: "SORT",
inputStage: {
stage: "FETCH",
inputStage: {
stage: "IXSCAN"
}
}
}
}
}}
FETCH
INDEX SCAN
{nFavorites: -1}
SORT

35. #MDBW17
EXPLAIN OUTPUT
> db.collection.find(…).explain()
{
…
stage: "IXSCAN"
keyPattern: {nFavorites: -1},
indexBounds: {
a: [ "[inf.0, 100000]" ]
},
… // Other index scan
// specific stats.
…
}}
FETCH
INDEX SCAN
keyPattern: {
nFavorites: -1
}
indexBounds: […]
…
SORT

36. #MDBW17
EXPLAIN OUTPUT
> db.collection.find(…).explain()
{
"queryPlanner" : {
…
"winningPlan" : {
// Encodes selected plan.
},
"rejectedPlans" : […]
},
…
}
FETCH
INDEX SCAN
SORT

37. #MDBW17
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}
EXPLAIN OUTPUT
FETCH
INDEX SCAN
SORT
COLL_SCAN
SORT

38. APPLYING THIS INFORMATION

39. #MDBW17
IS YOUR QUERY USING THE INDEX YOU EXPECT?

40. #MDBW17
IS YOUR QUERY USING THE INDEX YOU EXPECT?
FETCH
SORT
✓ COLLECTION SCAN
SORT
✗
INDEX SCAN
keyPattern: {nFollowers: -1}

41. #MDBW17
IS YOUR QUERY USING THE INDEX YOU EXPECT?

42. #MDBW17
IS YOUR QUERY USING THE INDEX YOU EXPECT?
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1})
.sort({nFavorites: -1})

43. #MDBW17
IS YOUR QUERY USING THE INDEX YOU EXPECT?
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1})
{ "queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {"nFavorites": -1},
"indexBounds": {
"nFavorites": ["[inf.0, 100000.0]"]
} } } },
"rejectedPlans": [ ] } }

44. #MDBW17
IS YOUR QUERY USING AN INDEX TO PROVIDE THE
SORT?

45. #MDBW17
IS YOUR QUERY USING AN INDEX TO PROVIDE THE
SORT?
FETCH
INDEX SCAN
✓ ✗FETCH
INDEX SCAN
SORT

46. #MDBW17
IS YOUR QUERY USING AN INDEX TO PROVIDE THE
SORT?

47. #MDBW17
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1})
{ "queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {"nFavorites": -1},
"indexBounds": {
"nFavorites": ["[inf.0, 100000.0]"]
} } } },
"rejectedPlans": [ ] } }
IS YOUR QUERY USING AN INDEX TO PROVIDE THE
SORT?
NO SORT STAGE ✅

48. #MDBW17
SORT_MERGE IS OK
✓SORT_MERGE
INDEX SCAN
FETCH
INDEX SCAN

49. #MDBW17
BONUS: IS YOUR QUERY USING AN INDEX TO
PROVIDE THE PROJECTION?

50. #MDBW17
COMPOUND INDEX TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{username: 1, nFavorites: -1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEng" 83092
"@MongoDBEng" 76032
… …

51. #MDBW17
COMPOUND INDEX
db.tweets.find({
username: {$in: ["@MongoDBEng", "@MongoDB"]},
nFavorites: {$gt: 50000}}})
TWITTER.TWEETS
INDEX:
{username: 1, nFavorites: -1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEng" 83092
"@MongoDBEng" 76032
… …

52. #MDBW17
COMPOUND INDEX
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
TWITTER.TWEETS
INDEX:
{username: 1, nFavorites: -1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEng" 83092
"@MongoDBEng" 76032
… …

53. #MDBW17
COMPOUND INDEX
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
TWITTER.TWEETS
{username: 1, nFavorites:
-1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEn
g"
83092
"@MongoDBEn
g"
76032
… …
INDEX SCAN
PROJECTION

54. #MDBW17
COMPOUND INDEX
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
TWITTER.TWEETS
{username: 1, nFavorites:
-1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEn
g"
83092
"@MongoDBEn
g"
76032
… …
INDEX SCAN
PROJECTION
"COVERED"

55. #MDBW17
COMPOUND INDEX
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1, other: 1})
{username: 1, nFavorites:
-1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEn
g"
83092
"@MongoDBEn
g"
76032
… …
TWITTER.TWEETS
INDEX SCAN
PROJECTION
FETCH
"NOT COVERED"

56. #MDBW17
COMPOUND INDEX
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
.sort({username: 1})
TWITTER.TWEETS
{username: 1, nFavorites:
-1}
"@Charlie" 29023
"@MongoDB" 87983
"@MongoDB" 60587
"@MongoDB" 7983
"@MongoDBEn
g"
83092
"@MongoDBEn
g"
76032
… …
INDEX SCAN
PROJECTION
"COVERED"
PROJECTION
"COVERED"
SORT

57. #MDBW17
PROJECTION
INDEX SCAN
✓ ✗
PROJECTION
INDEX SCAN
FETCH
IS YOUR QUERY USING AN INDEX TO PROVIDE THE
PROJECTION?

58. #MDBW17
THE POWER OF "QUERYPLANNER"
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• Is your query using an index to provide the projection?

59. #MDBW17
NEXT UP: "IT'S USING AN INDEX, SO WHAT'S TAKING
SO LONG?"
db.tweets.explain().find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
FETCH
INDEX SCAN
keyPattern: {createdDate: 1}

60. #MDBW17
IT'S USING AN INDEX, SO WHAT'S TAKING SO LONG?
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
FETCH
INDEX SCAN
keyPattern: {createdDate: 1}
12:02
12:03
12:04
…
INDEX SCAN
INDEX:
{createdAt: 1}

61. #MDBW17
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:02,
favorites: [
"@MongoDB",
"@taylorswift"
]}
FETCH
FETCH
filter: {
favorites: "@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate: 1}
IT'S USING AN INDEX, SO WHAT'S TAKING SO LONG?
12:02
12:03
12:04
…
INDEX:
{createdAt: 1}

62. #MDBW17
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:02,
favorites: [
"@MongoDB",
"@taylorswift"
]}
FETCH
FETCH
filter: {
favorites: "@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate: 1}
❌
IT'S USING AN INDEX, SO WHAT'S TAKING SO LONG?
12:02
12:03
12:04
…
INDEX:
{createdAt: 1}

63. #MDBW17
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:03,
favorites: [
"@eliothorowitz",
"@taylorswift"
]}
FETCH
FETCH
filter: {
favorites: "@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate: 1}
IT'S USING AN INDEX, SO WHAT'S TAKING SO LONG?
12:02
12:03
12:04
…
INDEX:
{createdAt: 1}

64. #MDBW17
SO HOW MANY OF THEM WERE THROWN OUT?
• What percentage of the index keys in the scanned range ended up
matching the predicate?
• What's the selectivity?

65. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

66. #MDBW17
01: OUTPUT IS VERY LARGE

67. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

68. #MDBW17
EXPLAIN MODE: "QUERYPLANNER"
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}

69. #MDBW17
• "executionStats"
EXPLAIN MODE: "EXECUTIONSTATS"
created by Mike Ashley from Noun Project
created by Creative Stall from Noun Project

70. #MDBW17
EXPLAIN MODE: "EXECUTIONSTATS"
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": { // New!
…,
"executionStages": {…}
}
…
}

71. #MDBW17
DETAILS: "EXECUTIONSTATS"
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : { /* Same as before. */ },
"executionStats": {
// Top-level stats.
"executionStages": {
stage: "SORT",
// Sort stats.
inputStage: {
// etc, etc.
}
}
}
…
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

72. #MDBW17
> db.tweets.find(…).explain("executionStats")
{
…,
"executionStats" : {
// Top-level stats.
"nReturned" : 390000,
"executionTimeMillis" : 4431,
"totalKeysExamined" : 390000,
"totalDocsExamined" : 390000,
"executionStages" : {…}
},
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
DETAILS: "EXECUTIONSTATS"

73. #MDBW17
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
…
"sortPattern" : { "nFollowers" : 1 },
"memUsage" : 20280000,
"memLimit" : 33554432,
"inputStage" : {…}
}
}
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
DETAILS: "EXECUTIONSTATS"

74. #MDBW17
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
…
"sortPattern" : { "nFollowers" : -1 },
"memUsage" : 20280000,
"memLimit" : 33554432,
"inputStage" : {…}
}
}
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
DETAILS: "EXECUTIONSTATS"

75. #MDBW17
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
"works" : 780003,
"advanced" : 390000,
"needTime" : 390002,
"isEOF" : 1,
"sortPattern" : { "b" : 1 },
…
"inputStage" : {…}
}
}
}
? FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
DETAILS: "EXECUTIONSTATS"

76. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
• These are all PlanStages
‒SortStage
‒FetchStage
‒IndexScanStage
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

77. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
• These are all PlanStages
• Each PlanStage implements
work()
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

78. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
• These are all PlanStages
• Each PlanStage implements
work(), returns one of:
• ADVANCED
• NEED_TIME
• IS_EOF
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

79. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

80. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
work()
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

81. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
work()
work()
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

82. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
work()
ADVANCED ID
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

83. #MDBW17
EXECUTION STATS: WORKS, ADVANCED, ETC.
ADVANCED {…}
work()
ADVANCED
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

84. #MDBW17
SORT
EXECUTION STATS: WORKS, ADVANCED, ETC.
NEED_TIME ADVANCED
work()
ADVANCED
FETCH
INDEX SCAN
keyPattern: {hashtags: 1}

85. #MDBW17
DETAILS: "EXECUTIONSTATS"
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
"works" : 780003,
"advanced" : 390000,
"needTime" : 390002,
"isEOF" : 1,
"sortPattern" : { "b" : 1 },
…
"inputStage" : {…}
}
}
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

86. #MDBW17
DETAILS: "EXECUTIONSTATS"
> db.tweets.find(…).explain("executionStats")
{"executionStats": {
"executionStages": {
stage: "SORT",
// Sort stats, includes "works", "advanced", …
inputStage: {
stage: "FETCH",
// Fetch stats, includes "works", "advanced", …
inputStage: {
// etc, etc.
}
}
}
}
…
}
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

87. #MDBW17
EXPLAIN MODE: "EXECUTIONSTATS"
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": { // New!
…,
"executionStages": {…}
}
…
}

88. APPLYING THIS INFORMATION

89. #MDBW17
HOW SELECTIVE IS YOUR INDEX?

90. #MDBW17
HOW SELECTIVE IS YOUR INDEX?
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
12:02
12:03
12:04
12:06
…
INDEX SCAN
FETCH
INDEX SCAN
keyPattern: {createdDate: -1}

91. #MDBW17
HOW SELECTIVE IS YOUR INDEX?
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{
"executionStats" : {
"nReturned" : 314,
"totalKeysExamined" : 2704, // < 12% matched
…
}
FETCH
INDEX SCAN
keyPattern: {createdDate: -1}

92. #MDBW17
WHAT'S THE MOST EXPENSIVE PART OF YOUR PLAN?

93. #MDBW17
WHAT'S THE MOST EXPENSIVE PART OF YOUR PLAN?
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
}) FETCH
filter: {
favorites: "@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate: 1}

94. #MDBW17
WHAT'S THE MOST EXPENSIVE PART OF YOUR PLAN?
db.tweets.explain("executionStats").find(…)
FETCH
executionTimeMillisEstimate: 431
INDEX SCAN
executionTimeMillisEstimate: 67

95. #MDBW17
WHAT'S THE MOST EXPENSIVE PART OF YOUR PLAN?
db.tweets.explain("executionStats").find(…)
FETCH
works: 2705
advanced: 314
needTime: 2391
INDEX SCAN

96. #MDBW17
OUR PROGRESS
• "queryPlanner"
• Is your query using the index you expect?
• Is your query using an index to provide the sort?
• Is your query using an index to provide the projection?
• "executionStats"
• How selective is your index?
• Which part of your plan is the most expensive?

97. #MDBW17
NEXT UP: "WHY WAS THIS PLAN CHOSEN?"
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
• We had an index on {favorites: 1}, would that have been
faster?
🤔

98. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

99. #MDBW17
EXPLAIN MODE: "QUERYPLANNER"
❓
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}

100. #MDBW17
QUERY PLANNING

101. #MDBW17
• MultiPlanStage::pickBestPlan()
QUERY PLANNING
…
MultiPlanStage
work()
work()
work()

102. #MDBW17
• MultiPlanStage::pickBestPlan()
QUERY PLANNING
…
MultiPlanStage
ADVANCED
NEED_TIME
ADVANCED

103. #MDBW17
• MultiPlanStage::pickBestPlan()
QUERY PLANNING
…
MultiPlanStage
Advances: 78 22 50

104. #MDBW17
• MultiPlanStage::pickBestPlan()
QUERY PLANNING
…
MultiPlanStage
Advances: 78 22 50

105. #MDBW17
EXPLAIN MODE: "ALLPLANSEXECUTION"
> db.tweets.find(…).explain("allPlansExecution")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": {
…,
"executionStages": {…},
"allPlansExecution": […] // New!
}
…
}

106. #MDBW17
EXPLAIN MODE: "QUERYPLANNER"
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…}
},
"rejectedPlans" : [
{…},
…
],
}}

107. #MDBW17
EXPLAIN MODE: "EXECUTIONSTATS"
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…}
},
"rejectedPlans" : [
{…},
…
],
"executionStats": {
"executionStages": {…}
}
}}

108. #MDBW17
EXPLAIN MODE: "ALLPLANSEXECUTION"
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…}
},
"rejectedPlans" : [
{…},
…
],
"executionStats": {
"executionStages": {…}
"allPlansExecution": [
{…},
{…},
…
]
}
}}

109. #MDBW17
EXPLAIN MODE: "ALLPLANSEXECUTION"
db.tweets.explain("allPlansExecution").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{
"executionStats": {
"allPlansExecution": [
{nReturned: 34,
executionStages: { /* Index Scan on "favorites" */ }
},
{nReturned: 101,
executionStages: { /* Index Scan on "createdDate" */ }
}
]
}
…
}

110. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands

111. #MDBW17
SLOW QUERIES
• Queries with response time >100ms (server side) are logged:
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find
{ find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort:
{ nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary:
IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:
2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872
} }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

112. #MDBW17
SLOW QUERIES
• Queries with response time >100ms (server side) are logged:
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find
{ find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort:
{ nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary:
IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:
2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872
} }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

113. #MDBW17
SLOW QUERIES
• Queries with response time >100ms (server side) are logged:
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command:
find { find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0,
singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0,
nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907
docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global:
{ acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r:
2872 } } } protocol:op_command 1493ms

114. #MDBW17
SLOW QUERIES
• Queries with response time >100ms (server side) are logged:
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find
{ find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort:
{ nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary:
IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1
cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } },
Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

115. #MDBW17
SLOW QUERIES
• Queries with response time >100ms (server side) are logged.
• Configurable via profiling parameter 'slowMs'

116. #MDBW17
THE PROFILE
• If turned on, queries show up in the system.profile collection
{ "op": "query",
"ns": "twitter.tweets",
"query": { "find": "tweets", "filter": { … }, "limit": 20, "sort": { … }, "projection": { … } },
"millis": 1355,
"planSummary": "IXSCAN { nFavorites: -1 }",
"execStats": {
"stage": "PROJECTION",
"inputStage": {
"stage": "SORT",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
} } } } } }

117. #MDBW17
OTHER COMMANDS
db.runCommand({
explain: {/* command */},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})

118. #MDBW17
OTHER COMMANDS
db.runCommand({
explain: {findAndModify: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})

119. #MDBW17
OTHER COMMANDS
db.runCommand({
explain: {update: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})

120. #MDBW17
OTHER COMMANDS
db.runCommand({
explain: {aggregate: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})

121. #MDBW17
AGGREGATION
• Aggregation is special…
runCommand({
aggregate: "collection",
pipeline: […],
explain: <true|false>
})

122. #MDBW17
AGGREGATION
• Aggregation is was special…
• 3.4 and earlier:
• 3.6 and beyond:
runCommand({
aggregate: "collection",
pipeline: […],
explain: <true|false>
})
runCommand({explain: {
aggregate: "collection",
pipeline: […],
},
verbosity: "…" })

123. #MDBW17
AGGREGATION
db.explain().aggregate([{$group: {…}}, {$project: {…}}])
{
stages: [
{$cursor: {…}},
{$group: {…}},
{$project: {…}},
]
}

124. #MDBW17
AGGREGATION
db.explain().aggregate([{$group: {…}}, {$project: {…}}])
{
stages: [
{$cursor: {
query: {…},
fields: {…},
queryPlanner: {/* same as query explain! */},
executionStats: {/* 3.6+ only, same as query explain! */}
}},
{$group: {…}},
{$project: {…}} ] }

125. #MDBW17
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about
plan selection
Log messages
What is explain? The profile
Other commands
SUMMARY

126. SOME FINAL THOUGHTS

127. #MDBW17
01: OUTPUT IS VERY LARGE

128. #MDBW17
COMPASS SHOUTOUT

129. #MDBW17
COMPASS SHOUTOUT

130. #MDBW17
COMPASS SHOUTOUT

131. #MDBW17
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS
Your Application
MongoDB
created by Mike Ashley from Noun Project

132. #MDBW17
Your Application MongoDB
Can I see all the tweets
with hashtag
"#MDBW17" from this
hour?
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

133. #MDBW17
Your Application MongoDB
Hmm… Let me
think about
that…
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

134. #MDBW17
Your Application MongoDB
Ah! Here are
your results!
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

135. #MDBW17
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS
Your Application MongoDB
What took you
so long?!

136. #MDBW17
Your Application MongoDB
What took you
so long?!
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

137. #MDBW17
Your Application MongoDB
What took you
so long?!
- network latency
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

138. #MDBW17
Your Application MongoDB
What took you
so long?!
- network latency
- large result set
{…} {…} {…} {…} {…} {…} {…}
{…}
{…}
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

139. #MDBW17
Your Application MongoDB
What took you
so long?!
- network latency
- large result set
- server contention
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS

140. #MDBW17
Your Application MongoDB
What took you
so long?!
- network latency
- large result set
- server contention
- query planning problem
02: EXPLAIN WON'T SOLVE ALL YOUR PROBLEMS