We've all spent hours listening and researching how Google says they interact with our sites. Server logs are a critical view into how Googlebot actually interacts with your sites. Learn how to identify different Googlebot behaviors, crawl waste, and optimization opportunities.
8. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
What exactly is Googlebot
crawling?
9. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Things this report is not
● solely traditional web pages
● details about which
Googlebot is crawling
● just pages 200 response
codes
● reflective of how many
unique pages are crawled
● bigger ≠ better
10. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Does reality match our
expectations?
12. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Was the crawl healthy?
13. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Googlebot is designed to be a
good citizen of the web...
For Googlebot a speedy site is
a sign of healthy servers...
If the site slows down or
responds with server errors,
the [crawl rate] limit goes down
and Googlebot crawls less.
Politeness is
job 0
https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-
googlebot.html
14. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Is it already too late?
18. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
How do I get logs?
19. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
credit: https://flic.kr/p/cnorAf
Make new allies.
20. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Ask: Is there already a log
management platform in place?
Be Clear: We do not want Personal
Identification Information (PII) and
request it be removed
BE SPECIFIC.
21. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Web
Server 1
Web
Server 2
Web
Server 3
CDN
DDOS Mitigation/Bot Manager
Logs can
come from
multiple
places in
your stack.
Load Balancer
57. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Use Case
Intermittent crawl
errors
58. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Query
Server parity
59. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Analysing Log Files
Screaming Frog Log Analysis
BigQuery + Google Cloud Storage Services
Excel + .csv
Big Query + .csv
Command Line
60. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Making the most of logs means adapting to
your environment and making new friends.
61. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Iterate.
Test.
Share what you learn.
62. Jamie Alberico | @Jammer_Volts | #TechSEOBoost
Thank you for your time, energy, and
being part of this wonderful
community
–
@Jammer_Volts
totally@not-a-robot.com