This document discusses site crawling tools and techniques. It provides information on various crawling tools, their uses, and pricing. Key things to look for when crawling a site include errors, redirects, duplicate titles and content, canonicals, image alt text, and PDF files. The document recommends crawling sites at minimum every quarter, before and after major updates, and if there are changes in organic traffic. Crawling can be used to test sitemaps, backlinks, and conduct content inventories.

1. Site Crawling
What To Do & What To Look For
@katemorris

2. The “Oh Shit” Moment
@katemorris

3. The “WTF” Moment
@katemorris

4. WHEN IS THE
LAST TIME YOU
@katemorris
CRAWLED
YOUR SITE?

5. CRAWLING TOOLS CRAWLING DATA
CRAWLING FUN CRAWLING TIMELINES
@katemorris

6. CRAWLING
@katemorris
TOOLS

7. @katemorris
Moz
Best For:
• In-House Marketers
• Beginning SEOs
Cost: $99/month+
User Score: 1-2

8. @katemorris
Raven
Best For:
• In-House Marketers
• Small Agencies
Cost: $99/month+
User Score: 3

9. Xenu Link Sleuth
@katemorris
Best For:
• Technical Experts
• Developers
Cost: Free
User Score: 4

10. Screaming Frog
@katemorris
Best For:
• Technical Marketers
• SEOs
• Developers
Cost: $160/year
User Score: 5

11. @katemorris
Deep Crawl
Best For:
• Expert SEOs
• Sites with Large
Footprints
Cost: $80/ month+
User Score: 6

12. Crawler User Scale
1 2 3 4
@katemorris
Unrelated to
SEO/Marketing
Novice Level
SEO
Knowledge,
Good SEO
Knowledge,
Non-Technical
Good
Technical,
Minimal SEO
5 6 7
Good SEO and
Technical
Knowledge
Expert Level SEO
Knowledge,
Minimal
Technical
Expert Technical
and SEO
Knowledge

13. Crawler User Scale
@katemorris
Unrelated to
SEO/Marketing
Novice Level
SEO
Knowledge,
Good SEO
Knowledge,
Non-Technical
Good
Technical,
Minimal SEO
Good SEO and
Technical
Knowledge
Expert Level SEO
Knowledge,
Minimal
Technical
BUILD
YOUR
OWN
Expert Technical
and SEO
Knowledge

14. THE
CRAW
@katemorris
L
All Examples Use Screaming
Frog or DeepCrawl

15. DeepCrawl Dashboard
@katemorris

16. 4XX/5XX Errors
@katemorris

17. 4XX/5XX Errors
@katemorris

18. Internal Redirects
@katemorris

19. External Redirects
@katemorris

20. Duplicate Titles
@katemorris

21. Duplicate Content
@katemorris

22. @katemorris
URL Issues

23. Meta Descriptions
@katemorris

24. Meta Descriptions
@katemorris
DeepCrawl breaks it down for you.

25. @katemorris
Canonicals

26. Image Alt Text
@katemorris

27. @katemorris
PDF Files

28. HOW OFTEN
DO I HAVE TO
@katemorris
DO THIS?

29. Every Quarter
@katemorris

30. Before and After All Major Site Updates
WEBSITE UPDATE WEBSITE UPDATE WEBSITE UPDATE WEBSITE UPDATE WEBSITE UP
@katemorris

31. Any Major Changes in Organic
Traffic
@katemorris

32. FUN CRAWLER
@katemorris
USES

33. Testing Sitemaps
@katemorris

34. Testing Sitemaps
@katemorris

35. Testing Backlinks
Issue: Backlink Analysis is time consuming,
especially checking backlink lists for live
links.
Solution: Make a text list of all the backlinks
and crawl. For Screaming Frog, set a
custom search for the client site.
@katemorris

36. Testing Backlinks
@katemorris

37. Testing Backlinks
@katemorris

38. Content Inventory
@katemorris

39. Confirm GA Implementation
@katemorris

40. @katemorris
Want more?
http://www.seerinteractive.com/blog/screaming-frog-guide

41. @katemorris
Do This
Download and find your favorite crawler.
Make a calendar entry that reoccurs every
three months to remind you to crawl your
site.
At minimum, review:
– Errors
– Redirects
– Duplicated Titles
– XML Sitemaps

42. Thanks
Icons provided by: Medialoot