Everyone's been talking about mobile-first all year long, but with 2018 looming, are you prepared for it? By now you’ve probably heard of the Majestic Million report and we thought we'd have a little fun and crawl it! Jon will show how many of the world’s largest sites are shaping up for Google's mobile-first Index. His talk will give an analytical overview of all key mobile aspects, such as site configuration, AMP, fetch time, mobile navigation, crawl depth, content differences, and mobile/desktop differences. Google offers best practice, but we will show you the data on how well the top sites are adjusting to mobile-first indexation.

1. @jondmyers @DeepCrawl
Mobile-First Preparedness- what we've learned from
crawling the top 1 million websites.
Jon Myers – Chief Growth Officer
@jondmyers @DeepCrawl
Source: http://glooadvertising.com.au/is-your-website-mobile-ready/

2. @jondmyers @DeepCrawl
Google is our world…

3. @jondmyers @DeepCrawl
Google is our world…
2 Trillion searches per year!

4. @jondmyers @DeepCrawl
Google is our world…
15% of searches new each day!

5. @jondmyers @DeepCrawl
Google is our world…
60% of search are on Mobiles!

6. @jondmyers @DeepCrawl
Really??...

7. @jondmyers @DeepCrawl
Sooooooo 2015! And before!
@jondmyers @DeepCrawl

8. @jondmyers @DeepCrawl
Google is thinking about it...
@jondmyers @DeepCrawl

9. @jondmyers @DeepCrawl
Helpful to see…
@jondmyers @DeepCrawl

10. @jondmyers @DeepCrawl
1 page at a time could take a while!…
@jondmyers @DeepCrawlSource: http://shop.adamjk.com/product/1-page-at-a-time

11. @jondmyers @DeepCrawl
History of my devices…

12. @jondmyers @DeepCrawl
History of Mobile…
WAP
HTML
1999 2007 2009 2014 2015 2017
Separate Mobile
Pages (Mobile Site
or Dynamic
Delivery)
Responsive
Design
Deep App
Linking
AMP &
Progressive
Web Apps
Mobile-
first
Indexing

13. @jondmyers @DeepCrawl
Time to get SH*T done…
@jondmyers @DeepCrawlSource: https://www.successimpulse.com/products/less-talk-more-action?variant=44575800839

14. @jondmyers @DeepCrawl
Heavy Data ahead…
@jondmyers @DeepCrawlSource: https://betanews.com/2015/09/14/8-early-warning-signs-of-problems-in-your-data-governance-plan/

15. @jondmyers @DeepCrawl
Being Majestic…
@jondmyers @DeepCrawl
Source: https://lafleur.marketing/blog/six-mad-men-marketing-tips-don-draper/

16. @jondmyers @DeepCrawl
What we did, why? And only we could…
@jondmyers @DeepCrawlSource: https://markfisherfitness.com/what-could-have-been/

17. @jondmyers @DeepCrawl
History of Mobile SEO Configuration…
Desktop
Mobile
Responsive
AMP pages
Mobile first
One version for desktop devices
One to rule them all - one version designed to work equally good
on desktop and mobile
Dedicated light weight version designed for a fast loading
Dedicated mobile pages served on a separate URL e.g. m.domain
or dynamically served on the same URL
Mobile becomes the PRIMARY version.

18. @jondmyers @DeepCrawl
Types of Mobile Configuration…
1. No Mobile Configuration (desktop only)
2. Responsive
3. Dynamic
4. Desktop + Dedicated Mobile
5. Desktop + AMP
6. Responsive/Dynamic + AMP
7. Desktop + Dedicated Mobile + AMP
8. Responsive + Dedicated Mobile + AMP
9. Responsive + Dynamic + Dedicated Mobile + AMP
10. Mobile/AMP only
11. Mobile + AMP

19. @jondmyers @DeepCrawl
Impact of Mobile-first…
-CRITICAL- - Desktop only (Heavily Affected)
-HIGH PRIORITY- - Desktop + Dedicated Mobile (Affected)
-HIGH PRIORITY- - Dynamic (Affected e.g. Content Issues)
-LOW PRIORITY- - Responsive (Not affected)
Impact of Mobile-first on different Mobile Configurations:

20. @jondmyers @DeepCrawl
Responsive Dynamic Separate
Mobile
Would expect Google to use both
user-agents on the same URL to
validate that the same content is
returned.
Google needs to crawl with both user
agents to validate the mobile version.
Hint: Use the Vary HTTP header!
Google needs to crawl the dedicated
mobile URLs with a mobile user agent to
validate the pages and confirm the
content matches the desktop pages.
How does Google crawl different configurations?

21. @jondmyers @DeepCrawl
Responsive design is the optimal situation…
“Responsive design is pretty
much the optimal situation
because regardless of which
device we use to crawl the site
we will see the same thing.
We’ll get the same content and
the structured data, the same
videos, the same images and the
same internal links. It’s all
there.”
Reference: https://youtu.be/7Aq9bFdfMuE?t=54m53s

22. @jondmyers @DeepCrawl
How does the world look?...
Source: http://abcnews.go.com/topics/news/doomsday-clock.htm

23. @jondmyers @DeepCrawl
How are sites set up for mobile?...

24. @jondmyers @DeepCrawl
Mobile Configuration Breakdown…
Breakdown of all sites where mobile configuration could be identified
Significant no. dynamic sites
– work to ensure meta data
is kept consistent across
different versions of page.
Responsive design dominates -
Path of least resistance
moving to Mobile-first Index
Separate Mobile unpopular –
high maintenance to ensure
desktop/mobile equivalence
15,000 sites
190,000 sites
795,000 sites

25. @jondmyers @DeepCrawl
Separate Mobile Sites – Are they up to scratch?...
Desktop
Page
Mobile
Page
Rel=canonical
61% of Separate Mobile sites have correct canonical to desktop page.

26. @jondmyers @DeepCrawl
Separate Mobile Sites – Are they up to scratch?...
Desktop
Page
Mobile
Page
No canonical
21% of Separate Mobile sites had no canonical to desktop equivalent.

27. @jondmyers @DeepCrawl
Separate Mobile Sites – Are they up to scratch?...
Desktop
Page
Mobile
Page
Other
page?
18% of Separate Mobile sites had canonicals pointing to wrong page

28. @jondmyers @DeepCrawl
Separate Mobile Sites – Desktop/Mobile Equivalence…
Of 15,000 Separate Mobile sites returning 200 response codes:
Desktop
Page
Mobile
Page
Matching title &
meta descriptions
Desktop
Page
Mobile
Page
Matching meta
descriptions
Desktop
Page
Mobile
Page
Matching title
7.7%
7.9%
5.8%

29. @jondmyers @DeepCrawl
Separate Mobile Issues…
1%
of separate mobile
sites had noindex
meta tag
25%
of all Separate Mobile sites
returned a non-200 status code
21% returned 3xx redirection
2% Separate Mobile sites returned
4xx errors
1% returned 5xx errors

30. @jondmyers @DeepCrawl
Google pushing for responsive web…
https://webmasters.googleblog.com/2017/09/how-to-move-from-
m-dot-urls-to.html
Separate Mobile
Google needs to crawl the dedicated
mobile URLs with a mobile user agent to
validate the pages and confirm the
content matches the desktop pages.

31. @jondmyers @DeepCrawl
Order of Mobile-first Indexing…
“If you have a vary header
which returns a different
page for a mobile user agent,
Google will use that as the
mobile page instead of the
responsive page.”
Reference:https://www.youtube.com/watch?v=8NyZypIfO
zI&t=36m31s

32. @jondmyers @DeepCrawl
Dynamic configurations…
Initial analysis found ?% of websites that serve
different content based on user agent do not have
a Vary: User-agent header.
Dynamic – same URL, different HTML
dependent on user agent

33. @jondmyers @DeepCrawl
Let’s talk about speed…
Source: https://play.google.com/store/apps/details?id=com.ea.game.nfs14_row

34. @jondmyers @DeepCrawl
Fetch times – The need for speed…
Percentage of sites split by fetch time
Fetch time includes time taken to fetch URL and display the HTTP response. This doesn’t
include the time taken to request or run any associated resources (such as images or scripts) on
the page.

35. @jondmyers @DeepCrawl
But are the top million sites really that fast?...

36. @jondmyers @DeepCrawl
Fetch times – Configuration breakdown…
Dynamically servedResponsive Separate Mobile
Average fetch time by configuration (secs):
Separate mobile sites have the slowest fetch times.
Responsive sites have fastest fetch times of
three mobile configurations.

37. @jondmyers @DeepCrawl
Fetch time with mobile desktop user agents…
19% of sites had a
fetch time between 1-
2 seconds.
68% of sites had a
fast fetch time – below
one second.
13% of sites had a
slow fetch time
exceeding 2 seconds

38. @jondmyers @DeepCrawl
What does Google have to say about site performance?...
Source: https://www.thinkwithgoogle.com/about/

39. @jondmyers @DeepCrawl
Google research – Performance per vertical…
Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/
Consumer sites
have faster more
responsive web
servers.
Business and
finance lagging
behind.
Technology my
arse!

40. @jondmyers @DeepCrawl
Google research – Performance per vertical…
Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/
Google says to aim for 2.4
seconds load time for a
page!!!???!!

41. @jondmyers @DeepCrawl
Lets Talk Page Size…

42. @jondmyers @DeepCrawl
Google research – Page Size…
Source: https://www.thinkwithgoogle.com/marketing-resources/data-measurement/mobile-page-speed-new-industry-benchmarks/
According to Google, sites
across all verticals are on
average larger than the
500kb recommendation.
Again, technology sites
aren’t leading the way.

43. @jondmyers @DeepCrawl
What did we find out about HTML size?...
94%<=200k
5%200k - 500k

44. @jondmyers @DeepCrawl
Accelerated Mobile Pages…
Percentage of sites with AMP version of homepage:
Responsive
0.63%
Dynamic
0.61%
Separate Mobile
0.86%

45. @jondmyers @DeepCrawl
What about HTTPS?...

46. @jondmyers @DeepCrawl
HTTPS – How many have made the switch?...
HTTPS
(22%)
HTTP
(78%)
Has HTTPS reached critical mass? Our data
says there’s a long way to go for the top one
million sites.
This figure is much lower than currently reported
Mozcast – 68% of first page results have SSL
encryption.
Source: http://mozcast.com/features

47. @jondmyers @DeepCrawl
HTTPS – Adoption across mobile configurations…
Dynamically served sites appear
to be particularly slow to adopt
HTTPS.
Surprisingly Separate Mobile sites
leading the way for the switch to
HTTPS.

48. @jondmyers @DeepCrawl
HTTPS doesn’t end there…
Watch out for non-secure form fields (we
found a lot of them) as Chrome is now
flagging these with security warnings.
12.7% incorrectly configured sites with links
between protocols and some sites with mixed
content e.g. a HTTPS page with reference to a
script on an HTTP URL.

49. @jondmyers @DeepCrawl
Carving up Configurations by Country…

50. @jondmyers @DeepCrawl
Splitting the Majestic Million by TLDs…
Percentage of sites per TLD

51. @jondmyers @DeepCrawl
.edu & .gov are leading the way with responsive design…

52. @jondmyers @DeepCrawl
…and .co.uk TLDs aren’t far behind…
ResponsiveDynamic Separate

53. @jondmyers @DeepCrawl
What next? Plenty of issues…
Non Secure Form Fields
Pages without Valid Canonical
Tag
Failed URLs
Empty Pages
Max Fetch Time
Thin Pages
Missing H1 Tags
Non-301 Redirects
Pages with Duplicate Body
Duplicate Pages
Malformed URLs
High External Linking
Max URL Length
Missing Titles
Pages with Duplicate Titles
No Descriptions & No Snippets
Unauthorised Pages
5xx Errors
Max Links
Broken Pages (4xx Errors)
Max Description Length
Short Titles
Unlinked Paginated Pages
Hreflang to Non-200 URLs
Non-rel Alted AMP Pages
All Broken Links
Non-reciprocal Mobile/AM
Duplicate Description Sets
Max Content Size
Uncategorised HTTP Response Codes
Max Title Length
Duplicate Body Sets
Duplicate Page Sets
Pages with Duplicate Descriptions
Canonical to Non-200
Max Redirections
Max HTML Size
Short Descriptions
Redirect Loops
Duplicate Title Sets
And more…
Non-200 Mobile/AMP
Excessive Redirects In (Admin Only)
Mobile Links Out Mismatch
Mobile Links In Mismatch
Mobile Word Count Mismatch
Mobile Content Mismatch
Duplicate Pages including Primary

54. @jondmyers @DeepCrawl
In a nutshell…

55. @jondmyers @DeepCrawl
Separate Mobile sites are poorly configured

56. @jondmyers @DeepCrawl
Dynamic sites need to check they have Vary HTTP response header…

57. @jondmyers @DeepCrawl
HTTPS adoption relatively low amongst top million sites…

58. @jondmyers @DeepCrawl
Fetch time is surprisingly low???…

59. @jondmyers @DeepCrawl
Watch this space for an even deeper dive into the data…
Source: http://dive-bohol.com/

60. @jondmyers @DeepCrawl
But what does the future hold?...

61. @jondmyers @DeepCrawl
Go beyond Responsive with PWA’s…
Progressive Web Apps is:
• Progressive
• Responsive
• App-like
• And more...
Progressive Web Apps can fall back to AMP.
Hint… first load as AMP.
You should think about it now.
Progressively Web App is THE FUTURE.

62. @jondmyers @DeepCrawl
Jon Myers
DeepCrawl
Chief Growth Officer