A few of the recent findings we discovered whilst working on an SEO beast which cover crawling, server log file analysis, site speed optimization and database optimizations. Technical SEO insights
5. SITE
DEFAULT
SETTINGS
IN
CRAWL
GSC
SITE
TYPE SITE DEFAULT
SECONDS
BETWEEN
REQUESTS
SITE DEFAULT
REQUESTS
PER
SECOND
CMS
TYPE CDN
USED?
SHARED HOSTING
/
DEDICATED
SERVER
Business
Directory
–
Data driven
10 0.1 NONE
-‐ CUSTOM
BUILT
APPLICATION
NO DEDICATED
SERVER
Gambling
– data
driven
10 0.1 NONE – CUSTOM
BUILT
APPLICATION
NO DEDICATED
SERVER
Travel
/
Auto 5 0.2 CUSTOM BUILD
/
WORDPRESS
HYBRID
CLOUDFLARE DEDICATED
SERVER
Distribution Service
(Database
Driven
Site)
0.003 333.333 NONE – CUSTOM
BUILT
APPLICATION
NO SHARED
HOSTING
Ecommerce
fashion 0.001 100 SHOPIFY SHOPIFY SHOPIFY
PLATFORM
Business
Blog 0.008 125 WORDPRESS NO SHARED
HOSTING
Ecommerce
Fashion 0.003 333.333 SHOPIFY SHOPIFY SHOPIFY
PLATFORM
6. SITE
DEFAULT
SETTINGS
IN
CRAWL
GSC
SITE
TYPE SITE DEFAULT
SECONDS
BETWEEN
REQUESTS
SITE DEFAULT
REQUESTS
PER
SECOND
CMS
TYPE CDN
USED?
SHARED HOSTING
/
DEDICATED
SERVER
Ecommerce
– Home &
Garden
0.016 MAGENTO NO DEDICATED
SERVER
Jobs board
– Dynamic
data
333.333 0.003 UMBRACO NO DEDICATED
SERVER
Ecommerce -‐ Gifts 125 0.008 WORDPRESS CLOUDFLARE SHARED HOSTING
Business Consultancy
– Brochure
Site
333.333 0.003 WORDPRESS NO DEDICATED SERVER
Jobs
Board 100 0.01 UMBRACO CLOUDFLARE DEDICATED
SERVER
7. Popular
CMS
’Rule
Patterns’
(URL
Parameters)
ALL
WILL
HAVE
SOME
COMMON
PATTERNS
WHICH
CAN
BE
LEARNED
FOR
EFFICIENCY
BUILDS
‘HINTS’
ON
WHAT
NOT
TO
CRAWL
26. IT
CAN
INCREASE
TTFB
Source:
https://blog.cloudflare.com/ttfb-‐time-‐to-‐first-‐byte-‐considered-‐meaningles/
27. TTFB
is
Often
Increased
With
Gzip
Compression
Compression
has
a
cost
because
it
increases
TTFB
but
the
web
page
loads
quicker.
So,
it’s
better
for
users
but
TTFB
is
negatively
impacted
whist
the
page
is
unzipped
Gzip Compression
Source:
https://blog.cloudflare.com/ttfb-‐time-‐to-‐
first-‐byte-‐considered-‐meaningles/
Gzip Compression
effects
TTFB
negatively.
But
the
page
loads
faster
in
the
end
28. IS
TTFB
REALLY
ALL
THAT
??
Source:
https://blog.cloudflare.com/ttfb-‐time-‐to-‐first-‐byte-‐considered-‐meaningles/
H
29. You
should
reduce
your
server
response
time
under
200ms
(Google
Developers)
45. Messy
Crawl
with
Builder
Cache
Included
Files
(DEPENDENT
FILES
ARE
GIVEN
PRIORITY)
46. Cache
Control
in
.htaccess
#
One
year
for
image
files
<filesMatch ".(jpg|jpeg|png|gif|ico)$">
Header
set
Cache-‐Control
"max-‐age=31536000,
public"
</filesMatch>
#
One
month
for
css and
js
<filesMatch ".(css|js)$">
Header
set
Cache-‐Control
"max-‐age=2628000,
public"
</filesMatch>
62. Sources
&
References
• http://ohgm.co.uk/crawl-‐rate-‐defaults/
• https://developers.google.com/speed/docs/insights/Server
• https://varvy.com/pagespeed/defer-‐loading-‐javascript.html
• https://www.red-‐gate.com/simple-‐talk/sql/t-‐sql-‐programming/fuzzy-‐
searches-‐sql-‐server/
• Bar-‐Yossef,
Z.,
Keidar,
I.
and
Schonfeld,
U.,
2009.
Do
not
crawl
in
the
dust:
different
urls with
similar
text. ACM
Transactions
on
the
Web
(TWEB), 3(1),
p.3.
• https://www.seroundtable.com/google-‐grammar-‐seo-‐24571.html