SlideShare ist ein Scribd-Unternehmen logo
1 von 52
Downloaden Sie, um offline zu lesen
Crawling &
Indexing for
JavaScript
Heavy Sites
{TametheBots}
@davewsmart #brightonSEO
In the beginning ...
There was the web.
It served flat documents,
and the SEOs said it was
good
{TametheBots}
@davewsmart #brightonSEO
But then we wanted to do stuff ...
We wanted to buy things,
blog, chat on forums.
{TametheBots}
@davewsmart #brightonSEO
So dynamic server
side scripting was
born, and SEOs were
like “Cool!”
So then we wanted to interact ...
Right in the Browser!
We wanted to break away from
flat documents.
{TametheBots}
@davewsmart #brightonSEO
JavaScript was
born, and the SEOs
said ...
AAAAAAAAAAAA
HHHHHHHHHH
AAAAAAAARGH!
Which is kinda understandable
Search engines took a while to catch up
But the universe is different now*
*For some search engines!
{TametheBots}
@davewsmart #brightonSEO
Evergreen to the rescue!
Google used to render with an OLD browser.
In May ‘19 it got a BIG upgrade
It now runs up-to-date
Chromium!
{TametheBots}
@davewsmart #brightonSEO
Evergreen to the rescue!
Now executes modern JavaScript!
No more testing against Chrome 41!
{TametheBots}
@davewsmart #brightonSEO
Evergreen to the rescue!
What about Bing?
October ‘19, Evergreen too!
{TametheBots}
@davewsmart #brightonSEO
So it works just like my browser?
Nearly, but not quite …
The goals are different.
{TametheBots}
@davewsmart #brightonSEO
So it works just like my browser?
Humans want to see some pages.
Search engines want to gather
{TametheBots}
@davewsmart #brightonSEO
billions
How do they do that?
By accessing the web differently,
fetching only what they need, somewhat at
their own leisure.
{TametheBots}
@davewsmart #brightonSEO
How do they do that?
{TametheBots}
@davewsmart #brightonSEO
From: developers.google.com/search/docs/guides/javascript-seo-basics
Myths about the render queue
It’s not days, it’s mins.
Pretty much everything
goes through the WRS.
{TametheBots}
@davewsmart #brightonSEO
Some ground rules
A page needs a proper URL
Googlebot needs <a href=""> to get around.
{TametheBots}
@davewsmart #brightonSEO
Some ground rules
Non-200, or pages with noindex will not get
rendered.
{TametheBots}
@davewsmart #brightonSEO
Some ground rules
JavaScript, and API resources cannot be
blocked by robots.txt
(if you want them to work)
{TametheBots}
@davewsmart #brightonSEO
So, what’s the fuss about?
Some is about nothing.
But there are potential
lumps in
the custard.
{TametheBots}
@davewsmart #brightonSEO
Permission Denied!
Service Workers
Location
Notifications
...
{TametheBots}
@davewsmart #brightonSEO
Websockets? Nope!
Not designed to provide initial content.
Googlebot will tell you it does support, but
then fail anyway.
{TametheBots}
@davewsmart #brightonSEO
Web Workers? Kinda, Mostly!
Great for loading off heavy processes.
But … some unpredictable behaviour,
especially if it performs
fetches.
{TametheBots}
@davewsmart #brightonSEO
Solution?
Fail Gracefully!
Make sure the important
content loads anyway.
{TametheBots}
@davewsmart #brightonSEO
How to test?
● Mobile-Friendly Test
● Rich Results Test
● URL inspection tool (live)
{TametheBots}
@davewsmart #brightonSEO
We need to talk about caching
Google cache “aggressively”
They probably won’t listen to your
Cache-Control headers
{TametheBots}
@davewsmart #brightonSEO
We need to talk about caching
It is what allows their scale, and it’s a good
thing for you.
Less fetched = more budget
to fetch the good stuff.
{TametheBots}
@davewsmart #brightonSEO
But you need to work with it
Images, CSS, JavaScript & API crawls can
all be cached.
{TametheBots}
@davewsmart #brightonSEO
But you need to work with it
Some things it doesn’t matter for.
Some things it does!
{TametheBots}
@davewsmart #brightonSEO
JavaScript files
● Your site needs /app.js
● You update /app.js
{TametheBots}
@davewsmart #brightonSEO
JavaScript files
● Googlebot tries the new
page with the old /app.js
{TametheBots}
@davewsmart #brightonSEO
Who ya gonna call? Cache-busters!
● “Fingerprint” your files.
● /app.5787ee49.js
{TametheBots}
@davewsmart #brightonSEO
Who ya gonna call? Cache-busters!
Many frameworks & bundlers
can do this for you!
{TametheBots}
@davewsmart #brightonSEO
What about CSS?
● Less critical
● But can cause mobile
usability issues
{TametheBots}
@davewsmart #brightonSEO
What about CSS?
Same cache-busting solution
{TametheBots}
@davewsmart #brightonSEO
What about APIs?
If your content is loaded in via an API call,
this can be cached too.
You have a decision to make.
{TametheBots}
@davewsmart #brightonSEO
What about APIs?
Is freshness actually needed?
{TametheBots}
@davewsmart #brightonSEO
What about APIs?
Something like related
products? It might not, so
perhaps just let it cache.
{TametheBots}
@davewsmart #brightonSEO
What about when it matters?
If freshness does matter ...
Timestamp the call i.e:
/api/latest-news?ts=123456
POST not GET
{TametheBots}
@davewsmart #brightonSEO
How to test?
Live tools are made to bypass cache
Use URL Inspection Tool (not the live test)
Look at the rendered HTML
{TametheBots}
@davewsmart #brightonSEO
In your search console
{TametheBots}
@davewsmart #brightonSEO
Is the content up to date?
Is the content there?
Does the content look as up to date as the
last crawl date?
{TametheBots}
@davewsmart #brightonSEO
I’m a human, not a browser!
Ok, HTML isn’t always easy to
read. Click the copy button, then
go to that page.
{TametheBots}
@davewsmart #brightonSEO
I’m a human, not a browser!
Open devTools, right click on
<html> in the elements panel,
Select Edit as HTML
{TametheBots}
@davewsmart #brightonSEO
Hat tip to Oliver H.G. Mason (@ohgm), he's also
mentioned this solution, in a less clumsy way at:
https://www.contentkingapp.com/academy/url-in
spection-tool/#view-tested-page
I’m a human, not a browser!
Select all the code in the box, paste the
code from URL Inspection tool & Enter.
{TametheBots}
@davewsmart #brightonSEO
The page should now be
as Google rendered it.
Measure it in your log files!
No changes mean
gbot is hitting mostly
pages / robots.txt
{TametheBots}
@davewsmart #brightonSEO
Measure it in your log files!
I pushed a change,
with new filenames.
All the resources!
{TametheBots}
@davewsmart #brightonSEO
Some final thoughts
The dev team knows and cares about
users, they might not know about
googlebot.
{TametheBots}
@davewsmart #brightonSEO
Some final thoughts
Be their friendly guide,
not their nemesis.
{TametheBots}
@davewsmart #brightonSEO
Some final thoughts
I am not a JavaScript salesman.
Sometimes JavaScript isn’t the best way. If a pure HTML
solution is better & you can advocate for it, do!
{TametheBots}
@davewsmart #brightonSEO
Some final thoughts
TheOldWays™ are still valid!
{TametheBots}
@davewsmart #brightonSEO
Some final thoughts
The web is growing though!
JavaScript hasn’t killed the document web, it’s added to it.
As SEOs, we might be called on to support that.
{TametheBots}
@davewsmart #brightonSEO
Some great resources:
Google’s Dev Docs for JavaScript related SEO:
https://developers.google.com/search/docs/guides/javascript-seo-basics
Opening devTools:
https://developers.google.com/web/tools/chrome-devtools
SEO Mythbusters Video on JavaScript:
https://youtu.be/EZtCgrpa6ss
Martin Splitt’s JavaScript Hangouts, ask live questions!
Keep an eye out here:
https://developers.google.com/search/events#calendar
{TametheBots}
@davewsmart #brightonSEO
Bye Bye!
I’ve been Dave Smart, sorry about that :D
You can reach me at @davewsmart on
twitter, or find me at tamethebots.com
Ta-Ta For Now!
{TametheBots}
@davewsmart #brightonSEO

Weitere ähnliche Inhalte

Was ist angesagt?

Crawl Budget Conqueror - Take Control of Your Crawl Budget
Crawl Budget Conqueror - Take Control of Your Crawl BudgetCrawl Budget Conqueror - Take Control of Your Crawl Budget
Crawl Budget Conqueror - Take Control of Your Crawl BudgetCatalyst
 
BrightonSEO 2017 - SEO quick wins from a technical check
BrightonSEO 2017  - SEO quick wins from a technical checkBrightonSEO 2017  - SEO quick wins from a technical check
BrightonSEO 2017 - SEO quick wins from a technical checkChloe Bodard
 
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration Branded3
 
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...Sam Taylor
 
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
11 Advanced Uses of Screaming Frog Nov 2019 DMSS11 Advanced Uses of Screaming Frog Nov 2019 DMSS
11 Advanced Uses of Screaming Frog Nov 2019 DMSSOliver Brett
 
20 free SEO Tools you should be using - 20180829
20 free SEO Tools you should be using - 2018082920 free SEO Tools you should be using - 20180829
20 free SEO Tools you should be using - 20180829Christoph C. Cemper
 
BrightonSEO Takeaways September 2017
BrightonSEO Takeaways September 2017BrightonSEO Takeaways September 2017
BrightonSEO Takeaways September 2017Semrush
 
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!DanielCartland
 
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawlHow To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawlDeepCrawl
 
Methods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenMethods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenChris Green
 
Easier and faster tagging with Kermit
Easier and faster tagging with KermitEasier and faster tagging with Kermit
Easier and faster tagging with KermitAlban Gérôme
 
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...DeepCrawl
 
Are you there Page Experience? It's Me, DevTools.
Are you there Page Experience? It's Me, DevTools.Are you there Page Experience? It's Me, DevTools.
Are you there Page Experience? It's Me, DevTools.Rachel Anderson
 
MeasureFest July 2021 - Session Segmentation with Machine Learning
MeasureFest July 2021 - Session Segmentation with Machine LearningMeasureFest July 2021 - Session Segmentation with Machine Learning
MeasureFest July 2021 - Session Segmentation with Machine LearningRichard Lawrence
 
TechSEO Boost 2018: You Have Structured Data, Now What?
TechSEO Boost 2018: You Have Structured Data, Now What?TechSEO Boost 2018: You Have Structured Data, Now What?
TechSEO Boost 2018: You Have Structured Data, Now What?Catalyst
 
How To Get a 100% Lighthouse Performance Score
How To Get a 100% Lighthouse Performance Score How To Get a 100% Lighthouse Performance Score
How To Get a 100% Lighthouse Performance Score Polly Pospelova
 
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...Rachel Costello
 
Combatting Crawl Bloat & Pruning Your Content Effectively
Combatting Crawl Bloat & Pruning Your Content EffectivelyCombatting Crawl Bloat & Pruning Your Content Effectively
Combatting Crawl Bloat & Pruning Your Content EffectivelyCharlie Whitworth
 

Was ist angesagt? (18)

Crawl Budget Conqueror - Take Control of Your Crawl Budget
Crawl Budget Conqueror - Take Control of Your Crawl BudgetCrawl Budget Conqueror - Take Control of Your Crawl Budget
Crawl Budget Conqueror - Take Control of Your Crawl Budget
 
BrightonSEO 2017 - SEO quick wins from a technical check
BrightonSEO 2017  - SEO quick wins from a technical checkBrightonSEO 2017  - SEO quick wins from a technical check
BrightonSEO 2017 - SEO quick wins from a technical check
 
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration
SearchLeeds 2018 - Steve Chambers - Stickyeyes - How not to F**K up a Migration
 
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
Crawling & Indexing for eCommerce Sites - Sam Taylor, BrightonSEO (Crawling &...
 
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
11 Advanced Uses of Screaming Frog Nov 2019 DMSS11 Advanced Uses of Screaming Frog Nov 2019 DMSS
11 Advanced Uses of Screaming Frog Nov 2019 DMSS
 
20 free SEO Tools you should be using - 20180829
20 free SEO Tools you should be using - 2018082920 free SEO Tools you should be using - 20180829
20 free SEO Tools you should be using - 20180829
 
BrightonSEO Takeaways September 2017
BrightonSEO Takeaways September 2017BrightonSEO Takeaways September 2017
BrightonSEO Takeaways September 2017
 
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
We’ve analysed the SEO of over 100 eCom sites - this is what we’ve learned!
 
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawlHow To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
How To Tackle Enterprise Sites - Rachel Costello, Technical SEO, DeepCrawl
 
Methods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris greenMethods of Testing Internal Linking with Chris green
Methods of Testing Internal Linking with Chris green
 
Easier and faster tagging with Kermit
Easier and faster tagging with KermitEasier and faster tagging with Kermit
Easier and faster tagging with Kermit
 
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
Cut the Crap: Next Level Content Audits with Crawlers - Sam Marsden, SEO & Co...
 
Are you there Page Experience? It's Me, DevTools.
Are you there Page Experience? It's Me, DevTools.Are you there Page Experience? It's Me, DevTools.
Are you there Page Experience? It's Me, DevTools.
 
MeasureFest July 2021 - Session Segmentation with Machine Learning
MeasureFest July 2021 - Session Segmentation with Machine LearningMeasureFest July 2021 - Session Segmentation with Machine Learning
MeasureFest July 2021 - Session Segmentation with Machine Learning
 
TechSEO Boost 2018: You Have Structured Data, Now What?
TechSEO Boost 2018: You Have Structured Data, Now What?TechSEO Boost 2018: You Have Structured Data, Now What?
TechSEO Boost 2018: You Have Structured Data, Now What?
 
How To Get a 100% Lighthouse Performance Score
How To Get a 100% Lighthouse Performance Score How To Get a 100% Lighthouse Performance Score
How To Get a 100% Lighthouse Performance Score
 
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
Conflicting Website Signals & Confused Search Engines - Rachel Costello, Tech...
 
Combatting Crawl Bloat & Pruning Your Content Effectively
Combatting Crawl Bloat & Pruning Your Content EffectivelyCombatting Crawl Bloat & Pruning Your Content Effectively
Combatting Crawl Bloat & Pruning Your Content Effectively
 

Ähnlich wie Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stox
Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick StoxEverything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stox
Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stoxpatrickstox
 
Responsive Web Design Process
Responsive Web Design ProcessResponsive Web Design Process
Responsive Web Design ProcessSteve Fisher
 
Going native with html5 web components
Going native with html5 web componentsGoing native with html5 web components
Going native with html5 web componentsJames York
 
Stefan Judis "Did we(b development) lose the right direction?"
Stefan Judis "Did we(b development) lose the right direction?"Stefan Judis "Did we(b development) lose the right direction?"
Stefan Judis "Did we(b development) lose the right direction?"Fwdays
 
Stapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoStapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoChristian Heilmann
 
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments'
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments' SearchLove London | Dave Sottimano, 'Using Data to Win Arguments'
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments' Distilled
 
Debugging rendering problems at scale
Debugging rendering problems at scaleDebugging rendering problems at scale
Debugging rendering problems at scaleGiacomo Zecchini
 
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel CostelloThe Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel CostelloRachel Costello
 
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018patrickstox
 
Google Analytics
Google AnalyticsGoogle Analytics
Google AnalyticsRohan Dighe
 
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Solving Complex JavaScript Issues and Leveraging Semantic HTML5Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Solving Complex JavaScript Issues and Leveraging Semantic HTML5Hamlet Batista
 
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry White
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry WhiteGTM Clowns, fun and hacks - Search Elite - May 2017 Gerry White
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry WhiteGerry White
 
The road to professional web development
The road to professional web developmentThe road to professional web development
The road to professional web developmentChristian Heilmann
 
Successful Teams follow Standards
Successful Teams follow StandardsSuccessful Teams follow Standards
Successful Teams follow StandardsChristian Heilmann
 
Aucd ppt
Aucd pptAucd ppt
Aucd ppticidemo
 
Using Cool New Frameworks in (Mobile) Domino Apps
Using Cool New Frameworks in (Mobile) Domino AppsUsing Cool New Frameworks in (Mobile) Domino Apps
Using Cool New Frameworks in (Mobile) Domino AppsTeamstudio
 
Web Components - Rutgers Tech Meetup Fall 2014
Web Components - Rutgers Tech Meetup Fall 2014Web Components - Rutgers Tech Meetup Fall 2014
Web Components - Rutgers Tech Meetup Fall 2014Yair Aviner
 

Ähnlich wie Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021 (20)

Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stox
Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick StoxEverything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stox
Everything That Can Go Wrong Will Go Wrong - Tech SEO Boost 2017 - Patrick Stox
 
Responsive Web Design Process
Responsive Web Design ProcessResponsive Web Design Process
Responsive Web Design Process
 
Going native with html5 web components
Going native with html5 web componentsGoing native with html5 web components
Going native with html5 web components
 
Stefan Judis "Did we(b development) lose the right direction?"
Stefan Judis "Did we(b development) lose the right direction?"Stefan Judis "Did we(b development) lose the right direction?"
Stefan Judis "Did we(b development) lose the right direction?"
 
Stapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San FranciscoStapling and patching the web of now - ForwardJS3, San Francisco
Stapling and patching the web of now - ForwardJS3, San Francisco
 
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments'
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments' SearchLove London | Dave Sottimano, 'Using Data to Win Arguments'
SearchLove London | Dave Sottimano, 'Using Data to Win Arguments'
 
Debugging rendering problems at scale
Debugging rendering problems at scaleDebugging rendering problems at scale
Debugging rendering problems at scale
 
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel CostelloThe Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
The Future of Technical SEO | Women in Tech SEO 2019 | Rachel Costello
 
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018
Troubleshooting SEO for JS Frameworks - Patrick Stox - DTD 2018
 
Google Analytics
Google AnalyticsGoogle Analytics
Google Analytics
 
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Solving Complex JavaScript Issues and Leveraging Semantic HTML5Solving Complex JavaScript Issues and Leveraging Semantic HTML5
Solving Complex JavaScript Issues and Leveraging Semantic HTML5
 
SEO and Accessibility
SEO and AccessibilitySEO and Accessibility
SEO and Accessibility
 
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry White
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry WhiteGTM Clowns, fun and hacks - Search Elite - May 2017 Gerry White
GTM Clowns, fun and hacks - Search Elite - May 2017 Gerry White
 
The road to professional web development
The road to professional web developmentThe road to professional web development
The road to professional web development
 
Successful Teams follow Standards
Successful Teams follow StandardsSuccessful Teams follow Standards
Successful Teams follow Standards
 
Using HTML5 sensibly
Using HTML5 sensiblyUsing HTML5 sensibly
Using HTML5 sensibly
 
Aucd ppt
Aucd pptAucd ppt
Aucd ppt
 
Using Cool New Frameworks in (Mobile) Domino Apps
Using Cool New Frameworks in (Mobile) Domino AppsUsing Cool New Frameworks in (Mobile) Domino Apps
Using Cool New Frameworks in (Mobile) Domino Apps
 
Web Components - Rutgers Tech Meetup Fall 2014
Web Components - Rutgers Tech Meetup Fall 2014Web Components - Rutgers Tech Meetup Fall 2014
Web Components - Rutgers Tech Meetup Fall 2014
 
BDACA - Lecture6
BDACA - Lecture6BDACA - Lecture6
BDACA - Lecture6
 

Kürzlich hochgeladen

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 

Kürzlich hochgeladen (20)

Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 

Crawling & Indexing for JavaScript Heavy Sites brightonSEO 2021

  • 1. Crawling & Indexing for JavaScript Heavy Sites {TametheBots} @davewsmart #brightonSEO
  • 2. In the beginning ... There was the web. It served flat documents, and the SEOs said it was good {TametheBots} @davewsmart #brightonSEO
  • 3. But then we wanted to do stuff ... We wanted to buy things, blog, chat on forums. {TametheBots} @davewsmart #brightonSEO So dynamic server side scripting was born, and SEOs were like “Cool!”
  • 4. So then we wanted to interact ... Right in the Browser! We wanted to break away from flat documents. {TametheBots} @davewsmart #brightonSEO JavaScript was born, and the SEOs said ...
  • 6. Which is kinda understandable Search engines took a while to catch up But the universe is different now* *For some search engines! {TametheBots} @davewsmart #brightonSEO
  • 7. Evergreen to the rescue! Google used to render with an OLD browser. In May ‘19 it got a BIG upgrade It now runs up-to-date Chromium! {TametheBots} @davewsmart #brightonSEO
  • 8. Evergreen to the rescue! Now executes modern JavaScript! No more testing against Chrome 41! {TametheBots} @davewsmart #brightonSEO
  • 9. Evergreen to the rescue! What about Bing? October ‘19, Evergreen too! {TametheBots} @davewsmart #brightonSEO
  • 10. So it works just like my browser? Nearly, but not quite … The goals are different. {TametheBots} @davewsmart #brightonSEO
  • 11. So it works just like my browser? Humans want to see some pages. Search engines want to gather {TametheBots} @davewsmart #brightonSEO billions
  • 12. How do they do that? By accessing the web differently, fetching only what they need, somewhat at their own leisure. {TametheBots} @davewsmart #brightonSEO
  • 13. How do they do that? {TametheBots} @davewsmart #brightonSEO From: developers.google.com/search/docs/guides/javascript-seo-basics
  • 14. Myths about the render queue It’s not days, it’s mins. Pretty much everything goes through the WRS. {TametheBots} @davewsmart #brightonSEO
  • 15. Some ground rules A page needs a proper URL Googlebot needs <a href=""> to get around. {TametheBots} @davewsmart #brightonSEO
  • 16. Some ground rules Non-200, or pages with noindex will not get rendered. {TametheBots} @davewsmart #brightonSEO
  • 17. Some ground rules JavaScript, and API resources cannot be blocked by robots.txt (if you want them to work) {TametheBots} @davewsmart #brightonSEO
  • 18. So, what’s the fuss about? Some is about nothing. But there are potential lumps in the custard. {TametheBots} @davewsmart #brightonSEO
  • 20. Websockets? Nope! Not designed to provide initial content. Googlebot will tell you it does support, but then fail anyway. {TametheBots} @davewsmart #brightonSEO
  • 21. Web Workers? Kinda, Mostly! Great for loading off heavy processes. But … some unpredictable behaviour, especially if it performs fetches. {TametheBots} @davewsmart #brightonSEO
  • 22. Solution? Fail Gracefully! Make sure the important content loads anyway. {TametheBots} @davewsmart #brightonSEO
  • 23. How to test? ● Mobile-Friendly Test ● Rich Results Test ● URL inspection tool (live) {TametheBots} @davewsmart #brightonSEO
  • 24. We need to talk about caching Google cache “aggressively” They probably won’t listen to your Cache-Control headers {TametheBots} @davewsmart #brightonSEO
  • 25. We need to talk about caching It is what allows their scale, and it’s a good thing for you. Less fetched = more budget to fetch the good stuff. {TametheBots} @davewsmart #brightonSEO
  • 26. But you need to work with it Images, CSS, JavaScript & API crawls can all be cached. {TametheBots} @davewsmart #brightonSEO
  • 27. But you need to work with it Some things it doesn’t matter for. Some things it does! {TametheBots} @davewsmart #brightonSEO
  • 28. JavaScript files ● Your site needs /app.js ● You update /app.js {TametheBots} @davewsmart #brightonSEO
  • 29. JavaScript files ● Googlebot tries the new page with the old /app.js {TametheBots} @davewsmart #brightonSEO
  • 30. Who ya gonna call? Cache-busters! ● “Fingerprint” your files. ● /app.5787ee49.js {TametheBots} @davewsmart #brightonSEO
  • 31. Who ya gonna call? Cache-busters! Many frameworks & bundlers can do this for you! {TametheBots} @davewsmart #brightonSEO
  • 32. What about CSS? ● Less critical ● But can cause mobile usability issues {TametheBots} @davewsmart #brightonSEO
  • 33. What about CSS? Same cache-busting solution {TametheBots} @davewsmart #brightonSEO
  • 34. What about APIs? If your content is loaded in via an API call, this can be cached too. You have a decision to make. {TametheBots} @davewsmart #brightonSEO
  • 35. What about APIs? Is freshness actually needed? {TametheBots} @davewsmart #brightonSEO
  • 36. What about APIs? Something like related products? It might not, so perhaps just let it cache. {TametheBots} @davewsmart #brightonSEO
  • 37. What about when it matters? If freshness does matter ... Timestamp the call i.e: /api/latest-news?ts=123456 POST not GET {TametheBots} @davewsmart #brightonSEO
  • 38. How to test? Live tools are made to bypass cache Use URL Inspection Tool (not the live test) Look at the rendered HTML {TametheBots} @davewsmart #brightonSEO
  • 39. In your search console {TametheBots} @davewsmart #brightonSEO
  • 40. Is the content up to date? Is the content there? Does the content look as up to date as the last crawl date? {TametheBots} @davewsmart #brightonSEO
  • 41. I’m a human, not a browser! Ok, HTML isn’t always easy to read. Click the copy button, then go to that page. {TametheBots} @davewsmart #brightonSEO
  • 42. I’m a human, not a browser! Open devTools, right click on <html> in the elements panel, Select Edit as HTML {TametheBots} @davewsmart #brightonSEO Hat tip to Oliver H.G. Mason (@ohgm), he's also mentioned this solution, in a less clumsy way at: https://www.contentkingapp.com/academy/url-in spection-tool/#view-tested-page
  • 43. I’m a human, not a browser! Select all the code in the box, paste the code from URL Inspection tool & Enter. {TametheBots} @davewsmart #brightonSEO The page should now be as Google rendered it.
  • 44. Measure it in your log files! No changes mean gbot is hitting mostly pages / robots.txt {TametheBots} @davewsmart #brightonSEO
  • 45. Measure it in your log files! I pushed a change, with new filenames. All the resources! {TametheBots} @davewsmart #brightonSEO
  • 46. Some final thoughts The dev team knows and cares about users, they might not know about googlebot. {TametheBots} @davewsmart #brightonSEO
  • 47. Some final thoughts Be their friendly guide, not their nemesis. {TametheBots} @davewsmart #brightonSEO
  • 48. Some final thoughts I am not a JavaScript salesman. Sometimes JavaScript isn’t the best way. If a pure HTML solution is better & you can advocate for it, do! {TametheBots} @davewsmart #brightonSEO
  • 49. Some final thoughts TheOldWays™ are still valid! {TametheBots} @davewsmart #brightonSEO
  • 50. Some final thoughts The web is growing though! JavaScript hasn’t killed the document web, it’s added to it. As SEOs, we might be called on to support that. {TametheBots} @davewsmart #brightonSEO
  • 51. Some great resources: Google’s Dev Docs for JavaScript related SEO: https://developers.google.com/search/docs/guides/javascript-seo-basics Opening devTools: https://developers.google.com/web/tools/chrome-devtools SEO Mythbusters Video on JavaScript: https://youtu.be/EZtCgrpa6ss Martin Splitt’s JavaScript Hangouts, ask live questions! Keep an eye out here: https://developers.google.com/search/events#calendar {TametheBots} @davewsmart #brightonSEO
  • 52. Bye Bye! I’ve been Dave Smart, sorry about that :D You can reach me at @davewsmart on twitter, or find me at tamethebots.com Ta-Ta For Now! {TametheBots} @davewsmart #brightonSEO