Charlie WhitworthSEO Director

@WhitworthSEO http://www.slideshare.net/whitworthseo

Banc

what will we cover?

bancmedia.-com

Crawl Bloat & Pruning

@Whitworth-SEO

beat the bloat

Crawl Bloat

Making search engines work too hard

and not taking full advantage of your

crawl budget@Whitworth-

SEObancmedia.-

com

crawl definitions

=Crawl Budget

bancmedia.-com

@Whitworth-SEO

Crawl Rate & Crawl Demand

crawl rate (limit) : maximum fetching rate

Search Console Limits

bancmedia.-com@WhitworthSEO

Website/Server Health

crawl demand

Popularity (Traffic) Staleness

@Whitworth-SEO

bancmedia.-com

How much does Google want to see your URLs?

Crawl budget – determined by rate & de-mand

What Google can & wants to crawl

@WhitworthSEO bancmedia.-com

• Empty pages• No valued added• Dupe

• Soft 404• SPAM• Hacked content

crawl budget

“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-

generate pages based on URL parameters”

Gary Ilyes 

bancmedia.-com

@Whitworth-SEO

how to identify crawl bloat

bancmedia.-com

@Whitworth-SEO

how to identify crawl bloat

bancmedia.-com

@Whitworth-SEO

why is this a problem?

bancmedia.-com@WhitworthSEO

Don’t work them too hard

Search engine bots hate crawling rubbish

why is this a problem?

Engagement metrics are key

especially now we’re “mobile first”

@Whitworth-SEO

bancmedia.-com

examples of rubbish: search result URLs

/search/bancmedia.-

com@WhitworthSEO

/sessionkey=

/search-results?q=

examples of rubbish: paginated URLs

bancmedia.-com

rel=next/prevnoindex

@Whitworth-SEO

examples of rubbish: faceted navigations

json-ld

disallow parametersbancmedia.-

com@Whitworth-

SEO

noindex

who’s seen this?

cut the crap

Get Rid Of It

@Whitworth-SEO

bancmedia.-com

plan

noindex

<meta name=“robots" content=“noindex">Won’t deal with crawl bloat in the short term

Still an effective tacticbancmedia.-

com@WhitworthSEO

url parameters

@WhitworthSEO

Powerful tool when used properly

No need for developer sup-port

bancmedia.-com

xml sitemaps

Redirected URLs

Noindexed URLsPoor quality pages

@WhitworthSEO bancmedia.-com

robots.txt

bancmedia.-com@WhitworthSEO

Exercise Caution!

Powerful crawl bloat tool when used effect-ively

robots.txt

Blocking Crawl Paths – Stems Authority Flow

bancmedia.-com@WhitworthSEO

Google needs to see tags to honour them!

google’s hint

Google gives you all you need to monitor your results

All these features are your friends

@WhitworthSEO bancmedia.-com

crawl stats

Great way to gauge your efforts

@Whitworth-SEO

bancmedia.-com

bancmedia.-com@WhitworthSEO

Pruning

the prune

pruning – the fun bit

SEO Housekeep-ing

Now you have your crawl under control, show off your best bits

bancmedia.-com@WhitworthSEO

pruning – why?

• Spinning

• Excessive blog-ging

• Sparse articles@WhitworthSEO bancmedia.-

com

• Over optimisa-tion

• Saturation

stage one – content audit/content is king

Content auditing still essential nonethe-less

bancmedia.-com@WhitworthSEO

SEO’s most annoying cliché

stage two: the prune

410 – Gone

@WhitworthSEO bancmedia.-com

For pages you want gone forever

stage two: the prune

bancmedia.-com@WhitworthSEO

Won’t help crawl bloat short term

Will disappear eventu-ally*

stage two: the prune

Crawl Bloat

Authority

Indexa-tion

@WhitworthSEO bancmedia.-com

stage two: the prune

NoIndex

Will be crawled less over time ensures a high qual-ity SERP

For duplicate content, cannibalisation and API style pages

@WhitworthSEO bancmedia.-com

stage two: canonical

For duplicate content, cannibalisation and API style pages

@WhitworthSEO bancmedia.-com

canonical

stage three: content repurposing/enhance-ment

@WhitworthSEO bancmedia.-com

EvergreenSeasonalLong formResearch

search quality guidelines

@WhitworthSEO bancmedia.-com

Search Quality Evaluator Guidelines

stage four: the finishing touch

Link Profile Ana-lysis

Links pointing at ex-cluded pagesUsual disavow re-

quest

@WhitworthSEO bancmedia.-com

Don’t ruin all your hard work with a crap link profile

crawl bloat and pruning checklist

Does your site have crawl bloat?Choose the appropriate fixes for your sitePrune your remaining content

Continually improve and enhance@WhitworthSEO bancmedia.-

com

final tips: thanks for coming

@WhitworthSEO bancmedia.-com

Thanks for Listen-ing!

2 Tips