Combatting Crawl Bloat & Pruning Your Content Effectively

Charlie WhitworthSEO Director

@WhitworthSEO http://www.slideshare.net/whitworthseo

Banc

what will we cover?

bancmedia.-com

Crawl Bloat & Pruning

@Whitworth-SEO

beat the bloat

Crawl Bloat

Making search engines work too hard

and not taking full advantage of your

crawl budget@Whitworth-

SEObancmedia.-

com

crawl definitions

=Crawl Budget

bancmedia.-com

@Whitworth-SEO

Crawl Rate & Crawl Demand

crawl rate (limit) : maximum fetching rate

Search Console Limits

bancmedia.-com@WhitworthSEO

Website/Server Health

crawl demand

Popularity (Traffic) Staleness

@Whitworth-SEO

bancmedia.-com

How much does Google want to see your URLs?

Crawl budget – determined by rate & de-mand

What Google can & wants to crawl

@WhitworthSEO bancmedia.-com

• Empty pages• No valued added• Dupe

• Soft 404• SPAM• Hacked content

crawl budget

“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-

generate pages based on URL parameters”

Gary Ilyes

bancmedia.-com

@Whitworth-SEO

how to identify crawl bloat

bancmedia.-com

@Whitworth-SEO

why is this a problem?


Don’t work them too hard

Search engine bots hate crawling rubbish

why is this a problem?

Engagement metrics are key

especially now we’re “mobile first”

@Whitworth-SEO

bancmedia.-com

examples of rubbish: search result URLs

/search/bancmedia.-

com@WhitworthSEO

/sessionkey=

/search-results?q=

examples of rubbish: paginated URLs

bancmedia.-com

rel=next/prevnoindex

@Whitworth-SEO

examples of rubbish: faceted navigations

json-ld

disallow parametersbancmedia.-

com@Whitworth-

SEO

noindex

who’s seen this?

cut the crap

Get Rid Of It

@Whitworth-SEO

bancmedia.-com

noindex

<meta name=“robots" content=“noindex">Won’t deal with crawl bloat in the short term

Still an effective tacticbancmedia.-

com@WhitworthSEO

url parameters

@WhitworthSEO

Powerful tool when used properly

No need for developer sup-port

bancmedia.-com

xml sitemaps

Redirected URLs

Noindexed URLsPoor quality pages


robots.txt


Exercise Caution!

Powerful crawl bloat tool when used effect-ively

robots.txt

Blocking Crawl Paths – Stems Authority Flow


Google needs to see tags to honour them!

google’s hint

Google gives you all you need to monitor your results

All these features are your friends


crawl stats

Great way to gauge your efforts

@Whitworth-SEO

bancmedia.-com


Pruning

the prune

pruning – the fun bit

SEO Housekeep-ing

Now you have your crawl under control, show off your best bits


pruning – why?

• Spinning

• Excessive blog-ging

• Sparse articles@WhitworthSEO bancmedia.-

com

• Over optimisa-tion

• Saturation

stage one – content audit/content is king

Content auditing still essential nonethe-less


SEO’s most annoying cliché

stage two: the prune

410 – Gone


For pages you want gone forever



Won’t help crawl bloat short term

Will disappear eventu-ally*


Crawl Bloat

Authority

Indexa-tion



NoIndex

Will be crawled less over time ensures a high qual-ity SERP

For duplicate content, cannibalisation and API style pages


stage two: canonical

For duplicate content, cannibalisation and API style pages


canonical

stage three: content repurposing/enhance-ment


EvergreenSeasonalLong formResearch

search quality guidelines


Search Quality Evaluator Guidelines

stage four: the finishing touch

Link Profile Ana-lysis

Links pointing at ex-cluded pagesUsual disavow re-

quest


Don’t ruin all your hard work with a crap link profile

crawl bloat and pruning checklist

Does your site have crawl bloat?Choose the appropriate fixes for your sitePrune your remaining content

Continually improve and enhance@WhitworthSEO bancmedia.-

com

final tips: thanks for coming


Thanks for Listen-ing!

2 Tips

Combatting Crawl Bloat & Pruning Your Content Effectively

Marketing

Transcript of Combatting Crawl Bloat & Pruning Your Content Effectively