akamai.com

[Q4 2014 ]

• Prevalence of third-party content bots is increasing

• Used to collect data from other websites

• Poorly coded bots can impact site performance

• May resemble a denial of service (DoS) attack

• Secure web presence requires a comprehensive

understanding of how they affect performance

[Download the Q4 2014 Global DDoS Attack Report for

supporting data and analysis]

= bots, spiders & scrapers

2 / [The State of the Internet] / Security (Q4 2014)

= purposes of bots and scrapers

3 / [The State of the Internet] / Security (Q4 2014)

Akamai has seen bots and scrapers used for many purposes,

such as:

• Setting up fraudulent sites

• Reuse of consumer price indices

• Analysis of corporate financial statements

• Search and metasearch engines

• Data mashups

• Analysis of stock portfolios

• Competitive intelligence

• Location tracking

= example of a bot targeting a financial aggregator

4 / [The State of the Internet] / Security (Q4 2014)

In this example, the bot targeted the financial

aggregator to scrape a large amount of data quickly.

Figure 1: Bot targeting financial aggregator to scrape large amount of data quickly

= four categories of bots and scrapers

Figure 2: Desirability is based on how much the site owner wants to host the bot. Aggressiveness the rate of

requests from the bot and its impact on site availability.

5 / [The State of the Internet] / Security (Q4 2014)

• Mitigation techniques vary depending on bot classification

• Volume of requests can help determine platform

• The sequence and pages a bot targets can reveal intent

• The user-agent header can sometimes provide a unique and

identifiable user agent

• Additionally, Whois can sometimes identify bot owners

= triage and categorization

6 / [The State of the Internet] / Security (Q4 2014)

= corresponding mitigation strategies

Figure 3: Mitigation strategies are based on bot desirability and aggressiveness

7 / [The State of the Internet] / Security (Q4 2014)

• Bots and scrapers will continue to affect organizations

in all industries

• Every website should develop a strategy to address

and mitigate undesirable bot behavior

• Each organization will also need to a framework to evaluate

which bots to allow access to it site

• Companies will also need to analyze and modify security

policies to keep them current with the changing times

= moving forward

8 / [The State of the Internet] / Security (Q4 2014)

• Download the Q4 2014 State of the Internet Security Report

• The Q4 2014 report covers:

/ Analysis of DDoS attack trends

/ Breakdown of average Gbps/Mbps statistics

/ Year-over-year and quarter-by-quarter analysis

/ Types and frequency of application-layer attacks

/ Types and frequency of infrastructure attacks

/ Trends in attack frequency, size and sources

/ Where and when DDoSers launch attacks

/ Case study and analysis

= Q4 2014 global attack report

9 / [The State of the Internet] / Security (Q4 2014)

• StateoftheInternet.com, brought to you by Akamai,

serves as the home for content and information intended to

provide an informed view into online connectivity and

cybersecurity trends as well as related metrics, including

Internet connection speeds, broadband adoption, mobile

usage, outages, and cyber-attacks and threats.

• Visitors to www.stateoftheinternet.com can find current and

archived versions of Akamai’s State of the Internet

(Connectivity and Security) reports, the company’s data

visualizations, and other resources designed to put context

around the ever-changing Internet landscape.

= about stateoftheinternet.com

10 / [The State of the Internet] / Security (Q4 2014)