!!

!

Every Second – in over 50,000 Categories

eBay Analytics

3

>50 TB/day new data

>150 PB/day >100 Trillion pairs of information

Millions of queries/day

>7500 business users & analysts

>50k chains of logic

24x7x365 99.98+% Availability

turning over a TB every second Structured/Unstructured

Near-Real-time

>100k data elements

Always online

Processed

Big

Detail

Designing for the Unknown >85% of analytical workload is NEW & Unknown The metrics you know are cheap The metrics you don’t know are expensive – but high in potential ROI Exploration & Testing are core pillars of an analytics-driven

organization

incremental storage

Volume

DATA

processing

change

incremental storage

Volume

Velocity DATA

structured

incremental storage

processing

change

Volume

Variety Velocity DATA

semi-structured un-structured

www.wallpapertimes.com

$’s per year in incremental revenue

Value > Cost

!   Data Growing Faster

•  Impact

16

questions later

structure later

($0.04/GB, $80/2TB)

single HDFS instances >50PB

Value > Cost

Data

Synonyms  derived  from  top  queries  in  item  query  clusters  texas  instruments  ba  ii  plus   /  ba  ii  plus  brighton  handbag   brighton  purse  lenovo  x200   thinkpad  x200  king  bedspread   king  coverlet  rockabilly  dress   swing  dress  1963  ford  falcon   63  falcon  jessica  simpson  hair  extensions   jessica  simpson  hairdo  

 Abbrevia7ons/acronym  derived  from  query  transi7ons  

stanford  ky   stanford  kentucky  dc  sub   dc  subwoofer  snowboard  helmet  l   snowboard  helmet  large  motorcycle  cam   motorcycle  camera  diamond  amp   diamond  amplifier  

Toys and Hobbies ATC > Artist trading card in ART ATC > Automatic Tool Change in Business and Industrial

Service

Offline Online

Editorial

Big Data Store NoSQL

Small Data

Code

Clients

Search

Selling

Others…

Behavioral Logs

Document Data

<3 milliseconds per query 1.2 billion queries per day 1,000’s of queries per second per machine

Human Judgment

German Compound Words •  German compound words can be arbitrarily created and extremely long

Adidastrainingsanzug (Adidas track suit) Rindfleischetikettierungsüberwachungsaufgabenübertragungsgesetz

(beef labeling regulation & delegation of supervision law) •  Syntactically, words can be combined and split in many ways. •  Some words shouldn’t be de-compounded.

beiden (both) – bei(at) den(the) •  Too many candidates for

Granitpflastersteine (granite paving stones) Granit(granite) pflastersteine(cobblestones) Granit(granite) pflaster(paving/band-aid) steine(stones)

•  Binding characters Hochzeitsschuhe (grammatically correct, 593 hits on ebay.de) Hochzeitschuhe (129 hits on ebay.de).

Data Warehouse + Behavioral

Data Warehouse

Semi-Structured SQL++

Structured SQL

Low End Enterprise-class System

Contextual-Complex Analytics Deep, Seasonal, Consumable Data Sets

Production Data Warehousing Large Concurrent User-base

Enterprise-class System

Unstructured Java/C++/Pig/Hive

Structure the Unstructured Detect Patterns

Commodity Hardware System

8+PB 60+PB 40+PB

Hadoop

Analyze & Report

Discover & Explore

Brian knows the satisfaction and importance of good search results, and his team is responsible for ensuring that the millions of queries entered onto the eBay website provide just that. The words “Did you mean…?” are incredibly meaningful to Brian as he combs through a universe of queries altered by synonyms, acronyms, attributes, and expansions. He’s been doing this sort of work since he joined eBay nine years ago. Brian has loved technology ever since junior high school, when he played the game “Lunar Lander” on a paper teletype before video games existed, and pulled pranks in the local Radio Shack. When Brian gets outside, he goes backpacking on Mount Whitney, enters triathlons, and walks on water (barefoot water skiing).