Cassini: A Simulation Framework for Evaluating
Designs for Sponsored Search Markets
Soam Acharya, Prabhakar Krishnamurthy, Ketan Deshpande, Tak W. Yan, Chi-Chao Chang
Yahoo! Inc.2821 Mission College Boulevard
Santa Clara, CA 95054
Topics
• Overview/Motivation
• Requirements
• Architecture
• Methodology
• Applications/Results
• Future Directions
Cassini Overview
• What is it?– Discrete Event Simulation System
• support simulations of different marketplace designs, policies and technologies
• Provide rapid assessment of revenue/search (RPS), click-through-rate (CTR) and cost-per-click (CPC) impact
– Compare % change vs. a baseline
– Other metrics calculated depend on specific
experiment
Contribution
• General purpose sponsored search auction simulator– Built-in support
• Auction structure, ranking, and payment policies, budgets
• Advertisers, campaigns, bids• User click model• Search events
– Extensible, modular architecture
Motivation
• Alternative: live tests• Problems
– Expensive– Time consuming
• Preparation, SLAs
• Must run long enough for statistical significance
– Incomplete• Not possible to explore all aspects of marketplace
– Eg. advertiser long term effects
and
Topics
• Overview/Motivation
• Requirements
• Architecture
• Methodology
• Applications/Results
• Future Directions
Requirements for a Simulation Framework
• Mimic Sponsored Search Auction mechanisms– Ranking, budgeting, pricing
• User behavior– Click model– Use actual log traces as input
• Advertiser behavior– Advertiser action controls
• Performance– Process large quantities of data– Need to complete large numbers of runs quickly
• Others:– Extensible– Support for market mechanisms
Overall Architecture
External Ad Ranker
Ad Server
Ad Information
Ad Information
BudgetFiltering
Ranking
Pricing
ClickGenerator
Budget & Advertiser
ManagementYSMImpression &Click Logs
OfflineClick ModelGeneration
QueryTrace
MetricComputation
Simulation LogOutput
Click Model
Output DB
Topics
• Overview/Motivation
• Requirements
• Architecture
• Methodology
• Applications/Results
• Future Directions
Methodological Issues
• Sponsored search auctions are complex– Advertisers adapt to events and outcomes– Users adapt to market structure and policies and
auction outcomes– Advertiser budgets introduce dependencies
between markets
• Input and event space is multidimensional with interactions– Simulation of joint distribution can be too time
consuming
Approach
• Simplifying assumptions in current version of Cassini– Advertiser actions are at equilibrium– Static user click model– Each auction is independent
• Except when budget management designs are being evaluated
• Approach– Take samples of actual historical search traffic– Focus on only the most significant sources of variation in traffic
• Week-end vs week-day traffic• Samples from different times in history
• Sampling– Using full day traffic for simulation is infeasible– Random sample of searches works well except with budgets
• Best option: Ignore budgets unless it is the focus of experimentation• A very small proportion of traffic can provide reasonably good estimates of RPS
(revenue per search)• With budgets, estimates are biased upwards
– Otherwise, reasonably small (almost) closed micro-markets can be used
Micro-market Sampling
• A micro-market is a collection of accounts and keywords such thatAll spend due to these accounts and
keywords occurs within the collection
• Run simulations with multiple micro-markets
Topics
• Overview/Motivation
• Requirements
• Architecture
• Methodology
• Applications/Results
• Future Directions
Applications of Cassini at Yahoo!
• Screened candidates of ranking algorithms for live testing
• Evaluated different design options for matching algorithms
• Estimated the potential of budget optimization
• Others
Evaluation of Matching Algorithms
Impact of Matching Methods on
-1%
0%
1%
2%
3%
4%
5%
6%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% Ads Dropped
% CTR Change from Baseline
Method 1
Method 2
Method 3
Topics
• Overview/Motivation
• Requirements
• Architecture
• Methodology
• Applications/Results
• Future Directions
Cassini – Future Directions
• Advertiser bidding agent– Support automated, adaptive bidding agent – Allow different bidding strategies to be implemented– Bidding languages
• Scale to full traffic• Open interface (other groups within Yahoo)• Self-service architecture
Related Work
• Simulations of sponsored search auction designs– Feng, Bhargava, Pennock; Kitts, LeBlanc
• Simulations of other types of auctions– Yankee Auctions (Bapna, Goes, Gupta); FCC
Spectrum Auctions (Csirik et al)
• Bidding Agents, Bots– Wurman, Wellman, and Walsh; Jennings; Powell
Design Decisions
• Query driven metaphor
• Allow collaboration:– leverage models from other groups
• Multiple iterations for the same set of inputs
• Maintain as much state as possible:– new metrics easily computed
• Generate as much state as possible– Copious quantities of log files– Turn off for performance
Advertiser Actions
• Static• Adjust bids, budgets• Target
– Individual advertisers– Groups
• Predefined• Randomly pick a certain percentage of
advertisers within each group– Eg. Select 25% of advertisers in cluster E
Cassini Implementation Notes
• # of lines:– 22K lines of C++, Perl, shell, SQL– 56K lines of C++ libraries
• Performance– Single instance per machine– 80K unique queries over one day
• Several million queries• Order of hours
– Capacity: memory bound– Speed: Disk I/O bound
Advertiser Actions Examples
Bid Increase % Budget Increase %Cluster ProportionMean Std Dev Mean Std DevA 85 18 4 10 4B 85 8 2 5 2C 85 5 2 10 4D 85 5 2 5 4E 85 5 2 5 2F 85 0 2 10 4G 85 5 2 5 2H 85 5 2 0 2I 85 18 4 10 4J 85 24 5 5 2K 85 5 2 0 2UNK 40 10 4 5 2
Bid Increase % Budget Increase %Cluster ProportionMean Std Dev Mean Std DevA 100 30 0 10 0B 100 15 0 5 0C 100 10 0 10 0D 100 10 0 5 0E 100 10 0 5 0F 100 0 0 10 0G 100 10 0 5 0H 100 10 0 0 0I 100 30 0 10 0J 100 30 0 5 0K 100 10 0 0 0UNK 100 10 0 5 0
Setting 1
Setting 2
Simulation Setup
• Inputs– Bid Landscape
• Accounts, ads, bids– Other
• Budgets– Advertiser actions
• Bid and budget changes (stochastic)
• Events– Search– Clicks– Advertiser actions
• Calibration– User click model
• What do we want to use simulation for?
Design Exploration
• Use reference simulation run to verify “invariants”: data and parameters– Similar to production set-up whose performance is well-
understood– Compare to actual performance over a number of data
samples
• Comparison to bucket tests
Future Directions
• Open interfaces for click model, ranking algo, matching algo, etc.– New click models
• Self-service – ease of use, user interface, job management– Leverage work from Yahoo Pipes, other log/data processing
groups.
• Better analysis support – pre- and post- simulation analysis– See above
Top Related