RTBkit 2.0 Roadmap Preview

Open-Source RTB for Everyone

2.0 Roadmap Preview

Copyright © 2014 Datacratic Inc. All rights reserved.

Overview

● About the RTBkit Project● How you can participate● RTBkit Today – 1.0● What’s New in 2.0● 2.0 Roadmap● Resources


About the project

A Little History

• RTBKit was created by machine-learning and digital marketing company Datacratic

• Code base evolved from running RTB in production from 2011-present

• Open sourced in Feb. 2013, with ongoing support from Datacratic

• Apache-style governance started Jan. 2014


Participation and Governance• Apache-style governance

• BDNFL - Benevolent Dictator Not for Life

• Councillors• Committers

• Outside contributions welcome• Github pull request workflow --

committers review and merge• Contributor guidelines• Users can become Contributors• Contributors can become

Committers -- currently two outside Committers


Development Support

Development Support● Getting Started Guide● Developer Tutorials● Working integration test system

o mock Exchange and Ad Servero fixed-price Bidding Agent

● Example Code● Documentation● Ubuntu AMI● Google Group support● Pull request review and support

© Datacratic Inc. 2014 All Rights Reserved.

Support, Community, Adoption• Free support from the

community and Datacratic• 250+ active developers• 35 committers, 11 outside of

Datacratic• Users in 25+ countries• ~25 installations in prod: N.

America, Germany, France, Russia, Argentina, China, India

• Datacratic supports development, code review, governance and evolution

• Participation and contributions from Rubicon Project


Addresses the RTB Value Problem

SYSTEM SELECTION VALUE

Scale

Speed

Distribution

Show user an ad?

What ad?

What is it worth?

What should I pay?Reliability

General/Technical Specific/Business

Provided by RTBkit Customized by User


RTB Technology LandscapeSystem Pros Cons Degree of

Difficulty

Exchange / DSP UI

Easy to get started ● Manual, hard to scale● Lack of control over bidding

strategy and data

Low

Intermediate Hosted Bidding

● More control over bidding strategy and use of data

● Don't have to do Ops

Strategy and use of data mediated by vendor and product features

Medium

● Core systems and integration problems solved

● Benefit from community● Flexible customization● Flexible use of data

● Requires significant development● Responsible for operations of a

high-volume distributed system

High

Roll Your Own Bidder

Full control of all aspects of the system

Solve all the hard systems problems yourself first, then build the rest of the system you need.

Highest


1.0 Overview


Architectural OverviewRTBkit Core

● Router● Banker● Post Auction Service● Service Monitor● Agent Configuration Service

Plugins● Exchange Connectors● AdServer Connector● Bidding Agent● Augmenter● Logger


Bidder Core Advantages● Core working bidder system● High-performance real-time components● Multiple data center support● Reliable global banker● Guaranteed response time to exchanges● High-performance parsing, routing, filtering,

logging and monitoring● Arbitrary use of custom business logic and

your data


Inventory Integration● Ships with 9 Exchange Connectors

o Rubicon Projecto AdXo FBXo AppNexuso Nexageo MoPubo GumGumo BidSwitch

● OpenRTB 2.1 compatible


Bid Request Lifecycle

RTBkitRouter

Exchange Connector

Static Filters

Dynamic Filters

Augmenter

Post Auction Service

Ad Server

Conversion Source

Bidding Agents

Ad Server Connector

Exchanges


RTBkit Data FlowsFive asynchronous data

flows flow through the Router:

● Bid request processing

● Banking updates

● Event Matching

● Notifying Bidding Agents of Events

● Filtering and Bidding Agent configuration

RTBkitRouter

Exchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter


Agent Config

Exchanges


Banker Responsibilities and DesignBudget Management

● Single source of truth for budget● Authorizes and caps spendingDesigned for Reliability

● Designed for high-latency, low-bandwidth ● Insulates banker state by using "shadow

account" bookkeeping● Totals always go up -- you can always

reason about the relative timing of entries● Updates global state once per minute● Atomic, idempotent persistenceCurrency Support

● Strongly typed 64-bit int Currency, won't allow cross-currency conversions

● Automatic scaling conversions (e.g. CPM to micro-dollars)


Account Hierarchies & Spend Tracking● Spend accumulates from Children to Parents● Temporary bookkeeping for bids happens in

shadow accounts in separate processes● Shadows sync once per second

o Router shadow - tracks budget committed to pending bids

o PAS shadow - tracks budget to debit on Wins and to credit on Losses

BudgetA

SpendA:B

Master Banker

SpendA:C

Shadow Spend

A:B

Router Slave Banker

Shadow Spend

A:C

Shadow Spend

A:C

PAS Slave Banker

Shadow Spend

A:B


Monitoring and Operations Tools● Extensive instrumentation● Lock-free, high-performance Carbon

logging library, with tunable sampling rate, one-second granularity and useful aggregate functions

● Can add any custom metrics● Operational dashboard included


Filter Priorities and PerformanceFilter Priorities and Performance

● Execution order optimized for performance● Broad, inexpensive filters run first● Expensive filters run later (or not at all!)● Only fast filters on real-time thread

● Static filters support batch processing

● Filter matching tests match and retrieves eligible creatives in one pass

FormatLocation

ExchangeLanguage

Creative

Agent

Exchange

Location

Language

Host

URL

Segments

Hour of Week

Fold Position

User Partition

Agent

Bid Request

Bid RequestBid

RequestBid Request


Augmenter Implementation● Provides thread pool● Enforces 5ms timeout● Sync and async support● Ships with Redis Augmenter

● Separate config for each Bidding Agent● Can subscribe to other RTBkit data

streams to write datao e.g. - frequency cap Augmenter

subscribes to PAS MATCHEDWINs

Router

Bid Request

Wicked Fast DB TM

Thread Pool

Augmenter Impl.

Augmenter


Data Sink Callback

Augmented Bid

Request


Custom Bidding Agent Implementation● C++ or JavaScript● Programmatic configuration● Custom bidding logic based on

o bid attributeso a custom Win Cost Model to adjust

for desired margin and data costs● Custom Pacing logic supportedCustom Bidding

Agent

ConfigurationBidding Logic

Win Cost ModelPacing

Currency Helpers

Router

Bid Request1Bid Response

2

onWinonLossonNoBudgetonTooLateonDroppedBidonInvalidBid

3


onImpressiononClickonVisit 4


2.0 Roadmap Preview


The Big Story in 2.0 … Plugins!● Why Plugins? We want to support

an ecosystem of revenue opportunities on RTBkit.

● Design supports this goal● Plugins identified by URL● Can have multiple versions● Support multiple RTBkit versions● Support multiple RTBkit

installations● Requests can be authenticated● Vendors of public plugins can set

up direct billing


Public Plugin Ecosystem

● Major goal of RTBkit 2.0● RTBkit users can leverage

third-party Plugins

● Public Plugin requirements:● Multitenant – one plugin can

serve many RTBkit instances● Multi-location – one logical plugin

can be deployed in multiple locations

● Authentication and Security● Per-call metering for billing● Support multiple RTBkit versions

RTBkit

Plugin

Plugin

Plugin

Plugin

Plugin

Plugin Plugin

Plugin

Plugin Plugin

Plugin

PluginPlugin

Plugin

Plugin

RTBkit

RTBkit

Plugin

Plugin


Plugins – Ecosystem Business Use Cases

● RTBkit users benefit and Vendors benefit:● Bidder Plugins – campaign

management and bidding logic● Bid Optimization● First-party Data in bidding logic● Per-bid bid price modifier● First-party Data for attribution● Third-party Data● Custom Attribution● Offline Analytics● Real-time Analytics● Real-time Audience Modeling


Private Plugins

● Live in same network as RTBkit or reachable across firewall

● Simple Private Plugins● Single HTTP endpoint responding to

Configuration and Plugin API calls● Configuration calls hardcoded

● Scalable Private Plugins● Multiple Plugin Workers behind load

balancer respond to Plugin API calls● Configuration calls hardcoded


Public Plugins

● Plugin providers can make Plugins available to multiple RTBkit installations

● Provider gives clients authentication token and configures firewall

● Configuration API in the public Internet is zone-aware and provides access to Plugin endpoints

● Supports Plugin marketplace


RTBkit 2.0 Plugin-based Architecture● All extension points redefined as

passive HTTP plugins:● OO Services● C++ Any Language You Want™

● Standard approaches to:● Configuration / Discovery● Deployment● Security● Compatibility

● Public and Private● Even RTBkit core redefined as a

family of APIs


Plugin Endpoints – Config & Worker● Plugins expose /config endpoint● Configuration calls are HTTPS● Configuration brokers keys and

accepts IPs to allow traffic from● Configuration uniquely identifies

and versions a Plugin● RTBkit Core uses own

Configuration Plugin to call available Plugins /config, retrieve their URLs

● Plugins mark themselves as dead

RTBkit

Configuration Plugin

Plugin

/conf /do_work


Plugin APIs● Configuration

● Returns list of Live and Dead Plugins

● Bidder● Receives stream of Bid Requests● Returns stream of Bid Responses● Multiple logical bidders behind one

endpoint disambiguated with ext field

● Analytics● Receives stream of data for use in

offline reporting systems● Event Id● “Channel” for the type of Event● list of subscribed Plugins

● Data● Plugin API implementation of

Augmenter

RTBkit 2.0

Bidder

Analytics Config

Data


Plugin Configurations● Plugins configure RTBkit 2.0 Core

using Configurations● Id, Version, Type● URLs● Rate

● Bidder Plugins● Is Banking● Pre-processing Subscriptions – such

as Filtering● Data Plugin Bidder is using

● Banking Plugins● Budget● Bid Rate

● Analytics● Channels

RTBkit 2.0

Bidder

Analytics Plugin X

Banker


Multi-Data Center Support & Routing

● RTBkit instances know their location – tied to a region

● By default only connect to Plugins in the same region

● RTBkit ignores Plugins not responding to heartbeat calls

● Standard list of locations used by RTBkit and Plugins

● Timeouts and failures handled by type of Plugin

West East


RTBkit and Plugin Security

● HTTPS only used when calling the Configuration API

● All HTTP requests signed with a HMAC key returned by the Configuration API● Supports rejecting forged requests

● RTBkit periodically calls Plugin Configuration endpoint to update IPs that will call it

● Plugins should reverse proxy to reduce attack surface


Packaging and Compatibility

● Test suite to ensure:● Plugins compatible with RTBkit 2.0● Changes to RTBkit 2.0 compatible

with connected Plugins

● Plugins speak OpenRTB 2.2● RTBkit 2.0 package options:

● Docker● Binary distribution of executables,

headers, libs

● Backwards compatible with RTBkit 1.0 – Router will support 1.0 or 2.0 interfaces via dependency injection

RTBkit 2.0

Plugin 2

Plugin 3 Plugin 1

Plugin 4


RTBkit 2.0 Core APIs● Control Blocks

● Pre-packaged sets of RTBkit services required to control an RTBkit installation

● Lower scaling requirements● One required per data center● REST API

● Worker Blocks● Pre-packaged set of RTBkit services

that require high throughput and may need to be horizontally scaled

● High scaling requirements● At least one per data center, but may

deploy many for scaling or redundancy


2.0 Roadmap and Progress

Already Delivered in 2014● HTTP Bidding Agent APIs –

Precursor to full 2.0 API

Up Next – Q4 2014● First API open source code

releases -- written in Go not C++● RTBkit updates to be compatible

with new APIs

Up Next – Q1 2015● Data and Logger HTTP APIs for

Data and Analytics Plugins© Datacratic Inc. 2014 All Rights Reserved.

RTBkit Resources

Links● http://github.com/rtbkit● http://rtbkit.org● https://groups.google.com/a/rtbkit.

org/forum/#!forum/discuss

Developers Getting Started Guide● https://github.com/rtbkit/rtbkit/wiki/

Getting-Started


http://github.com/rtbkit

http://rtbkit.org/

https://groups.google.com/a/rtbkit.org/forum/

https://groups.google.com/a/rtbkit.org/forum/

https://github.com/rtbkit/rtbkit/wiki/Getting-Started

https://github.com/rtbkit/rtbkit/wiki/Getting-Started

RTBkit & Rubicon Project Deliver Scalable Ad Tech Infrastructure & Extensible Bidding Stack

• Rubicon Project is a leading technology company automating the buying and selling of advertising.

• Rubicon Project & RTBkit have combined forces to deliver a scalable ad tech infrastructure, and makes it easy for media buyers to deploy their own proprietary bidding stack with the open and extensible architecture of RTBkit.

• The OpenDSP team is a contributor to the RTBkit open source project and active member of the RTBkit community.

Join the Alpha Program

There are limited seats available in the alpha program. Contact the OpenDSP team at [email protected] to get started.

About Datacratic

Datacratic is an enterprise software company that develops machine learning and artificial intelligence technology, which enables real-time machine based decisioning to be deployed into a wide range of applications.

Datacratic’s products are currently optimized for:• Data Management Platforms (DMPs)• Demand Side Platforms (DSPs)• Agency Trading Desks (ATDs)• E-Commerce web sites• Others in the real-time marketing ecosystem.

Mark Weiss @marksweiss

Head of Customer Solutions at Datacratic

www.datacratic.com

opensource.datacratic.com/

We're hiring! datacratic.com/site/careersCopyright © 2014 Datacratic Inc. All rights reserved.

http://www.datacratic.com/

http://opensource.datacratic.com/

http://datacratic.com/site/careers

Host a Meetup!

RTBkit 2.0 Roadmap Preview

Technology

Transcript of RTBkit 2.0 Roadmap Preview