3/18/15 Billing&Payments Eng Meetup II - Payments Processing in the Cloud
6/18/14 Billing & Payments Engineering Meetup I
-
Upload
mathieu-chauvin -
Category
Presentations & Public Speaking
-
view
69.724 -
download
2
Transcript of 6/18/14 Billing & Payments Engineering Meetup I
Billing & PaymentsEngineering Meetup
June 18th, 2014
• Mathieu Chauvin – NetflixValidating Payment Information
• Taylor Wicksell – NetflixFlexible Billing Integrations Using Events
• Jean-Denis Greze – DropboxMistakes (and Wins) Building Payments at Dropbox
• Anthony Zacharakis – LumosityBilling Migrations: Maintaining Your (Data) Sanity
• Alec Holmes – SquareScaling Merchant Payouts at Square
=== Break ===• Paul Huang – SurveyMonkey
Billing Globalization and Auto Renewal Optimization• Emmanuel Cron – Google Wallet
Moving from SE to Host Card Emulation• Feifeng Yang / Michael Chen – Electronic Arts
Ecommerce at EA• Krishnan Sridhar – LinkedIn
Real-Time Analytics and Smart Routing
Validating Payment Information
Mathieu Chauvin@matchauvin – linkedin/matchauvin
Signing-Up for Netflix
• Subscription Based• 30 Days Free Trial• Payment Information
Collection
• Pro Forma• Luhn Check (Mod10)• Card Type vs Card# RegEx• Authorization / Reversal
Payment Validation 101
But Wait! You Can Do More!
Authorization Amount
• $0.00, $1.00, or Full Price Authorization• Don’t Forget the Reversal!• Constraints by Country or by Card Type
Response Types
• Hard Declines• Soft Declines
MCC
• What is an MCC?• Figure out the Best MCC Fit
BINs
• What is a BIN?• Card Types: Credit, Debit, Prepaids• What About Non-Reloadable Prepaid Cards?
Forking Traffic
• Do You Use Several Payment Processors?• Why Don’t You Test Which One Performs
Better?
Use Cases
• Sign-Up• Updating Payment Information• Returning Customer
Country Specific Behavior
• Adapt To Your Market
The Feedback Loop
• A/B Tests• Analyze, Project & Evaluate Risks• Test Again
Conclusion
• Try Everything• Build Your Application Dynamically• The Feedback Loop!
Thank You!
Flexible Billing Integrations Using Events
Taylor Wicksell
The Billing Team
• Signups/Cancellations• Recurring Billing• Price and Tax Calculations• Discounts / Gifts• Financial Reporting
Current Architecture
• Netflix Data Center• Batch Processing and
Customer Requests• Oracle (Single-Master)• Data driven
Cloud Migration - Primary Goals
• Join Netflix in the Cloud• High Availability• Multi-Regional Active/Active• Scalability
New Design• Amazon EC2• Multi-Regional
• Cassandra• Event Driven
Order Flow
Cross-Cutting Concerns
• Publishing Data To Interested Parties– Hold Messaging– Financial Reporting
• Metrics/Analytics• Velocity Checks• Debugging
Wiretap to Common Event Pipeline
Common Event Pipeline
Current Implementation• SQS Entry Point• Custom routing service• Custom code for each
endpoint integration• Weak ordering of events
Future Implementation• Suro / Kafka Entry Point• Configurable Routing and
Transformation• Pluggable endpoint
integration• Option for strong ordering
of events
Sting – Broad Analytics
Druid – Event Aggregation Metrics
ElasticSearch – Event Level Debugging
Mistakes (and Wins) Building Payments At Dropbox
Jean-Denis Grèze & Dan WheelerJune 18th, 2014
Backend Tips• Not about increasing conversion• Not about pricing• Not about plan and feature
optimizations• Not about upselling• Not about consumer SaaS at scale• Not about self-serve in
SMB/SME/Enterprise
Pains of Scaling Payments• Thousands of customers to millions of
customers• SMB to Enterprise
– Custom flows!• International expansion
– Fraud– New payment methods (delayed
settlements)– Different price points
Out Of The Dark Ages• For a long time, only 0.5 engineers
worked on payments and billing• March 2013: consult w/ leading
payment engineers, PMs and executives on how to build an amazing payments team– 15+ in 1 year
Advice• Build a payments + billing backend
that is:– Flexible
• Migrations (sadface)• Requirements will change – often
– Auditable• Always know why and state changed
– Append Only• Never lose data
Team Ca$h
Team Ca$h• Engineers
– Some finance background– A lot of systems/analytics background
• PMs– Some finance/accounting background– Some product/marketing background
• Advisors– Payments experts– Tight feedback loop w/ finance
• Designers
Migrations
Migrations• You will have to migrate
– 3rd-party vaulting to self vaulting– New markets = new processors– If you are a growing company, your internals will
require migrations• Stakes are high
– Double-billing? Forgetting to charge some users? Inadvertently moving users from one pricing category to another?
• Old way:– Ad hoc– Tons of tests
I. Leverage Existing Code: Equivalence
• Write equivalence between old and new implementations (database, API, 3rd party providers, tests, etc.)
• Run everything through both systems at once, with equivalence being tested
• Every step of the way check that equivalence relations hold (e.g., old-style invoice has a new-style invoice equivalent)
• Turn off old system when everything works for X amount of time
Migration Pro-Tip• If you can migrate in both
directions at will on a per-endpoint basis, your life will be awesome and people will love you.
Moneytree
Logging• Dark Ages = tons of logging
• Very comprehensive, but ad-hoc = too much effort to re-create state
• Human error
II. Automated Logging• Automatically log
– Any DB write (graph, relational, etc.)– Any 3rd party API call (and some internal calls
like email)• Pre-log
– Any incoming 3rd party payload• Can recreate past actions if we introduced a
regression• LOG A REASON (and code version)
– 1 year is a long time
Automated Logging =
Time Machine
Not too much data• Dropbox = large scale for SaaS• Hundreds of millions of users
(provisioning is hard)• Tons of data, but still payments data
<< file storage data– Although we do have the benefits of
amazing infrastructure
III. States, Not Deltas• Generally # states << # deltas• States
– Pro 100 + Packrat– DfB + Harmony Enabled
• Deltas– Add 5 licenses, 6 licenses, etc.– Add 20 licenses and switch from monthly to
yearly• Use states and let the system figure out
how to get from start to end
IV. Possibility & Transitions Use Same Code Paths
• No difference between:– entity.is_valid_transition(end_state)– entity.perform_transition(end_state)
• Except that writes are turned off for the former.
• No change for “is something possible” logic to be different than “do the thing” logic.
States Are Nicetransition_space = MoneytreeTransitionsSpace.build_cross_product( entity=me, gateways=ALL_GATEWAYS, plans=[Pro100, Pro200, Pro500, DfB, DfBTrial], schedules=[Monthly, Yearly], currencies=ALL_CURRENCIES, features=ALL_FEATURES, tax_profile=[NoTax, SimpleTax, ComplexTax],)# …If transition_space.supports(FEATURE_PACKRAT): # …
Write Protection• Seems dumb, but need to be careful
not to accidentally change values in payments world. Have clearly-defined code paths that can touch state, talk to 3rd party components, etc.
V. One More Lesson• Payments + Billing != Finance• Business requirements don’t always
translate to what’s best/easiest in the world of accounting. You need to flexibly work in both worlds – can’t risk the user experience to make your finance dep’t happy (and vice-versa)
• Get a great PM
VI. Why Payments Are Cool?• Infrastructure?
– Payments service– Provisioning service
• Product?– Upsells? (+50% increase in revenue per user)– Gating features?
• Product Infrastructure!– Build a successful structure by emphasizing
hard problems in both worlds!
Now + The Future• Other Cool Projects
– ML for risk/anomaly detection (e.g., for payment methods that don’t settle immediately)
– Price AB testing (*)– Cross-platform upsell framework
• Questions?– [email protected]– [email protected]
• Hiring– Get in touch!
Team Ca$h
Lumos Labs, Inc.
Data SanityA short treatise on
rebuilding a payment system
Lumos Labs, Inc.
the background
Lumos Labs, Inc.
Circa 2012Decided to rewrite our payment system
Lumos Labs, Inc.
ReasonsOld payment system has a lot of limitations
Lumos Labs, Inc.
Payment system limitations• Hard to add additional gateways• Models don’t reflect business
reality• Not built for reporting• Code is brittle
Lumos Labs, Inc.
3 months later…
Lumos Labs, Inc.
New payment system features• Trivial to add new payment
methods• Subscriptions are the core
model• Built with reporting in mind• Separate, well encapsulated
library
Lumos Labs, Inc.
Works like a charm!
Lumos Labs, Inc.
wait… just one problem
Lumos Labs, Inc.
Having two parallel payment systems is
not sustainable
Surprise!
As it turns out
Lumos Labs, Inc.
Just deprecate the old one
Lumos Labs, Inc.
Just deprecate the old one• Don't migrate anyone, let old
users churn out naturally (will take forever)
Lumos Labs, Inc.
Just deprecate the old one• Don't migrate anyone, let old
users churn out naturally (will take forever)
• Migrate everyone, but only most critical/current subscription info (loses history)
Lumos Labs, Inc.
Just deprecate the old one• Don't migrate anyone, let old
users churn out naturally (will take forever)
• Migrate everyone, but only most critical/current subscription info (loses history)
• Migrate everyone + full history (tricky, lots of edge cases)
Lumos Labs, Inc.
so, what’d we choose?
(drum roll)
Lumos Labs, Inc.
We decided to migrate everyone and their entire
billing history
Lumos Labs, Inc.
Why?One systemOne history
One source of truth
Lumos Labs, Inc.
Lumos Labs, Inc.
enter the sanity check
Lumos Labs, Inc.
enter the sanity checkJust make a sanity check before and after the migration to ask questions
Lumos Labs, Inc.
enter the sanity checkBoth models should answer certain questions the same way, e.g:• How much did the user pay for a
subscription?• How many total transactions did the user
make?• Was auto-renewal enabled on X date?
Lumos Labs, Inc.
class SanityCheck Methods = [:subscriber?, :transaction_count] # etc.
def initialize(record) @record = record @before_values = SanityCheck.values(record) end
def self.values(record) Methods.map { |m| [m, record.send(m)] }.to_h endend
enter the sanity check
Lumos Labs, Inc.
class SanityCheck ...
def check @after_values = SanityCheck.values(record) @diff = diff(@before_values, @after_values) end
def diff(a, b) a.delete_if { |k, v| b[k] == v } .merge!(b.dup.delete_if { |k, v| a.has_key?(k) }) endend
enter the sanity check
Lumos Labs, Inc.
enter the sanity checkUser.each do |user| sanity_check = SanityCheck.new(user) user.migrate! sanity_check.check if sanity_check.diff.any? # sanity check failed -- log an error, rollback, etc. raise ActiveRecord::Rollback else # woo hoo, success! user.update_attributes(:migrated => true) endend config/initializers/
user_auth.rb
Lumos Labs, Inc.
the result
Lumos Labs, Inc.
A resounding success!
the result
Lumos Labs, Inc.
(Mostly)
the result
Lumos Labs, Inc.
Did not work in all cases●Payment system behavior changed
over time
Lumos Labs, Inc.
Did not work in all cases●Payment system behavior changed
over time●Some concepts did not map
between systems
Lumos Labs, Inc.
Handling the edge cases●Replayed history as it would play out
in the new system●Still kept most critical information
the same (e.g. transaction timestamps)
Lumos Labs, Inc.
Skip ahead to todayMigrated 99.9994%* of all our users successfully to the new system
*The remaining .0006% live on for business, not technical reasons
Scaling Merchant Payouts at Square
Alec Holmes
‣ Infrastructure team that moves money
‣ Card processing (money in)‣ Settlements (money out)
Payments Team
‣ Monolithic Rails app‣ Database storage and IO limits‣ Controls around accounting data
Disaster Ahead
Monorail
Database
Payments Bills
Deposits
Where We Started
Ledger
Database
Payments Bills
DepositsMonorail
Database
API
Current System
Challenge:Migrating Live Data
‣ Stricter money-moving event model‣ Same event streams to all systems‣ Smart queueing of events
Challenge:Moving to APIs
‣ Replace direct data access with API calls
‣ Hadoop for analytics queries
That's it. All pennies accounted for!
Billing Globalization and Auto-Renewal Optimization
• Paul Huang• Engineering Manager• SurveyMonkey• June 18th, 2014
Our mission is to help people make
better decisions
We help people get answers to their questions
customers create surveys distribute to others recipients respond customers analyze and gain insight
Our customers have invented many ways to use our tool
98
90MUnique visitors every month
17K New sign-ups
daily “SurveyMonkey turns online
surveys into a hot business.”
“ Start-up companies using ‘freemium’ business models, including SurveyMonkey, are thriving as the cost of computer power and storage falls.”
One of the hottest startups to watch. 2.4M
Survey responses are generated daily
SurveyMonkey is the world’s largest survey company
Billing Globalization
In 2010...
Currencies:1 currency (USD).
Payment Methods: Credit Card, Invoice.
Pricing: one price per package.
Revenue: $ MM
Billing Globalization
In 2014...
Currencies: 39 international currencies.
Payment Methods: Credit Card, PayPal, Debit Card, Bank Transfer, iTunes, Invoice.
Pricing: each package with different price per currency, multiple prices allowed for price testing.
Revenue: $$$$ MM
Auto-Renewal Optimization
Retry Logic: retry failed payments at different intervals, ~60% retry success rate
Account Updater: get users’ updated credit card data (number / expiration date) before next renewal date, ~4% users at 96% success rate
Immediate Renewal: charge pending invoices when users update payment accounts, ~4% retry success rate improvement
TheRoad
Ahead
Upcoming Billing Projects in 2014
VAT 2015: preparations for VAT rule changes in Europe in 2015
Brazil Payment Processing: set up local entity, integrate with a new Payment Service Provider in Brazil
Continue Improving Retry Logic: increase retry frequencies
The End (BTW, we’re hiring…)
E-commerce at EA
Feifeng Yang / Michael ChenTech Lead Payment / Software Engineer @ Electronic Arts
E-commerce platform• Power hundreds of games,
millions of gamers• FIFA• Battlefield• SimCity, Sims• Origin• Play4free, etc
• Foundation services to EA game studios
Foundation services• EA Origin Store: http://store.origin.com• Payment Gateway• Ewallet System• Billing System• Subscription System• Risk and Fraud System• Shopping Cart System• Catalog System• Rating and Revenue Allocation System
Foundation services
Billing and Payment• Preorder and subscription• Package goods• EA Ewallet: million of transactions per day• Product types: base game, bundles, DLC, PDLC, etc• Payment methods
• Card processing• PayPal• Mobile billing• Other ~30 local payment methods.
Technologies• RESTful API framework: SimpleHttpServer (home grown, ported from open source project SimpleWeb)
• Application: OSGI, Use Equinox as OSGI container• Datastore: MySQL + Sharding, Oracle (to be retired)• Cache: Coherence cache (to be retired); Memcached• Security: OAuth 2.0, CAS 2.0 (Central Authentication SSO)
• JMS messaging: TIBCO, iHUB.
112
* Jun 1st to Oct 17th