Optimizing for change: Taking risks safely & e-commerce

Optimizing for change: Taking risks safely & e-commerce

Kellan Elliott-McCrea@kellanCTO, Etsy

Monday, October 8, 12

Launched June 18, 2005 in Brooklyn875,000 monthly active sellers33.5MM items for sale$525MM in sales in 20111.43B page views, in Aug102 engineers74 releases, yesterday


Take more risks.Build a better software.

Have more fun.


“Sure that works when you’re building social software but what about a real business with $$$ involved?” - everybody always


Continuous Deployment:

small changes, pushed frequently


you can’t avoid making mistakes

you can avoid making BIG mistakes


MTBFMTTR

What are you optimizing for?


MTBFMTTR


4 core techniques: 1. Put a Button On It 2. Branch in Code 3. Trunk is Always Deployable4. Dark/Incremental Launches


Put a Button On It.


4 core techniques:if ($cfg[‘awesome_new_search’]) {

# new hotness$rsp = do_solr();

} else {# boring old stuff$rsp = do_grep();

}

Branch in code:use features flags


4 core techniques:Branch in code:use features flags

for free you get:1% launchesadmin only launchesdark launchessplit tests


any engineer can launch an experiment to 1% of users

57 experiments live right nowMonday, October 8, 12

Metrics driven development

measure everything!feedback loops!


Engineers love to measure

make it ridiculously easy



StatsD::timing("page.render", $msec);


Metrics aren’t optional

a feature isn’t done without metrics


Make metrics visible

remove the passwords


Some tools:

Graphite, Ganglia, Logster*, StatsD*, event beacons, log files, EMR, Vertica, Splunk


Getting started? Use StatsD

StatsD @Instagram, Pinterest, Github, Mozilla, LAN.com, Zynga, Kickstarter, LivingSocial and 70+ other companies


Step 1: your 5 core metrics:

@ Etsy:sign ups, logins, checkout, new listings, posts in the bugs forums


Who watches the graphs?


USE COMPUTERS!

Automate your analysis



holtWintersConfidence(Upper|Lower)



continuous integration: unit tests, coding standards, static analysis, risky code paths


Make effective security easy by default

Make insecure patterns “grep-able”


Actively monitor for attacks.

Spikes in 500s and failed logins are your first clue.


“I discovered the vuln late Friday afternoon and wasn't quite ready to email it to them. Saturday morning, I confirmed the hole was still there and fixed a few bugs with my demo.I had my girlfriend test it from her house. It didn't work for her. I tested again and it had stopped working for me. Sure enough, it was now properly sanitized and had the correct JSON MIME type.The following Monday I received a response thanking me for reporting it, and telling me I was right. “


Treat independent security researches with respect.


“Culture eats strategy for breakfast”*

(*possibly apocryphal) Monday, October 8, 12

Thank you!


Optimizing for change: Taking risks safely & e-commerce

Documents

Transcript of Optimizing for change: Taking risks safely & e-commerce