FROM NASA TO STARTUPS TO BIG COMMERCEBuilding Maintainable, Scalable Projects

DANIEL GREENFELD

FROM NASA TO STARTUPS TO BIG COMMERCEBuilding Maintainable, Scalable Projects

DANIEL GREENFELD

Let’s get

started

BIGPLANS

https://www.flickr.com/photos/archer10/2215343914/

https://www.flickr.com/photos/ramcguire/19953122

Early Design Mistakes

and Technical

Debt

Early Design Mistakes

Early Design Mistakes• Not understanding or knowing the requirements

• Over or under architecting

• Choosing the wrong tools

• Premature Optimization

• Management fiat made with ignorance or bad data

• “Brilliant code”

2006

Electronic Giftcards

Epic Volume

Our Challenge: ”Brilliant Code”

Refactor

Technical Debt

Loan Against

the Future

Borrowing now…

…hoping you can

pay it off later

Loan Against the Future

• Get it done fast!

• Grouping data poorly

• Hardcoding

• Upcoming technology will fix it

• Leave testing and documentation for later

Technical Debt

Challenges

Taking out the Loan

Great Engineering Team

Early Design Mistakes

and Technical

Debt

Hard to Maintain Enhance Scale Projects

Example

http://science.nasa.gov

My public NASA work

http://science.nasa.gov

My public NASA work

So Many Things to Get Wrong

is a constant

Failure

But that’s okay

How Do We Build for the

Future?

The Basics

SR-71

http://commons.wikimedia.org/wiki/File:Lockheed_SR-71_Blackbird.jpg

Kelly JohnsonRenowned and prolific aircraft design engineer

Invented Seriously Awesome Planes!!!

http://en.wikipedia.org/wiki/Lockheed_P-38_Lightning#

mediaviewer/File:Lockheed_P-38J_Lightni

ng_-_1.jpg

http://commons.wikimedia.org/wiki/File:Usaf.u2.750pix.jpg

http://commons.wikimedia.org/wiki/File:Lockheed_SR-71_Blackbird.jpg

http://en.wikipedia.org/wiki/Kelly_Johnson_(engineer)

Kelly Johnson

AREA 51

Really Good Manager

Renowned and prolific aircraft design engineer

http://en.wikipedia.org/wiki/Kelly_Johnson_(engineer)

KISS

Kelly Johnson Said:

Keep It Simple, Stupid

Keep it Simple Stupid

If Kelly Johnson kept it simple,

why can’t we?

http://en.wikipedia.org/wiki/Falcon_9 Projects are like rockets

Balancing 10 story building on top of an engine

Sophisticated project • Millions of users • Millions of rows of data • Involving finances

balancing

KISS

Kelly Johnson Said:

Keep It Simple, Stupid

Simple != Simplistic

Needless Complexity

is Bad

Bad ComplexityOn a recent pre-Eventbrite API project…

I was accused of making an

API too complex

to use!

@pydanny

46

Decorators are great for API design!

@pydanny

47

Partials of Decorators

http://pydanny.com/python1partials1are1fun.html

@pydanny

48

Callables in Partials

@pydanny

49

Callables in Decorators

My design was brilliant!

@pydanny

50

My API design is…

brilliant! brilliant! brilliant!

Bad Complexity

• It’s so easy to understand!

• I don’t find it abstract at all

• Really

• What’s wrong with decorators + partials + callables?

On my recent pre-Eventbrite API project…

@pydanny

52

My API design is…

Something no one

wants to use.

Or Maintain.

Lesson Relearned

• Since business logic is already complex…

• Write tools that are simple to understand.

• Easy to maintain

• Because…

Don’t use that design again

Maintenance Over Time

is Hard

Case Study: Every Project

• Your project has been around for a while

• Problems

• Fixing bugs is hard

• Adding features is hard

Why is it harder?• Project is no longer the bright and shiny

• Adding features adds to complexity

• Bugs caused by unforeseen edge cases

• Not enough tests make catching developer introduced bugs harder

• Mistakes at the beginning are really starting to show

Original Engineer(s) was an Idiot

Original Engineer(s) are always idiots

I’ve yet to join a project where I didn’t feel like

ranting all the time

Rant Time!• Why use a relational database?

• Why not use a non-relational database?

• Why this programming language?

• Why not use OO or functional programming techniques?

• Why use OO or functional programming techniques?

• What the heck is this programming pattern anyway?

Caveat: The Constant of the Worst Code

Ever

Hindsight is 20/20

• No one predicts with 100% accuracy

• Not on software projects

• It’s easy with hindsight for us to complain about the decisions made.

Reality Check

• Making accurate predictions is hard

• Projects grow organically

• At least you are getting paid to work on this, right?

Be Understanding

• Don’t be a jerk.!

• Try to understand why things evolved the way they did.

• Forgive your predecessor

• They can provide useful information!

• Circumstances can and will be reversed

Complex Foundations: Harder to maintain

Complexity2=

Complex Foundations

Complex Business+

Complexity2=

Hard to Debug

Hard to Enhance+

Simplicity is a

virtue

Simplicity is a VirtueFoundations: Simple

But not Simplistic

Feature Construction: Simple

Bug Fixes: Simple

Be careful with databases that make simplistic approaches easy.

JSON => SQL

Pro-TipWould it work in a spreadsheet?

relational databaseYES!

No! non-relational database

Simplicity isn’t always the answer

Sometimes you do need complexity

Right at the Beginning

Valid Reason for Complexity:

Database Transactions

Transactions aren’t complex, but engineers often think they are.Note:

http://commons.wikimedia.org/wiki/File:WinonaSavingsBankVault.JPG

http://commons.wikimedia.org/wiki/File:Okayama_Red_Cross_Hospital.jpg

Database transactions protect

finances and

lives

Valid Cause of Complexity

transactionsIf one operation fails,

revertrevertrevert

Don’t code solutions to handle

transactions

Use databases with transactions

PostgreSQL MySQL

Avoiding Database Transactions is Simplistic

You have to build special workarounds to make things reliable

(Some very successful non-bitcoin payment systems have done this)

Is a simplistic approach to avoid complexity…

A) Bad Design Decision? !

B) Technical Debt?

Answer

• Plan to fix later?

• Otherwise:

Technical DebtBad Design Decision

How do we remember all

this stuff?

Best Practices

Insist on Best Practices

Cleaner Consistent Stronger Elegant CODE

Easier to Debug

Easier to Test

Easier to Add Features

New Staff Ramp Up

FasterFasterFaster

How to Best PracticeResearch and find the common consensus

PEP-8

PEP-20

Example: Python

Find ReferencesResearch and find the common consensus

Research and find the common consensus

Find References

Research and find the common consensus

• MySQL

• PostgreSQL

• MongoDB

• Redis

• Django

• Python

• CouchDB

• Cassandra

• BigTable

• Flask

Find References

Make project, package, and module boilerplate the same.

Boilerplate Consistency

What’s the right, current way to start a project?

Django, Flask, SOA, DevOps, etc

(Lightning talk on this later)

Best-practice focused project templates!

Don’t Reinvent the Wheel

http://commons.wikimedia.org/wiki/File:Triple_Rotacaster_commercial_industrial_omni_wheel.jpg

Example: CMSDon’t build your own!

Unless your project is about reinventing wheels

• Email Service

• Operating System

• Comments

• Etchttp://en.wikipedia.org/wiki/Omniwheel

Omniwheels are often employed on small, autonomous robots used in academia!

Management Pro-Tip Part 1

If engineers consistently refuse to follow defined best practices…

They purposefully lower the Bus Factor

http://commons.wikimedia.org/wiki/File:B43OleBillatIWMLondon.jpg

Management Pro-Tip Part 2

Get new ones

Best Practices for

Scaling

Scaling usually isn’t a problem

97% ChanceYou Ain’t Gonna Need it

A 3% Case

“Register here”

http://commons.wikimedia.org/wiki/File:President_Barack_Obama.jpg

Two Quick Scaling Fixes

Based on the fact that the early scaling problems are almost always database-related.

It’s almost always about the data

First Fix: Caching

The first healthcare.gov patch implemented by their tiger team

added caching.

Second Fix: Indexing

Almost every datastore supports indexes.

Apply them to places with the most data reads.

Explore other scaling

options

Once caching and indexes are in place…

Not before

Caching and Indexing!

That’s called “premature optimization”

http://en.wikipedia.org/wiki/Falcon_9

Summary

pydanny.com

Engineer at

Co-Author of

Architecture Team