The Three Things Any Software Development Organization Needs

And what to avoid like the plague

Jochen Frey

1

Number One

2

What if ...

... you could run a search engine designed for billions of documents on your laptop?

... try out new ranking & classification algorithms at runtime (and in production)?

... you could rebuild the complete dataset REALLY quickly?

3

Then you could ...

... run lots of regression tests.

... use a symbolic debugger to find defects in your distributed system (and step fast).

... find out if spam filter V2 really is better than spam V1 (or upgrade as you go).

... add this new data field really easily to your content.

4

Tight Development Loop

Iterating is good. Iterating fast is better!

We rarely hit the best solution in the first attempt.

This is really all about experimentation & transparency.

Have tools that allow for fast iteration over the whole (or parts of) the development process.

- Edit / Compile / Test complete system one machine.- Automated system deployment.- Deploy the SW infrastructure.- Deploy a whole environment (dev / stage / prod).- Execute a full regression test (automated).

5

Tight Development Loop

Side Effects:- Make it possible to develop & test in the coffee shop

(or Mexico). - Easy to execute code for debugging- Refactoring with confidence- Bring new developers up to speed quick (because

they can make mistakes)- Repeatable & consistent- Happy Engineers!

Avoid like the Plague- The latest programming languages & tools- Non-versioned code & tools- Too much process (this is about experimentation)

6

Number Two7

What if ...

... you found a person who did spam detection at Yahoo ... even though you don’t have a running system, yet?

... you found a person who managed 2500 machines all by himself, when you don’t even have a colo?

8

Gray Hair

Eventually your company and your system will have to grow up. Make it fast and painless.

Pick strategic roles- Scientists- Architects- QA- OPS- maybe even Managers (gasp!!)

Spend some money (and equity) on the right people- Interview thoroughly (make them write code)- Use your VC / advisors / network- Pay recruiters (if you must)

9

Gray Hair

Avoid like the plague- Architects who don’t write production code (or

unit tests)- Ops guys who can’t hack a Linux box- Scientists who don’t code

10

Gray Hair... and some baby fuzz

Keep the seniors alert with some fresh blood.

Hire some youngsters (specifically: Interns)- Give them meaningful work- Allow them to make mistakes (not too many & make sure they

learn from them)- Keep them if they’re good!

Have some regular engineers as well.

Avoid like the plague- Mandatory happy hour - Cool programming languages

11

Number Three12

What if ...

... your engineers didn’t have “root”?

... you weren’t allowed to install the latest Ruby gems on your staging box?

... you couldn’t fix problems in production?

(even though you’re in a startup)

13

Ops / QA Nazi

You might be happier (and faster) with:

Consistent- Hardware (VM / machine configuration)- Server images- Releases- Testing procedures- Neat cabling!!!

Roles- Release manager- Tech Ops- QA

Must be senior enough to say “NO” (and make it stick).

14

Ops / QA NaziFocus engineers by keeping them away from OS setup / QA processes / production configuration.

Someone who is responsible for the basic (as in: underlying) things and does them well.

- Backups- Monitoring- OS images

Specialized knowledge- Load balancers- IP routing- Security- EC2 scripting

As close to Engineering as possible (see “tight development loop”)

15

Ops / QA Nazi

Avoid like the plague:- Giving root access to developers.- Chasing the upgrade.- Outsourcing this function. The closer to engineering, the

better.- Engineers NOT wearing pagers.

16

jochen.frey@lithium.com

17