Pavan Sudarshan@pavanks

Anandha Krishnan@anandhak (Jake)

Disclaimer

This talk is mostly platform/language independent.

But, we think the Ruby community is very aligned to most of what we will be talking about

We have made a lot of mistakes*...

…all our life as hackers

We want to talk about one such mistake with a couple of examples

* No employers or paranoid androids were hurt in the process!

About 3 months back...

I saw this guy walk into a pharmacy and ask for some medicine for a toothache

Between us, we have seen people ask...

Medicines for

→ Headache→ Fever→ Dysentery→ Rashes→ Cuts from a knife (I am still scared about this

one!)

Do you see anything wrong with this?

A pharmacy...

Potentially has a cure for pretty much most ailments

But the important thing is…

To figure out which ailment you are actually suffering from?

If choosing the right medicine was that simple, the world would just have...

Pharmacists

and not

Doctors

Doctors treat attack symptoms by...

→ Coming up with a problem hypothesis

→ Using Diagnostic tests

→ Prescribing the right treatment

→ Validating that the symptoms vanish

Pharmacists understand...

→ The details of the medicines

→ They can reverse engineer to find the disease based on the medicines

→ Are not responsible for prescribing treatment

Now, with this context, lets get back to our mistake...

Our mistake...

The way we dealt with bad symptoms in our code bases & projects

Anecdote - 1

Pesky regression bugs

Every time we built something, a bunch of other things broke.

Goal

Reduce the number of these bugs which were around 10 every iteration to 5

Some facts

→ We took over the project from another team

→ Our knowledge of the code base was bad

→ Test Coverage was extremely low

What would you do if you were to fix this?

Low coverage was staring us straight in the eyes

Fix coverage first

→ Regression bugs could be caught by tests

→ We focussed on increasing coverage

→ Started monitoring and improving coverage steadily over a period of 2 months

Our thought process was something like this...

Metric/Tool

Symptom

Solution

Provokes Improves

And the result?

We ended up...

→ Improving coverage drastically

→ However, things got marginally better, but just that, marginal

While us devs were happy about the coverage, the project manager was

extremely frustrated

This and a lot of similar mistakes...

Across different projects, drove us to rethink our approach.

We ended up realising something basic...

Never improve a metric.

Solving a problem should automatically improve the metric.

Instead of this...

Metric/Tool

Symptom

Solution

Provokes Improves

Targets

Problem

Symptom

Solution

Metrics

MeasuresProblem

Solution

Metrics

Problem

Solution

Metrics

Identifies

Fixes

We basically stopped doing this...

→ Crawl github to find a suitable gem

→ Setup it and start monitoring

→ Spend time and effort to improve what you are monitoring

Lessons learnt about Test Coverage

Measuring

→ Controller specs vs Model specs coverage

→ Just line coverage is not very useful*

→ Line→ Branch→ Unique path

* The Single Metric Fallacy

Reporting

→ Coverage is inherently not red/green

→ Code climate does a very good job

Ratcheting

→ Double edged sword

→ Make sure people are not scared/hate coverage

Improving→ Adding unit tests to old untested code bases

is dangerous

→ Higher level tests (functional tests) are a great safety net for refactoring

Always keep in mind the problem you’re solving

Anecdote - 2

About 5 percent of requests take a very long time to return

Goal

Find the root cause of the unpredictable response times and fix it.

Questions?