November 2014

Managing Service Integration for High Uptime In a Lean Startup

Shobana RadhakrishnanVice President of Engineering, Mindflash

2

Agenda

• Services at Scale• Mindflash Customer API• Mindflash Integration with External Services• Approach and Lessons Learned

3

Questions

• How do you manage service deployment for high uptime and quality?

• How can you guarantee high uptime when relying on several external services?

• How can you do these in a lean startup?

4

Mindflash At a Glance

Mindflash App

Mindflash Customer API Services

Course Videos

Player

DeveloperManagerTrainer/

Prospect

5

Lean Startup**

**Eric Weis: http://theleanstartup.com/principles

6

Lean Startup Principles at Mindflash

• Minimum Viable Product (MVP)

• Ship Frequently• Monitor Outcome• Iterate

7

Typical Constraints

• Budget• Resources• High Release

Frequency

8

Monolithic vs. Microservices

Monolithic• N-tier Application• Code dependencies• Shared database• Larger units of deployment

Microservices• Independent services• Easier to manage and

deploy• Loose Coupling

9

Our Journey

• Started with a monolithic service – FLEX, .NET• Independent service and middle layers• Shared database with read rep• 20+ external services -> microservices

architecture

10

Our Goals

• Uptime and Response Time• High website performance• Low error rates

11

Key Metrics for Mindflash

• Course Creation Rate• Course Completion Rate• Uptime and Response Time

12

Mindflash API

13

Why

• Bulk Operations with UX flexibility• Varied needs for same functionality• Users more self-sufficient

14

API Functionality

• REST Services• User and User Group

Management• User Enrollment • Reporting

15

API Call FlowAPI

Customer 1API

Customer 2API

Customer 3API

Customer 4API

Customer 5

Services Layer

API Gateway

16

API Considerations

• Heterogeneous Needs• User release cycles vary• Backwards compatibility• Objects evolve continuously

17

Mindflash External Service Integration

18

Trends

• Growth of PaaS Services• Evolution of Hub and Data

Analysis Solutions• Public API and Open Source

19

Services we Integrate With

20

Integration

• Building and Maintaining Component• API Migration and Upgrades• Security and Bug Patches

21

Approach and Lessons Learned

22

Mindflash Development Philosophy

• Reuse, don’t reinvent• Action and Iteration• Upgrade continuously

23

System Design Best Practices

• Loose Coupling• Chain-of-responsibility pattern• Single Responsibility Principle

24

Programming Best Practices

• Defensive Programming• Critical vs. non-critical

operations• Real-time vs. Asynchronous• Feature Flags

25

Real-time Vs. Asynchronous

MF

Trainers and Prospects

Trainees

Hydra

Billing

RDS

Analytics

File Conversion

Feature Integration

Developers

MF API

Gateway

26

Critical vs. Non-Critical

MF

Trainers and Prospects

Trainees

Hydra

Billing

RDS

Analytics

File Conversion

Feature Integration

Developers

MF API

Gateway

27

Feature Flags – our implementation

• Feature Flags persisted in DB• Set per account and pricing tier• Common Uses

– Manage rollout for risky changes– Adjust feature offering based on pricing tier– Enable limited testing of a feature

28

Feature Flipping in Node/* Defining a Feature */

var flipper = require(‘flipper’);flipper.add (‘newPlayer’);

/* Testing for a Feature */Var flipper = require(‘flipper’);If (flipper.newPlayer) {}Else {}

/* Another Way */Flipper.isEnabled(‘newPlayer’);

29

Enabling and Disabling

/* Enable */flipper.enable(‘newPlayer’);

/*Disable */Flipper.disable(‘newPlayer’);

/* Finding out all current features */flipper.allFeatures();

30

Continuous Deployment

Build

Deploy to QA

Check in

Run Tests

Deploy to PROD

Rule-based subset that needs to be rebuilt/deployed

31

Process Best Practices

• Agile, Continuous Deployment

• Early customer beta• Automate, automate more• Monitor, monitor more

32

Results

• Releases without fallout• Shorter deployment cycles• Much lower error rates• Consistently high uptime

33

Lessons Learned

• Chain-of-responsibility pattern effective

• Design for easy migration• Comprehensive monitoring• Failure Detection and Recovery• Environment Convergence

34

Future Improvements

• Staging Environment• Blue Green Deployment• Nightly Performance

Testing

35

References

• http://mindflash.github.io/mf-api-example/app/index.html#/sandbox

• Developer Blog – http://www.mindflash.com/developers

36

Thank you!

• https://www.linkedin.com/in/shobanaradhakrishnan

• @shobanarad

Questions?