February 2015

Managing Service Integration for High Uptime In a Lean Startup

Shobana RadhakrishnanVice President of Engineering, Mindflash

2

Agenda

• Introduction• Managing External Service Integration• Mindflash API• Microservice Architecture at Mindflash

3

Mindflash At a Glance

Mindflash App

Mindflash Customer API Services

Course Videos

Player

DeveloperManagerTrainer/

Prospect

4

Some Numbers

• 1000+ Enterprise Customers• Thousands of trainees/program• 2 million courses completed• Increasing mobile adoption• Increasingly global programs

5

Lean Startup**

**Eric Weis: http://theleanstartup.com/principles

6

Lean Startup Principles at Mindflash

• Minimum Viable Product (MVP)

• Ship Frequently• Monitor Outcome• Iterate

7

Questions

• How do you manage service deployment for high uptime and quality?

• How can you guarantee high uptime when relying on several external services?

• How can you do these in a lean startup?

8

• Monthly releases• Somewhat automated• Multi-hour deployment, fall-out after releases• Test vs. Production not completely aligned

2013 and Prior

9

• Ability to make multiple releases a day (Major features still monthly)

• Highly automated• Deployment in minutes, major fall-outs rare• Test vs. Production more completely aligned

Mid 2014 on

10

• Continuous feature releases• Full deployment automation• Deployment in minutes, major fall-outs rare• Test vs. Production fully aligned

Next

11

Microservice Architecture

Microservices is a software design pattern, in which complex applications are composed of small, independent services communicating with each other using highly decoupled language-agnostic APIs

12

Architectural Difference

Monolithic App Microservice-based App

Presentation Layer (View)

Middleware/Business Logic Layer (Controller)

Database (Model)

13

Monolithic vs. Microservices

Monolithic• N-tier Application• Code dependencies• Shared database• Larger units of deployment

Microservices• Independent services• Easier to manage and

deploy• Loose Coupling

14

When to Consider Microservices?

• Team working on the application• Quick Ramp-up for new Team Members• Continuous Deployment• Take advantage of emerging technologies

15

Three Angles For Microservice Architectures

16

Integration with SAAS/PAAS Services

17

Exposing API

18

Internal Microservices

19

Mindflash External Service Integration

20

Trends

• Growth of SaaS/PaaS Services• Evolution of Hub and Data

Analysis Solutions• Public API and Open Source

21

Architectural View

22

Services we Integrate With

23

Integration

• Building and Maintaining Component• API Migration and Upgrades• Security and Bug Patches

24

API Call FlowAPI

Customer 1API

Customer 2API

Customer 3API

Customer 4API

Customer 5

Services Layer

API Gateway

25

API Considerations

• Heterogeneous Needs• User release cycles vary• Backwards compatibility• Objects evolve continuously

26

Approach and Lessons Learned

27

Automate – BrowserStack, Jenkins, Selenium

28

Fail Fast and Recover Quickly – Pingdom, Crittercism, AWS Monitoring and Alerting

29

Measure and Iterate – Optimizely, Gainsight, Pingdom, Google Analytics, KISSMetrics,

Internal reporting

30

Upgrade Continuously

31

Deploy Small and Continuously

Build

Deploy to QA

Check in

Run Tests

Deploy to PROD

Rule-based subset that needs to be rebuilt/deployed

32

Implementation Best Practices

• Loose Coupling• Chain-of-responsibility pattern• Single Responsibility Principle• Defensive Programming• Clear separation of real-time

vs. asynchronous• Feature Flags

33

Real-time Vs. Asynchronous

MF

Trainers and Prospects

Trainees

Hydra

Billing

RDS

Analytics

File Conversion

Feature Integration

Developers

MF API

Gateway

34

Feature Flags – our implementation

• Feature Flags persisted in DB• Set per account and pricing tier• Common Uses

– Manage rollout for risky changes– Adjust feature offering based on pricing tier– Enable limited testing of a feature

35

Be Aware of Cultural Factors

36

Results

• Releases without fallout• Shorter deployment cycles• Much lower error rates• Consistently high uptime• Consistently low response

time

37

Internal Microservices

38

Our Initial Implementation

Emails

All Applications – TraineeWeb, TrainerWeb, Player etc.

Video Conversion Jobs

Async updates with Third Party Systems

39

Characteristics

• High Performance and Uptime• Easy Error Tracking• Highly Scalable System• Achieved Decoupling (sort of)• Achieved independent deployment (again,

sort of)• Independent code base (not really)• Decoupled deployment (not really)

40

New Approach

• Independent Verticals ie. Services• Hapi framework• Common Services across Verticals• Incremental code migration• Integrate tools such as Docker, Stackato,

Ansible, Kubernetes• More to come!

41

Coming Up

• Fully decoupled microservice implementation

• Integrate automated container management

• Fully automated backward compatibility testing

42

References

• Product Blog – http://www.mindflash.com/blog

• Developer Blog – http://www.mindflash.com/developers

43

Thank you!

• https://www.linkedin.com/in/shobanaradhakrishnan

• Twitter: #shobanarad, #mindflash

• We’re hiring!

Questions?