Managing Machine Learning Projects in Industry

Ewa Dominowska

Facebook, Engineering Manager

Agenda

• Building a Team

• Selecting and Framing a Problem

• Problem Solving Approach

• Evaluating Solutions

• Delivering Impact

Building a Team

• Engineer + ML Expert + Statistician + IR Expert

• Domain expertise

• Academic vs. industry experience

• Research + engineering + experimentation = applied research

• Investing (domain) vs. outsourcing (science)

Selecting a Lead

ML Expert

Engineer Manager

1 8

8

8

Organizational Structure

• Centralized Research Team

• Centralized Applied Research Team

• Embedded Researchers

• Team Members

• Academia

• Conferences, competitions, data releases, benchmarks

MSR, Facebook AI Lab

LiveLabs, FB Applied ML

Source: Bonkers World

Motivating

• Intellectual challenge

• Creative work

• Autonomy

• Purpose

• Mastery

• Recognition

• Publishing

• ConferencesSource: Motivationhacker

Source: Oreilly

Selecting and Framing a Problem

Selecting and Framing a Problem

Start with a business problem

Break down the problem

Understand the impact

Find the right data

Select an objective function

Build Models

Measure and Evaluate

ExperimentProductionalize

/ Scale

Problem Solving Approach

• Establish a baseline

• Check your assumptions

• Select a model\learning technique

• Select features

• Measure and evaluation

• Experiment

• Stability, scalability and robustnessSource: Sheldoncomics

Evaluating Solutions

• Defining the right metrics

• Offline evaluation

• A|B testing

• Meaningful vs. representative

• Representativeness and stability of results

• Offline vs. online metrics

• How to split traffic

• user, request, budget effects

• How long to run a test

• statistical significance, power, seasonality, novelty

• Calibration

• Model interactions

• Residue effects from previous experiments

Experimentation – Practical Lessons

Delivering Impact

• Plan for valuable failure

• Measure long term/steady state effects

• Engineering improvements

• Re-use of components, tools, models and frameworks

• Durability and robustness

• Data, context changes

• Measurement, monitoring, experimentation

Thank you!

We are hiring at Facebook!