Cell Tracking via Proposal Generation & Selection

Saad Ullah Akram,Juho Kannala, Lauri Eklund and Janne Heikkilä

saad.akram@oulu.fi 2

Overview

• Introduction• Importance

• Challenges: detection & tracking

• Current state of cell tracking field

• Our method• Cell proposal generation (CPN)

• Proposal graph

• Cell tracking via proposal selection

• Summary

saad.akram@oulu.fi 3

Overview

• Introduction• Importance

• Challenges: detection & tracking

• Current state of cell tracking field

saad.akram@oulu.fi 4

Time

Cell Tracking

• Process of:• Locating a moving cell over time &

• Detecting cellular events (division, death, entry, exit)

saad.akram@oulu.fi 5

Lineage of a cell

: highlights links between parent & daughter cells

Applications

• Understanding dynamic cellular behavior• Spread of diseases (e.g. cancer)

• Effectiveness/safety of a drug

• Organ/embryo development

• Gene profiling

saad.akram@oulu.fi 6

Automation

• Huge datasets• 1,000s of 3D images/day

• 10,000s cells in each image

• > 10 TBs of image data/day

• Subtle patterns

• Accurate/objective/repeatable

saad.akram@oulu.fi 7

Video from: F. Amat et al., “Fast, accurate reconstruction of cell lineages from large-scale fluorescence microscopy data.,” Nat. Methods, 2014.

Challenges: Tracking

• Difficult to model motion

• Similar cell appearances and shapes

saad.akram@oulu.fi 8

Similar shapes & appearances

Challenges: Tracking

• Low frame rate• Large movements

• Large shape and appearance changes

• Difficult to detect cell division

saad.akram@oulu.fi 9Parent cells in last frame before division

Large movement

Challenges: Detection

• Detection failures account for most tracking errors

• Large variation in cell shapes & appearances

• Microscopy modality

• Labeled structure

• Resolution

• Cell density

saad.akram@oulu.fi 10

Bri

ght

Fie

ldP

has

e

Co

ntr

ast

DIC

M

icro

sco

py

Fluo

resce

nce

Dye

s & P

rote

ins

Image from: E. Meijering, “Cell Segmentation: 50 Years Down the Road,” IEEE Signal Process. Mag., 2012.

Current cell tracking methods

• Tracking by detection• Detect cells in each frame

• Associate detections in nearby frames

• Pros₊ Low frame rate/resolution₊ Simple and modular

• ImageJ (TrackMate, MTrack2), CellProfiler (TrackObjects), etc

• Tracking by model evolution• Detect cells in 1st frame and

represent it using some model

• Segment current frame using the model from last frame

• Pros₊ Can handle noisy images

saad.akram@oulu.fi 11

Limitations of current cell tracking methods

• Cell detection methods assume• Cell appearances

• High intensity near the center

• Characteristic intensity profile at cell boundaries

• Cell shapes• Round or elliptical shapes

• Cell size within some narrow range

• Assumptions are often not satisfied for new sequences

• Lack robustness• Require tuning multiple parameters

saad.akram@oulu.fi 12

Overview

• Our method• Cell proposal generation (CPN)

• Proposal graph

• Cell tracking via proposal selection

saad.akram@oulu.fi 13

Tracking by proposal selection

1. Generate cell proposals• Use CNNs to learn the cell shapes and appearances

2. Link cell proposals using edges (represent cellular events)

3. Select subset of cell proposals & their associations (provides cell tracks)

saad.akram@oulu.fi 14

Cell proposals/candidates

• Cell detection can benefit from• Temporal information

• Machine learning to decide what a cell looks like

• A cell proposal is a candidate segmentation of a cell• Enables representation of multiple segmentation hypothesis in ambiguous regions

saad.akram@oulu.fi 15

Image Cell Detection Cell Proposals: 2 Cell Proposals: 3 Cell Proposals: 4

Cell proposal network (CPN)

saad.akram@oulu.fi 16

3

32

3

64

3

32

3

128

3

64

3

256

3

128

RO

IPo

olin

g 12

8

Can

did

ate

Bo

un

din

g B

oxe

s

Nx5

7

32

3

32

5

64

3

64

3

128

3

512

1

6k

3

256

4k

bb

ox

2k

sco

re

NMS NMS

NMS

1

16

33

64

3

32128Proposals

Proposal Masks

25

.25

.2

+

+

+

Fig: CPN: Convolutional, max-pooling, fully connected, and deconvolutional layers are shown. Proposed bounding boxes and segmentation

masks after non-maxima suppression (NMS) are shown for a selected area from Fluo-N2DL-HeLa dataset.

Bounding box network

Segmentation network

S

T

Tracking graph (g): Ground truth

saad.akram@oulu.fi 17

D1

D2 D3

D5D4

b)

• Cell tracks can be represented using a graph

• Nodes represent cells

• Special nodes (S & T): Initiate/terminatecell lineage trees

• Edges represent cell events• Cell entering field of view

• Cell exiting field of view

• Cell division (mitosis)

• Cell moving

a)

Tim

e

Graphical model (G)

• A graphical model (G) is used to reason about which cell tracks should be selected

• Nodes represent cell proposal probabilities

• Black edges represent proposal constraints

• Special nodes (S & T): Initiate/terminate cell lineage tress

• Only few edges of each type are shown to avoid clutter

• Edge weights represent cellular event probabilities• Cell entering field of view

• Cell exiting field of view

• Mitosis• Cell moving

saad.akram@oulu.fi 18

0.94 0.94

0.95 0.99

0.88 0.81

0.98

0.62

b)

0.42 0.860.11

a)

Tim

e

S

T

0.010.01

0.070.03

0.86

Proposal selection

saad.akram@oulu.fi 19

c)a)Ti

me

• Each sub-graph of our model (G) represents a tracking hypothesis

• Optimal tracks can be obtained by finding the sub-graph with the highest probability

0.94 0.94

0.95 0.99

0.88 0.81

0.98

0.62

b)

0.42 0.860.11

S

T

0.010.01

0.070.03

0.86

0.94 0.94

0.95 0.99

0.88 0.81

0.98

b)

0.86

S

T

0.86

Results

• Same model structure is used for all datasets

• 4 datasets from ISBI Cell Tracking Challenge [1]

• TRA:• Tracking score (0-1 (perfect))• penalizes errors in the tracks

• SEG:• Segmentation score• Instance level intersection over union

overlap.

saad.akram@oulu.fi 20

Video [1] Method TRA SEG

Fluo-N2DL-HeLa-02

CPN 0.983 0.845

EPFL [3] 0.97

KTH [4] 0.975 0.837

Fluo-N2DH-GOWT1-02

CPN 0.972 0.873

HEID [2] 0.95

EPFL [3] 0.95

PhC-C2DH-U373-02CPN 0.935 0.738

U-Net [5] 0.955 0.830

PhC-C2DL-PSC-01 CPN 0.943 0.661

[1] M. Maška et al., “A Benchmark for Comparison of Cell Tracking Algorithms,” Bioinformatics, 2014.

[2] M. Schiegg, et al., “Graphical Model for Joint Segmentation and Tracking of Multiple Dividing Cells,” Bioinformatics, 2015.

[3] E. Turetken, et al., “Network Flow Integer Programming to Track Elliptical Cells in Time-Lapse Sequences,” in T-MI, 2016.

[4] K. E. G. Magnusson, et al., “A Batch Algorithm using Iterative Application of the Viterbi Algorithm to Track Cells and Construct Cell Lineages,” in ISBI, 2012.

[5] O. Ronneberger, et al., “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in MICCAI, 2015.

Fluo-N2DL-HeLa-02

saad.akram@oulu.fi 21

Tim

e

PhC-C2DH-U373-01

saad.akram@oulu.fi 22

Tim

e

PhC-C2DL-PSC-01

saad.akram@oulu.fi 23

Tim

e

Overview

• Summary

saad.akram@oulu.fi 24

Summary

• Our tracking method₊ (Potentially) can be applied to other sequences₊ Has better performance than existing methods₋ Requires annotated data

• More Information:• Paper: https://arxiv.org/abs/1705.03386

• Code: https://github.com/SaadUllahAkram/CellTracker

saad.akram@oulu.fi 25

Thanks.Questions ?

saad.akram@oulu.fi 26