Developing a Robust Automated Image-Based …...©2006 Millennium Pharmaceuticals, Inc. 1 Techniques...
Transcript of Developing a Robust Automated Image-Based …...©2006 Millennium Pharmaceuticals, Inc. 1 Techniques...
©2006 Millennium Pharmaceuticals, Inc. 1
Techniques in High-Content Screening and Assay Development
Workshop
Developing a Robust Automated Image-Based Assay
Doug Bowman,Sr. Mgr Imaging Sciences
Millennium Pharmaceuticals
Drug Discovery TechnologyAugust 7, 2006
©2006 Millennium Pharmaceuticals, Inc. 2
The Big PictureThe Far Side by Gary Larson
• Cell-based assay• Immunofluorescence• Technologies / Instrumentation• Screening Assay• People
©2006 Millennium Pharmaceuticals, Inc. 3
• IF Staining– Fix cells to substrate– Permeabilize membranes– Block – Add primary– Add secondary
• Image Acquisition• Image Analysis
HCS Assays using Immunofluorescence
©2006 Millennium Pharmaceuticals, Inc. 4
• Provide physiological context for assaying drug activity via the quantitative analysis of cellular and sub-cellular events.
• Analysis also allows for the classification of cellular phenotypes based on morphological parameters.
“HIGH CONTENT”
Cellular “High Content” Imaging
Translocation
Protein Phosphorylation
NeuriteOutgrowth
Mitotic Index
Spindle Assembly / Morphology
©2006 Millennium Pharmaceuticals, Inc. 5
Tissue…
• High content also!– Many of same assays
• Similar issues• High resolution
scanning
100um 20x objective
©2006 Millennium Pharmaceuticals, Inc. 6
Technologies
• Instrumentation– Liquid handling robotics– Automated microscope-based
systems capable of acquiring high resolution images
• Software algorithms
OperaAutomatedMicroscope
Discovery-1
AcapellaProGuruMetaXpress
InCell1000
ArrayScan
©2006 Millennium Pharmaceuticals, Inc. 7
Image-based assays
= 2λ / image * 5 sites / sample * 3 mice / timepoint= 30 images / timepoint= 400+MB / 5 timepoints
• Large volume of data
= 2λ / image * 9 sites / well * 96 wells / plate= 1728 images / plate = 1GB+ / plate3 plates / hr (Opera) 1 plate / hr (Discovery-1)
= 3λ / image * 15 sites / membrane= 30 images / sample= 120MB / sample
©2006 Millennium Pharmaceuticals, Inc. 8
Working Group
Cell Biology Automation
Imaging
Assay Design
SoftwareDevelopment
ComputationalScience
©2006 Millennium Pharmaceuticals, Inc. 9
Developing a Robust Automated Image-Based Assay:
Guidelines
• Use automation where possible• Visualize images• Develop robust process controls• Use data-driven approach
©2006 Millennium Pharmaceuticals, Inc. 10
A data-driven approach to assay design
• Dissect components of the assay• Using control experiments
– Identify drivers of variability• Minimize them experimentally or account for them statistically
– Optimize parameter selection (robust)• Assess performance of assays in
project-relevant context
©2006 Millennium Pharmaceuticals, Inc. 11
HCS Assay Components
• Dissect components• Using control experiments
– Identify drivers of variability– Minimize them experimentally
or account for them statistically
Wet LabProtocols
ImageAcquisition
Image Analysis
AssayAnalysis
©2006 Millennium Pharmaceuticals, Inc. 12
HCS Assay Components
• Dissect components• Using control experiments
– Identify drivers of variability– Minimize them experimentally
or account for them statistically
Wet LabProtocols
ImageAcquisition
Image Analysis
AssayAnalysis
©2006 Millennium Pharmaceuticals, Inc. 13
Wet Lab Protocols
Reagents
Automation: liquid handling IF Staining Protocol
Automation: cell plating
Reproducibility
Day to day variability
©2006 Millennium Pharmaceuticals, Inc. 14
Older Multidrop tube cassettes found inaccurate – relative inaccuracy ~25% and CVs ~2%
New cassettes purchased and validated –relative inaccuracy ~2% and CVs ~0.7%
100ul manual (12 channel Multipipette) 4.43%100ul Multidrop 1.67%50ul Platemate 0.75%
50ul Tecan 0.80%10ul Precision 2000 1.34%
CVs
• Accuracy and Reproducibility– Compound addition / dilution– Cell Plating– Antibody addition / protocols
Wet Lab Protocols:
Testing liquid handling using Artel system
Artel MVS
©2006 Millennium Pharmaceuticals, Inc. 15
Facilitate a systematic comparison of a large number of assay parameters to maximize data quality while minimizing the time required to find optimal parameters.
- fix- perm- block- primary ab dilution - secondary ab dilution- buffers- wash buffers- number of washes- time/temp of incubation….etc
Wet Lab Protocols:
IF Staining Protocol
©2006 Millennium Pharmaceuticals, Inc. 16
Antibody Optimization
8 row trough
John Donovan
Matrix of 6 fixatives - 4 blocks - 4 abdilutions to assay
96 individual conditions
©2006 Millennium Pharmaceuticals, Inc. 17
PBS TBS MTSB
1:500
1:750
1:1000
0
4 Abdilutions
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
PFA PBS PFA TBS PFA MTSB
Roche
BSA
PBS
Mixed Serum
MeOH Perm buf 1 MeOH Perm buf 2 MeOH Perm buf 3
4 Abdilutions
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
3
4
1
2
PFA PBS PFA TBS PFA MTSB
Roche
BSA
PBS
Mixed Serum
MeOH Perm buf 1 MeOH Perm buf 2 MeOH Perm buf 3
Wet Lab Protocols:
Antibody Optimization
©2006 Millennium Pharmaceuticals, Inc. 18
DNA / Tubulin
Before After
• Comparison of > 400 parameters (~2 weeks)• Enhances analysis of spindle morphology• Save $ Antibody concentration
Scatter P lot
Calculated Column (2)01 02 03 04 05 06 07 08 09 10 11 12
H
G
F
E
D
C
B
A
S/N
Wet Lab Protocols:
Antibody Optimization
©2006 Millennium Pharmaceuticals, Inc. 19
HCS Assay Components
• Dissect components• Using control experiments
– Identify drivers of variability– Minimize them experimentally
or account for them statistically
Wet LabProtocols
ImageAcquisition
Image Analysis
AssayAnalysis
©2006 Millennium Pharmaceuticals, Inc. 20
Image Acquisition
Auto-Focus
Depth of Field Image Corrections
Instrument Choice
Reproducibility
Cell Plating: # sites
Illumination System
©2006 Millennium Pharmaceuticals, Inc. 21
Image Acquisition
Auto-Focus
Depth of Field Image Corrections
Instrument Choice
Reproducibility
Cell Plating: # sites
Illumination System
©2006 Millennium Pharmaceuticals, Inc. 22
Image Acquisition:
AutoFocus
• Image-based and Laser-based%Nuclear count in HT29 cells @ 24 hours
Millennium Pharmaceuticals Inc.
ML00618017-001-A
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic0
10
20
30
40
50
60
70
80
90
100
95%CI = 0.1498 µMEC30 = 1.51734 µM
©2006 Millennium Pharmaceuticals, Inc. 23
60 wells, 9 sites / well
Millennium Pharmaceuticals Inc.
AVG %Ratio Nucl/Cyto in U2OS cells
[CMPD] (µM)0.01 0.1 1 10
AV
G %
Int r
atio
0
20
40
60
80
100
12095%CI = 0.2107 µM
IC50 = 0.57859 µM
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 24
• Minimize experimentally– Instrument parameters– Plate selection
• Identify, Remove: Auto-focus Classifier– Classify several hundred images by hand– Use image processing measures to extract
quantities that vary based on edges and intensity
– Train a classifier to call out-of-focus wells
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 25
-40 um -20 um 0 um 20 um 40 um
60
70
80
90
100
110
-50 -40 -30 -20 -10 0 10 20 30 40 50
Z Position (um)
Tota
l Nuc
lei
Original
Segmented
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 26
• BayesNet classifier learns differences between in-focus and out-of-focus images
• Algorithm maximizes separation of classes on training set by:
– automatically assigning weights for each predictor
– automatically selecting threshold
Δ IntensitySharpness 2 Δ AreaSharpness 1
Total score
0.080.21 0.18
0.55
yes>1.4?in focus
noout-of-focus
Threshold
Belief parameters
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 27
• Results of BayesNet assessed using cross-validation:
– Algorithm is trained on 9/10ths of the data and tested on remaining 1/10th
– Process is repeated 10 times – Error rates are averaged
• 93% to 98% of the wells were called correctly (for 3 different cell types).
• Nearly all classification errors occurred in 'marginal' category
• Zero-R (baseline classification error)= 32%
•
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 28
Filtering out-of-focus images via machine learning
Millennium Pharmaceuticals Inc.[CMPD] (µM)
0.01 0.1 1 10
AV
G %
Int r
atio
0
10
20
30
40
50
60
70
80
90
100
110
120
130
95%CI = 0.2401 µM
IC50 = 0.58004 µM
Millennium Pharmaceuticals Inc.[CMPD] (µM)
0.01 0.1 1 10
AV
G %
Int r
atio
0
10
20
30
40
50
60
70
80
90
100
110
120
130
95%CI = 0.0711 µM
IC50 = 0.6034 µM
Original Data Filtered Data
Image Acquisition:
AutoFocus
©2006 Millennium Pharmaceuticals, Inc. 29
Image Acquisition
Auto-Focus
Depth of Field Image Corrections
Instrument Choice
Reproducibility
Cell Plating: # sites
Illumination System
©2006 Millennium Pharmaceuticals, Inc. 30
• What is it?– Fluorescence illumination system
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
1 33 65 97 129 161 193 225 257 289 321 353 385 417 449 481 513 545 577 609 641 673
Uniform sample
Image Acquisition:
Illumination system variation
©2006 Millennium Pharmaceuticals, Inc. 31
• Is it a problem? ↓ Cell counting assays↑↑ Cell intensity assays↑↑ Phenotyping
Flat-Field Correction
0.94
0.96
0.98
1
1.02
1.04
1.06
1.08
1.1
1.12
Nuclei Count Nuclei Intensity Tubulin Intensity
%Non FF CorrectedFF Corrected
Image Acquisition:
Illumination system variation
©2006 Millennium Pharmaceuticals, Inc. 32
Image Acquisition
Auto-Focus
Depth of Field Image Corrections
Instrument Choice
Reproducibility
Cell Plating: # sites
Illumination System
©2006 Millennium Pharmaceuticals, Inc. 33
Image Acquisition:
Depth of Field
20x
©2006 Millennium Pharmaceuticals, Inc. 34
- 4.0 um- 2.0 um0.0 um2.0 um4.0 um
Image Acquisition:
Depth of Field
©2006 Millennium Pharmaceuticals, Inc. 35
Z = -4.0um Z = -2.0um Z = 0.0um Z = 2.0um Z = 4.0um
“Best Focus”Single Plane
Image Acquisition:
Depth of Field
©2006 Millennium Pharmaceuticals, Inc. 36
Outof
focus
Multiple optical sections @ 2.0um intervals
In focus
“Best Focus” Image
Tissue Imaging• Challenge: depth of focus of the objective is less than
the flatness of the specimen. This results in portions of a single image plane being out of focus.
©2006 Millennium Pharmaceuticals, Inc. 37
HCS Assay Components
• Dissect components• Using control experiments
– Identify drivers of variability– Minimize them experimentally
or account for them statistically
Wet LabProtocols
ImageAcquisition
Image Analysis
AssayAnalysis
©2006 Millennium Pharmaceuticals, Inc. 38
Image Analysis
Cell Plating: density
Algorithm parameters Signal / Noise
Threshold algorithm
Reproducibility
Magnification
©2006 Millennium Pharmaceuticals, Inc. 39
Image Analysis:
Cell Plating Density• Assess
– Optimal cell plating density to minimize segmentation errors
• DOE:– Serial dilution– DAPI– 6 replicates
XXXXXXXXXXXXH
XXG
XXF
XXE
XXD
XXC
XXB
XXXXXXXXXXXXA
121110987654321
Cell plating density
Cell plating density
Cell plating density
Cell plating density
Cell plating density
Cell plating density
©2006 Millennium Pharmaceuticals, Inc. 40
Image Analysis:
Cell Plating DensityCell Plating vs. Count
0
500
1000
1500
2000
2500
3000
Cell Plating Density
Nucl
ei C
ount
©2006 Millennium Pharmaceuticals, Inc. 41
Image Analysis:
Threshold algorithm
0
200
400
600
800
1000
1200
1400
1 3 5 7 9
Drug Concentration
Nuc
lei C
ount
Object CountStd Area CountAdaptive Count
©2006 Millennium Pharmaceuticals, Inc. 42
Image Analysis:
Signal / Noise• Assess
– Optimal exposure times to minimize acquisition time and maximize Signal/Noise
• Image-based autofocus• Intensity measurements• Segmentation
– DOE:– Uniformity Plate– DAPI– Multiple exposures
XXXXXXXXXXXXH
XXG
XXF
XXE
XXD
XXC
XXB
XXXXXXXXXXXXA
121110987654321
©2006 Millennium Pharmaceuticals, Inc. 43
• Single Uniformity Plate, 9 sites/well
Decreasing exposure time
Image Analysis:
Signal / Noise
©2006 Millennium Pharmaceuticals, Inc. 44
Adaptive Threshold
0
20
40
60
80
100120
140
160
180
200
1 2 3 4 5 6 7 8 9 10 11 12
Row
Nuc
lei C
ount
3.912.772.111.581.33
Fixed Threshold - Auto
020
4060
80100120
140160
180200
1 2 3 4 5 6 7 8 9 10 11 12
Row
Nuc
lei C
ount
Fixed Threshold - User
0
20
40
60
80
100
120
140
160
180
200
1 2 3 4 5 6 7 8 9 10 11 12
Row
Nuc
lei C
ount
• Effect on thresholding
Image Analysis:
Signal / Noise
©2006 Millennium Pharmaceuticals, Inc. 45
Count Nuclei
0
0.2
0.4
0.6
0.8
1
1.2
3.91 2.77 2.11 1.58 1.33
eSNR
% o
f 160
*: p<10-100, Welch’s two-tailed t-testError bars are standard deviations
*
*
Image Analysis:
Signal / Noise
©2006 Millennium Pharmaceuticals, Inc. 46
• What objective to use?– Segmentation accuracy– Dose response curve re-sampling
Image Analysis:
Magnification
©2006 Millennium Pharmaceuticals, Inc. 47
10x
20x
FKHR-GFPDNA
Image Analysis:
Magnification
©2006 Millennium Pharmaceuticals, Inc. 48
10x
20x
FKHR-GFPDNA
Image Analysis:
Magnification
©2006 Millennium Pharmaceuticals, Inc. 49
Millennium Pharmaceuticals Inc.[CMPD] (µM)
0.01 0.1 1 10
AV
G %
Int r
atio
0
10
20
30
40
50
60
70
80
90
100
110
120
130
95%CI = 0.0744 µM
IC50 = 0.65515 µM
Millennium Pharmaceuticals Inc.[CMPD] (µM)
0.01 0.1 1 10
AV
G %
Int r
atio
0
10
20
30
40
50
60
70
80
90
100
110
120
130
95%CI = 0.0372 µM
IC50 = 0.64973 µM
10x 20x
Image Analysis:
Magnification
©2006 Millennium Pharmaceuticals, Inc. 50
• Assess– Effect of variability of user-defined parameters– Different software algorithms (vendors)
• DOE:– Dose Response– Multiple algorithmparameters
XXXXXXXXXXXXH
XCXG
XCXF
XCXE
XCXD
XCXC
XCXB
XXXXXXXXXXXXA
121110987654321
dose response
dose response
dose response
dose response
dose response
dose response
Image Analysis:
Algorithm parameters
©2006 Millennium Pharmaceuticals, Inc. 51
• User variability?
Image Analysis:
Algorithm parameters
©2006 Millennium Pharmaceuticals, Inc. 52
• Single Plate, 9 sites/well, dose-response• 30 variations of DNA, pH3 threshold
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0499µMIC30= 0.65911µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0499µMIC30= 0.65911µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0594µMIC30= 0.65607µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
TEST
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0462µMIC30= 0.60712µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0652µMIC30= 0.65776µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.05µMIC30= 0.61013µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.1432µMIC30= 0.66916µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0953µMIC30= 0.66289µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.198µMIC30= 0.66241µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0901µMIC30= 0.66033µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0605µMIC30= 0.65709µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0439µMIC30= 0.65786µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0439µMIC30= 0.65786µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0634µMIC30= 0.65637µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
TEST
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0442µMIC30= 0.60945µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0664µMIC30= 0.65259µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0448µMIC30= 0.60799µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.1733µMIC30= 0.66231µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0858µMIC30= 0.65996µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.1274µMIC30= 0.66117µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0707µMIC30= 0.65613µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.046µMIC30= 0.65204µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0476µMIC30= 0.65429µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0463µMIC30= 0.6547µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0463µMIC30= 0.6547µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0627µMIC30= 0.6526µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
TEST
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0404µMIC30= 0.6082µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0999µMIC30= 0.66593µM
%Nuclear count in HT29 cells @ 24 hour
Millennium Pharmaceuticals Inc.
[CMPD] (µM)0.1 1 10
AV
G %
Mito
tic
0
10
20
30
40
50
60
70
80
90
100
95%CI= 0.0685µMIC30= 0.65928µM
IC30 (µM)
0.1000.2000.3000.4000.5000.6000.7000.8000.9001.000
Algorithm Parameters (DNA, pH3 adaptive threshold)
IC30
(µM
)
DNApH3
Image Analysis:
Algorithm parameters
©2006 Millennium Pharmaceuticals, Inc. 53
HCS Assay Components
• Dissect components• Using control experiments
– Identify drivers of variability– Minimize them experimentally
or account for them statistically
Wet LabProtocols
ImageAcquisition
Image Analysis
AssayAnalysis
©2006 Millennium Pharmaceuticals, Inc. 54
Assay Analysis
# of replicates
Transfer metrics Variance Component Analysis
(Plate-Plate / Day-Day)
Instrument Choice
Reproducibility
96 vs 60 wells
# images / well
©2006 Millennium Pharmaceuticals, Inc. 55
Assay Analysis
# of replicates
Transfer metrics Variance Component Analysis
(Plate-Plate / Day-Day)
Instrument Choice
Reproducibility
96 vs 60 wells
# images / well
©2006 Millennium Pharmaceuticals, Inc. 56
Assay Analysis:
Variance Component Analysis
• Assess variability– Row / column– Plate-plate– Day-day
• DOE:– Single drug concentration (Uniformity plates)– 4 plates x 3 days
Day 1 Day 2 Day 3
©2006 Millennium Pharmaceuticals, Inc. 57
96 wells 60 wells
• Inner 60 wells had median CV which was 35% lower than that for all 96 wells
Assay Analysis:
Uniformity Data
©2006 Millennium Pharmaceuticals, Inc. 58
Single Plate
12 Plates •Variability among days was largest component followed by plate•Replicate dose response curves should be run on different plates on different days to account for largest amount of variability.
Total Mean: 27.7Total Variance: 32.0 (std dev: 5.7)Coefficient of Variation: 20.4%
Assay Analysis:
Uniformity DataVariance Component Analysis
©2006 Millennium Pharmaceuticals, Inc. 59
Assay Analysis:
Re-sampling
• Determine– How many sites / well?– How many replicate wells?– Is the assay robust?– Does assay perform well across facilities?
• Sample– Uniformity Plates– Control Compound– Multiple Compounds
©2006 Millennium Pharmaceuticals, Inc. 60
Bootstrapping
Population
• Method for estimating the sampling distribution of an estimator by re-sampling with replacement from the original sample
©2006 Millennium Pharmaceuticals, Inc. 61
Bootstrapping
Population Sample
• Method for estimating the sampling distribution of an estimator by re-sampling with replacement from the original sample
©2006 Millennium Pharmaceuticals, Inc. 62
Bootstrapping
Population Sample
• Method for estimating the sampling distribution of an estimator by re-sampling with replacement from the original sample
©2006 Millennium Pharmaceuticals, Inc. 63
Bootstrapping
Population Sample
• Method for estimating the sampling distribution of an estimator by re-sampling with replacement from the original sample
©2006 Millennium Pharmaceuticals, Inc. 64
Bootstrapping
Population Sample
• Method for estimating the sampling distribution of an estimator by re-sampling with replacement from the original sample
©2006 Millennium Pharmaceuticals, Inc. 65
Assay Analysis:
Control Compound• Assess
– Effect of number of replicate curves upon EC30 estimates
– Effect of number of images per well upon EC30 estimates
• DOE:– 6 dose response
curves– 5 plates
XXXXXXXXXXXXH
XCXG
XCXF
XCXE
XCXD
XCXC
XCXB
XXXXXXXXXXXXA
121110987654321
dose response
dose response
dose response
dose response
dose response
dose response
©2006 Millennium Pharmaceuticals, Inc. 66
Assay Analysis:
Control Compound Data
©2006 Millennium Pharmaceuticals, Inc. 67
Re-sampling Results
Distribution of EC30 values
©2006 Millennium Pharmaceuticals, Inc. 68
Error bars are 95% CI All CVs are less than 12%
-Results incorporate differences among plates and there were fourobservations with high values.
Re-sampling Results:
Determine # wells
©2006 Millennium Pharmaceuticals, Inc. 69
Going from 6 images to 9 has almost as much impact on standard error as going from 9 to 16.
10uM DMSO
Re-sampling Results:
Determine # sites/well
©2006 Millennium Pharmaceuticals, Inc. 70
Assay Analysis
# of replicates
Transfer metrics Variance Component Analysis
(Plate-Plate / Day-Day)
Instrument Choice
Reproducibility
96 vs 60 wells
# images / well
©2006 Millennium Pharmaceuticals, Inc. 71
Assay Analysis:
Transfer
• Assess– Compare re-sampled EC30 estimates to estimates
generated in Oncology Discovery• DOE
– 4 compounds– 3 replicates / compound– Each compound was run on 4 plates = 12 curves– All plates run in two facilities
©2006 Millennium Pharmaceuticals, Inc. 72
Four compound re-sampling results
•A and B were not significantly different, C and D were approximately 60% higher
• Three wells from each concentration were sampled.• Fit dose response curve and calculated EC30.• Total of 250 curves• Examine distribution of EC30 values
Compound
Log(
EC30
)
Assay Analysis:
Transfer
©2006 Millennium Pharmaceuticals, Inc. 73
• Uniformity Plates
• Single Compound
Assay Analysis:
Instrument selection
Discovery-1 Opera
* determination based on signal/noise, speed, robustness, availability
• Signal/Noise, acquisition time, CVs
• Z’, EC50 CVs, confidence interval
©2006 Millennium Pharmaceuticals, Inc. 74
Overall Assay Evaluation
• ~2 months• Confidence
– Immunofluorescence staining protocol
– Automation / liquid handling– Instrument / Instrument QC– Software Algorithm – Robust assay– Transfer from Discovery to
Platform Group
Platform validation
Variance ComponentsAnalysis
Bootstrapping on wellsand images
Bootstrapping withcontrol compound
Bootstrapping withreference panel
Identify sources of noise
Format assay
Determine robustness
Make site comparisons
Establish decision criteriaDecision criteria
simulation
Process controls
Platform validation
Variance ComponentsAnalysis
Bootstrapping on wellsand images
Bootstrapping withcontrol compound
Bootstrapping withreference panel
Identify sources of noise
Format assay
Determine robustness
Make site comparisons
Establish decision criteriaDecision criteria
simulation
Process controls
©2006 Millennium Pharmaceuticals, Inc. 75
Summary
• Look at the images
• Run uniformity plates to identify and minimize noise (biology, instrument, and algorithm)
• Use analysis modules throughout process
©2006 Millennium Pharmaceuticals, Inc. 76
Process Deliverables
• Development Guidelines– QC metrics: instruments, algorithm– IF staining protocol optimization– Image analysis optimization
• Production Guidelines – Statistical analysis for transfer– Ongoing QC metrics
©2006 Millennium Pharmaceuticals, Inc. 77
Acknowledgements• Molecular & Cellular Oncology
– Ben Amidon– Jeff Ecsedy– Natalie Roy D’Amore– Liz Carideo– Kara Hoar – Claudia Rabino– Michelle Tighe-Nestor, – Denise Driscoll– Deborah Wysong
• Cancer Pharmacology– Arijit Chakravarty– Katherine Galvin
• Lead Discovery– John Ringeling– John Donovan
• Computational Sciences– Sudeshna Das– Mike Pickard
©2006 Millennium Pharmaceuticals, Inc. 78
Breakthrough science. Breakthrough medicine.SM