CASE-STUDY-

VALIDATION of PCR based methodology

Beata Surmacz-Cordle Senior Analytical Development Scientist

UK RMP Pluripotent Stem Cell Platform Validation Workshop 2nd June 2016

RT-qPCR assay for detection of persistence of genetically modified T cells in autologous cell therapy

Introduction to Case Study

RT-qPCR assay Validation

Summary

Intro

CASE STUDY –Modified T cell receptor (TCR) autologous Cell Therapy

HarvestT Cells

Genetically engineer to modify specificity

T Cell

Infuse back into patient

Anti-Tumour Activity

Genetically modified T Cell Receptor Therapy

Patient monitoring:Multiple Clinical Trial time points

Start -1 month …few years

T Cell

T Cell

T Cell

T Cell

T Cell

T Cell

T Cell

Day 0

Persistence of TCR modified cells – RT-qPCR assayPersistence of TCR modified cells – RT-qPCR assay

How many TCR modified cells

per 1x10E6 WBC?

RT-qPCR assay evaluates the persistence of genetically modified T cell receptor cells directed against blood leukaemic cells;

Presence of genetically modified TCR expressing cells is analysed using RT-qPCR assay in samples pre and post infusion;

RT-qPCR is a clinical assay (Good Clinical Laboratory Practise) that is performed multiple times over few years testing numerous patient blood samples;

Detection of RNA or DNA?

RT-qPCR assay provides quantitative information about the number of cells that are actively transcribing the modified TCR receptor;

Assay is RNA based (RT +qPCR)

Persistence of TCR modified cells – RT-qPCR assay principle

SYBR Green based assay with TCR specific and house keeping primers (2 sets of primers)

Standard Curve based readout (Serial dilution of Transduced cells in mock Transduced cells)

Persistence of TCR modified cells – RT-qPCR assay principle

PBMC Preparation RNA isolation cDNA synthesis -RT qPCR assay

100K

10K

1K

100

10 cells

XTCR Cell Number

Ct

Validation?

PCR assay workflow to Validation

Optimisation (evidence)

Qualification (gain confidence and learn about assay limits)

Validation (demonstrate that it is suitable for intended purpose!)

• Gap Analysis

• Analytical Transfer

Process Transfer

• BSD• P-Diagram

Process Dissection

• FMEA• COGS

options

Risk vs COGS

• Priority Grid• Implementation

Plan

Development Plan

Ste

p 1

.1S

tep

1.2

Day 0120 minutes

Cell Product

Raw Material

Consumable

Name the step in bold

Put in detail

Add subprocesses if

required

Cell Product

Sample

Waste

Equipment

Decision Rejected ProductNo

Yes

Tissue Culture(Grade B)

Final Product

Name the step in bold

Put in detail

Analytical Lab(Unclassified)

Autoclave(Unclassified)

Analytical Lab(Unclassified)

Assay or Donor data

SampleAssay or Donor

data EquipmentDay 0

10 minutes

Project Code: Template100x04Last Modified: 24/02/2015Printed: 24/02/2015

Qu

ali

ty

Attrib

ute

1

Qu

ali

ty

Attrib

ute

2

Qu

ali

ty

Attrib

ute

3

Qu

ali

ty

Attrib

ute

4

Qu

ali

ty

Attrib

ute

5

Qu

ali

ty

Attrib

ute

6

Ba

tch

Co

mp

letio

n

Qu

ali

ty

Attrib

ute

7

Qu

ali

ty

Attrib

ute

8

Qu

ali

ty

Attrib

ute

9

Qu

ali

ty

Attrib

ute

10

Qu

ali

ty

Attrib

ute

11

Qu

ali

ty

Attrib

ute

12

Qu

ali

ty

Attrib

ute

13

Qu

ali

ty

Attrib

ute

14

Qu

ali

ty

Attrib

ute

15

Qu

ali

ty

Attrib

ute

16

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Process Operation 1 Input 1 C 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 17 17

Process Operation 1 Input 2 N 1 3 1 1 9 1 3 1 1 1 1 1 1 1 1 1 1 29 29

Process Operation 1 Input 3 X 1 1 9 9 9 1 1 1 Q 1 1 1 1 1 1 1 1 40 40

We

ig

hte

d S

co

re

Process Operation Process Parameters C/N/X

Quality Attributes (and weightings)

Sc

ore

RT-qPCR assay analysis tools (‘paper based’)

• Analysis showed that assay optimisation is required rather validation

RT-qPCR assay Optimisation and Qualification

Study 1 : Vicell dynamic range for automated T cell count;

Study 2 : Identification of RNA isolation method that gives highest yield;

Study 3 : Assessment of correct primer concentration to improve signal to noise ratio without loss of sensitivity;

Op

tim

isa

tio

n

Selection of Assay Pass/Fail criteria:

• (Noise to Signal ratio= Ct value cut off)• Number of technical replicates• Appropriate Positive and Negative controls (Transduced and mock Transduced

cells, Plasmids for both assays, NTC) Qu

ali

fica

tio

n

PCR assay workflow to Validation

Optimisation (evidence)

Qualification (gain confidence and learn about assay limits)

Validation (demonstrate that it is suitable for intended purpose!)

RT-qPCR assay validation

ICH, MIQE guidelines and GCLP..

qPCR Validation Importance

Evidence of optimistion Desirable

Specificity (gel, sequence, melt, digest) Essential

For SYBR Green, Cq of the NTC Essential

Standard curves with slope and y-intercept Essential

PCR efficiency calculated from slope Essential

Confidence interval for PCR efficiency or

standard error Desirable

R2 of standard curve Essential

Linear dynamic range Essential

Cq variation at lower limit Essential

Confidence intervals throughout range Desirable

evidence for limit of detection Essential

If multiplex, efficiency and LOD of each assay Essential

ICH Quideline Q2(R1) 1994

The MIQE Guidelines: Minimum Information for Publication of Quantitative Real-Time PCR Experiments. Clinical Chemistry 55:4 611–622 (2009)

Good Clinical Laboratory Practise (GCLP) by RQA, V2 2012

Parameters tested for RT-qPCR assay validation

Assay Linearity

Assay Sensitivity

Assay Accuracy

Assay Precision

Assay Specificity

Assay Robustness

Acceptance criteria must be well defined before validation for all parameters!

Linearity_1

.. the ability to produce test results that are directly, or by a well-defined mathematical transformation, proportional to the concentration of analyte in samples within a given range (ICH Q2, R1)

1 2 3 4 5 6 7 8 9 10 11 12

A 105 105 105 TD TD TD 105 105 105 TD TD TD

B 104 104 104 mockNTD mockTD mockTD 104 104 104 mockNTD mockTD mockTD

C 103 103 103 NTC NTC NTC 103 103 103 NTC NTC NTC

D 102 102 102 Control

plasmid

Control

plasmid

Control

plasmid102 102 102 Control

plasmid

Control

plasmid

Control

plasmid

E 10 10 10 10 10 10

F 1 1 1 1 1 1

G NoRT TD NoRT TD NoRT TD NoRT TD NoRT TD NoRT TD

H NoRT

mock Td

NoRT

mock Td

NoRT

mock Td

NoRT

mock Td

NoRT

mock Td

NoRT

mock Td

• R2 (1+/-0.1)• Slope and Efficiency (-3.6 to -3.1; 90-110)• (Residual Plot)

Linearity assessment

Acceptance criteria

Cell number

Ct

100K

10K1K

10010 cells

1 cell

• 4 independently generated standard curves (log dilution of transduced cells);

• RT-qPCR run on 4 different days;• All samples run in multiple replicates;

Linearity_2 Results

0 1 2 3 4 5 6

-0 .5

0 .0

0 .5

Ct(

pre

dic

ted

)-C

t (m

ea

su

red

)

L o g 1 0 [C e ll n u m b e r ]

C u rv e 1

C u rv e 2

C u rv e 3

C u rv e 4

Measurement

parameter

Desirable value

range

Results Standard Deviation

between data sets

Pass/Fail

R2* 1 +/- 0.1 0.993 0.007 PASS

Slope -3.6 to -3.1 -3.409 0.348 PASS

Efficiency (%) 90-110 98.09 9.68 PASS

* RT-qPCR assay meets desired criteria over a range of 102-105 cells

Residual Plot

The RT-qPCR assay fits within the range of acceptable values for

assay linearity between 102-105 cells

Sensitivity_1

… the ability of the assay to consistently quantify specimens that contain barely measureable analyte.

• 4 independently generated standard curves from 100K to 1 cell (log dilution)

• 10 replicates! (more might be required)

• 90% positive detection calls (Ct<37)• CV%<5

Sensitivity assessment

Acceptance criteria

100K

10K1K

100?10 cells?

LOQ – Limit of Quantification- the lowest actual amount of an analyte that can be reliably detected and at which the total error meets the laboratory’s requirements fro accuracy, so it can be quantifiedLOD – Limit of Detection the lowest amount of analyte in a sample that can be detected with a stated probability of 95% Confidence Intervals

1 cell?

0 1 2 3 4 5 6

-0 .5

0 .0

0 .5

Ct(

pre

dic

ted

)-C

t (m

ea

su

red

)

L o g 1 0 [C e ll n u m b e r ]

C u rv e 1

C u rv e 2

C u rv e 3

C u rv e 4

Residual Plot

LOQ and LOD are not trivial for PCR assays!

LOQ may be limited by the linearity of the STD curve (check for residuals plot)

LOQ should be within linear range

LOQ may be set by the spread of replicates

LOQ may never be lower than LOD

LOD CV >95% is important

LOD in ‘real life’ depends on:

- Extraction yield

- Loses during handling, transfer and storage

- Losses during dispensing

- Reverse Transcription yield

- Inhibition and interference in qPCR reaction

To determine LOD for full procedure you need to start from the beginning so from RNA extraction..

• Signal statistically different than background (no value readout for background =0)Difficult to apply to PCR assays

Sensitivity_2 Results

LOQ for RT-qPCR assay is at 100 cells

100K

10K

1K

10010 cells

Measurement parameter Acceptance value LOQ

results for TCR specific assay

Control

results for house keeping assay

TCR modified cell number 100 10 100 10

% positive detection calls (Ct

<37)90 97.5 40 100 100

CV (%) <5 2.87 1.44 1.91 1.32

Pass/ fail PASS FAIL PASS PASS

10K

100K

1K

100

Accuracy

… is defined as the closeness of the test results obtained by an analytical method to conventional true value or an accepted reference value and the mean value found from replicate or repeated assessment in the assay

• In this particular assay there is no ‘true value’ to test for accuracy, therefore accuracy cannot be measured

• Cell number readout is depended on multiple factors (i.e. flow cytometry assessed transduction efficiency as well as viable cell count and pipetting..)

AccuracyAssessment

Precision_1

… the closeness of agreement (degree of scatter) between series of measurements obtained from multiple sampling of the same homogenous samples.Precision is solely related to the random error and has no relation to accuracy. An assay can be precise but also inaccurate….!

• Repeatability – intra assay precision (data from linearity assessment on one day)

• Intermediate Precision- inter assay precision (data from linearity assessment on different days +2nd operator)

• Reproducibility – precision between laboratories (part of robustness study)

• Repeatability – SD<1CT and CV%<5• Intermediate precision - SD<1CT and CV%<5

Precision assessment

Acceptance criteria

Precision_2 Results

Measurement

parameter

Acceptance value

range

results Standard Deviation

between data sets

Pass/Fail

Standard Deviation <1.00 Ct 0.30 0.37 PASS

CV (%) <5.00 0.876 0.082 PASS

Measurement

parameter

Acceptance value

range

Results between data sets Pass/Fail

Standard Deviation <1.00 Ct Operator 1: 0.30 ± 0.37 Ct

Operator 2: 0.25 ± 0.27 Ct

PASS

CV (%) <5.00 Operator 1: 0.876 ± 0.892

Operator 2: 0.769 ± 0.736

PASS

Repeatability results are reported as mean values from data collected from 3 standard curves run on one day

Intermediate precision results are reported as the mean values of data generated by operator 1 from all previously analysed standard curves compared with the results from 1 standard curve over 4 days from operator 2.

The RT-qPCR assay fits within the range of acceptable values

for Repeatability and Intermediate precision

Parameters for RT-qPCR assay validation

Linearity

Sensitivity

Accuracy

Precision

Specificity

Robustness

Specificity_1

… is defined as the ability to distinguish known negative samples from known positive samples

• Specificity for Transduced cells as opposed to mock transduced cells• Target amplification specificity:

- melt curve comparison between control plasmid and tested assays- primer specificity against BLAST

• TCR target present in Transduced cells and absent in mock transduced cells (highest cell number used in the assay)

• TCR amplified product Tm=84-86°C and ABL amplified product Tm=78-81°C - Single peak only

• Not binding to endogenous TCR receptor as well as other abundant human gene targets Query cover=100% and E value<<2 (BLAST)

Specificity assessment

Acceptance criteria

Measurement

parameter

Acceptance

value range

Results Standard

deviation

between data

sets

Pass/

Fail

TCR plasmid melt peak (plasmid) 84-86°C 84.78°C 0.012 PASS

TCR assay melt peak (Standard curve) 84-86°C 85.06°C 0.097 PASS

House keeping plasmid melt peak (plasmid) 78-81°C 79.21°C 0.038 PASS

House keeping melt peak (Standard curve) 78-81°C 79.71°C 0.068 PASS

Specificity_2 Results

Measurement

parameter

Acceptance value range Results Pass/Fail

Transduced cells Amplified Ct<37 20-25Ct PASS

Mock transduced cells Not amplified <37 >40Ct PASS

TCR specific product

House keeping specific product

Measurement

parameter

Acceptance value range Results Pass/Fail

Blast search for House keeping assay Query cover =100% E<<2 Q=100%

E=0.0006

PASS

Blast search for TCR specific assay Query cover =100% E<<2 Q=85%

E=1.4

FAIL

Plasmid alignment for TCR specific

assay

Full alignment of primers

sequence with plasmid

YES PASS

The RT-qPCR assay fits within the range of acceptable values for Specificity

Robustness_1

… is a measure of assay capacity to remain unaffected by small, but deliberate variations in method parameters and provides an indication of its reliability

• Different laboratory (part of technology transfer)• Different operator (2 operators)• Different PCR instrument (Quantstudio 6/7 and Quant studio 12K

flex)

• R2 1+/-0.1• Slope -3.6 to -3.1• Efficiency 90 to 110%• Sensitivity the same across runs

Specificity assessment

Acceptance criteria

Acceptance

Value Range

Site 1 Run1 Site 1 Run2 Site 2 Run1 Site 2 Run2 Site 2 Run3

Pass/failOperator 1 Operator 2 Operator 1 Operator 2 Operator 2

Instrument 1Instrument

1

Instrument

2

Instrument

2

Instrument

2

Slope -3.6 to -3.1 -3.1 -3 -3.4 -3.49 -3.38

PASSR2 1+/-0.1 0.968 0.922 0.99 1 0.99

Efficiency 90-110% 109.7 109.5 96.84 93.43 96.63

SensitivityThe same in

all runs100 100 100 100 100 PASS

Robustness_2 Results

The RT-qPCR assay fits within the range of acceptable values for Robustness

RT-qPCR assay Validation Summary

Measurement

parameter

Acceptance value range Results Pass/ Fail

Linearity

Slope -3.6 to -3.1 -3.409 PASS

Efficiency% 90-110 98.09 PASS

R2 1 +/- 0.1 0.993 PASS

Sensitivity

TCR specific assay

% positive detection of

100 cells (Ct <37)90 97.5 PASS

CV (%) <5 2.87 PASS

Sensitivity

House keeping gene

% positive detection of

100 cells (Ct <37)90 100 PASS

CV (%) <5 1.91 PASS

Precision- repeatability

Standard Deviation <1.00 Ct 0.37 PASS

CV(%) <5.00 0.082 PASS

Intermediate Precision

Standard Deviation <1.00 Ct

Operator 1: 0.30 ± 0.37 Ct

Operator 2: 0.25 ± 0.27 Ct PASS

CV (%) <5.00

Operator 1: 0.876 ± 0.892

Operator 2: 0.769 ± 0.736 PASS

Specificity

WT1 melt peak 84-86°C 85.06 PASS

ABL melt peak 78-81°C 79.71 PASS

Robustness

Slope -3.6 to -3.1 Values in range PASS

Efficiency% 90-110 Values in range PASS

R2 1 +/- 0.1 Values in range PASS

RT-qPCR controls used throughout Validation

Positive Controls

- Plasmid vector encoding genetically modified TCR nucleotide sequence

- Plasmid vector encoding housekeeping gene nucleotide sequence

- 1x10E6 Transduced cells (stock)

Negative Controls

- NTC – no template control (reagent contamination and primers dimers)

- NO-RT (gDNA amplification)

- 1x10e6 Mock Transduced cells (non specific primers binding)

PCR assay workflow to Validation

Optimisation (evidence)

Qualification (gain confidence and learn about assay limits)

Validation (demonstrate that it is suitable for intended purpose!)

RT-qPCR assay Validation Summary

To validate RT-qPCR assay an understanding of wider assay context is required e.g. patient samples? (GCLP) or manufactured product testing (GMP);

Validation has to be specific to the chemistry used (SYBR Green, TaqMan, multiplex PCR);

Validation requires evidence of Optimisation and Qualification;

Parameters to be assessed during RT-qPCR validation are : Linearity, Sensitivity, (LOD and LOQ not trivial!) Accuracy, Precision, Specificity, Robustness

Validation must have PREDEFINED ACCEPTANCE CRITERIA

Think carefully about requirement for internal positive controls to control for PCR inhibition e.g. plasmids or synthetic templates;

RT-qPCR Validation Summary

Other parameters to be considered for PCR assay validations

Assay Specificity : Product Sequencing +gel run

Assay Specificity: Primer analysis (Primer 3 or other on line tools)

Assay sensitivity: RNA/DNA extraction spike control to control for extraction efficiency

Validation can take 2-4 months! Think about reagent stability (aliquoting and lot to lot comparison studies)

Detection of RNA or DNA? RT efficiency!

Consider ‘background’ of samples that are analysed (is internal positive control required?)

MIQE RT-PCR validation analysis = GenEx software

Cell and Gene Therapy Catapult12th Floor Tower Wing Guy’s HospitalGreat Maze Pond London SE1 9RT

+44(0)20 3728 9500

Cell and Gene Therapy Catapult is a trading name of Cell Therapy Catapult Limited, registered in England and Wales under company number 07964711.