Oliver Stegle and Karsten Borgwardt - ETH Z

Oliver Stegle and Karsten Borgwardt: Computational Approaches for Analysing Complex Biological Systems, Page 1

GP ApplicationsOliver Stegle and Karsten Borgwardt

Machine Learning andComputational Biology Research Group,

Max Planck Institute for Biological Cybernetics andMax Planck Institute for Developmental Biology, Tübingen

Outline

O. Stegle & K. Borgwardt GP Applications Tubingen 1

Application1: modelling physiological time series

Outline

Application1: modelling physiological time seriesOverviewGaussian process prior for heart rateResults

Application 2: differential gene expressionOverviewA Gaussian process two-sample testExperimental Results on ArabidopsisDetecting Temporal Patterns of Differential Expression

Application 3: Modeling transcriptional regulation using Gaussianprocesses

Summary

Motivation

I Human heart rate is an important physiological trait.

I Measurement over long periods only viable with poor sensors.

I Motivation Gaussian process model for heart rate.

Motivation

I Human heart rate is an important physiological trait.

I Measurement over long periods only viable with poor sensors.

I Motivation Gaussian process model for heart rate.

The problemDataset

4 days of heart data

0 1000 2000 3000 4000 5000 6000 70000

Features

I Different noise sources

I Two time scales, 24hrhythms

I Asymmetric around themean

I Auxiliary variablesindicative of noise

The problemDataset

4 days of heart data

∆HRmin

∆HRmax

1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 60000

Features

The problemDataset

zoomed view

2100 2200 2300 2400 2500 2600

Features

Application1: modelling physiological time series Overview

modelThe Inference Model

z1noisyclean very noisy

noiseaux. data

time step 1

z2noisyclean very noisy

noiseaux. data

time step 2

znnoisyclean very noisy

noiseaux. data

time step n

I (a) Gaussian process prior on latent heart rate

I (b) Clustering of auxiliary data to extract noise classes

I (c) Heavy tailed noise model taking classes into account

Application1: modelling physiological time series Gaussian process prior for heart rate

A GP prior for heart rate

I Covarianc function for shortrange fluctuations

I Long-range perdiodic signal

I Sum of (a) and (b)

I Log transformation (asymmetry)

0 50 100 150

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 40000

t / min

0 50 100 150

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 40000

t / min

0 50 100 150

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 40000

t / min

0 50 100 150

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 4000

−200

−100

t / min

0 1000 2000 3000 40000

t / min

I Short-range covariance

CS(x, x′) = C1 Matern3/2(x, x

′, δS)

I Periodic covariance

CL(x, x′) = C0 exp

[− (x− x′)2

[−2 ·

sin2( 2πpL · (x− x′))

]I Total covariance

C(x, x′) = CL(x, x′) + CS(x, x

I Non-linear transformation, reflecting strict positivity and asymmetryof heart rate

y∗t = logβ(yt)

′, δS)

[− (x− x′)2

[−2 ·

sin2( 2πpL · (x− x′))

]I Total covariance

C(x, x′) = CL(x, x′) + CS(x, x

y∗t = logβ(yt)

′, δS)

[− (x− x′)2

[−2 ·

sin2( 2πpL · (x− x′))

]I Total covariance

C(x, x′) = CL(x, x′) + CS(x, x

y∗t = logβ(yt)

′, δS)

[− (x− x′)2

[−2 ·

sin2( 2πpL · (x− x′))

]I Total covariance

C(x, x′) = CL(x, x′) + CS(x, x

y∗t = logβ(yt)

Clustering of auxiliary data

I Every datapoint can be member in one of K clusters.

I Use clustering approach to determine cluster membership

πn = {πn,1, . . . , πn,K} where

K∑k=1

πn,k = 1

Clustering of auxiliary data

I Every datapoint can be member in one of K clusters.

I Use clustering approach to determine cluster membership

πn = {πn,1, . . . , πn,K} where

K∑k=1

πn,k = 1

Robust noise model

I Remember: standard robust noise model, accounting for outliers

p(yn | fn) = πokN(yn∣∣ fn, σ2)+ (1− πok)N

(yn∣∣ fn, σ2∞)

I Here: clustering results (auxiliaries S) encode useful information.

y1|f1 S1

y2|f2 S2

yN |fN SN

I Generalize likelihood K noise components, one for each cluster anduse data-specific mixture probabilities πn

p(yn | fn) =K∑k=1

πk,nN(yn∣∣ fn, σ2k)

Robust noise model

(yn∣∣ fn, σ2∞)

y1|f1 S1

y2|f2 S2

yN |fN SN

p(yn | fn) =K∑k=1

Robust noise model

(yn∣∣ fn, σ2∞)

y1|f1 S1

y2|f2 S2

yN |fN SN

p(yn | fn) =K∑k=1

Application1: modelling physiological time series Results

Regression for Heart Data

Single data set

I Clusteringcolor-coded andHinton diagrams

I Three clusters- g,b,r-valuesfor responsi-bilities

I Noise parameters{σc}3c=1

optimised alongwith other hyperparameters

Regression for Heart Data

Double data-set

I Two sensors forone heart

I GPs overlap wellwithin error-bars

I Lower panel:difference plotand error bars

Fill-in testMotivation

I Evaluate predictive performance to benchmark alternative modelsI Fill-in test:

I Model is trained on a subset of the data to predict the remainder.I Log probability as criteria rewards models with appropriately sized error

Block size 1 minute

0.1 0.2 0.3 0.4 0.5 0.6−1

Fraction of missing data

(i)(ii)(iii)

I 5 different models comparedI (i) baselineI (ii) GP with ksI (iii) GP with ks + klI (iv) as (iii) but with Tlost

threshold noise modelI (v) as (iv) but with full robust

noise model

Block size 60 minute

Block size 60 minutes

0.1 0.2 0.3 0.4 0.5 0.6−1

−0.5

Fraction of missing data

I 5 different models comparedI (i) baselineI (ii) GP with ksI (iii) GP with ks + klI (iv) as (iii) but with Tlost

threshold noise modelI (v) as (iv) but with full robust

noise model

I Larger blocks removed rewardslong range covariance

Outline

Application 2: differential gene expression

Outline

Summary

Application 2: differential gene expression Overview

Response to External Stimuli

I Organisms such as plants respond toexternal stimuli:

I Heat/ColdI StarvationI Biotic stresses (fungus)I ...

I Changes in observed gene expressionlevels.

Differential Gene Expression

I An important goal is the identificationof differentially expressed genes.

I Identification of involved regulatorycomponents.

I Uncovering parts of the biologicalnetwork.

Differential Gene Expression

I An important goal is the identificationof differentially expressed genes.

I Identification of involved regulatorycomponents.

I Uncovering parts of the biologicalnetwork.

Differential Gene Expression in Time SeriesTime Series

I The response to external stimuli is a dynamic process.

I Hence the response should be studied as a function of time.

Differential Gene Expression in Time SeriesTime Series

I The response to external stimuli is a dynamic process.

I Hence the response should be studied as a function of time.

10 20 30 40Time/h

TreatmentControl

Non-differentially expressed

10 20 30 40Time/h

TreatmentControl

Differentially expressed

Differential Gene Expression in Time SeriesChallenges

I Time series expression profiles vary smoothly over time.

I Noisy observations – outliers.

I Multiple replicates.

I Few observations.

I Temporal patterns (intervals) of differential gene expression.

10 20 30 40Time/h

TreatmentControl

Non-differentially expressed

10 20 30 40Time/h

TreatmentControl

Differentially expressed

Application 2: differential gene expression A Gaussian process two-sample test

Gaussian process ModelModel comparison

I The basic idea – a comparison of two models:I The shared model: Expression levels are explained by a single process.I The independent model: Expression levels are explained by two

separate processes.

Gaussian process modelBayesian Network: Shared Model

I Data in conditions A and Bobserved at N time points withR replicates.

I A Gaussian process priorincorporates beliefs aboutsmoothness.

I Noise is is modeled separatelyper-replicate, σA/Br .

Gaussian process modelBayesian Network: Both Models

I The independent model follows in an analogous manner.

Shared model HS Independent model HI

Gaussian Process ModelInference

I Models are compared using the Bayes factor

Score = log

Independent model︷︸︸︷P (DA,DB |HI)

P (DA,DB |HS)︸︷︷︸Shared model

I Writing out the GP models explicitly leads to

Score = logP (YA |HGP,T

A, )P (YB |HGP,TB, )

P (YA ∪YB |HGP,TA ∪TB, ).

(YA/B : expression levels in conditions A and B; TA/B : observation time points)

Score = log

A,θI)P (YB |HGP,T

B,θI)

P (YA ∪YB |HGP,TA ∪TB,θS).

Score = log

A, θI )P (YB |HGP,T

B, θI )

P (YA ∪YB |HGP,TA ∪TB, θS ).

Gaussian Process ModelShared Model

I Given observed data from both conditions D = {YA/B,TA/B} theposterior distribution over latent function values f is

P (f |Y,T,θK,θL) ∝N (f |0,KT(θK))

c∈{A,B}

R∏r=1

N∏n=1

pL(ycr,tn | ftn ,θL),

I Covariance funciton (kernel)

I Noise model

I Hyperparameters θS = {θK,θL} (length scale, noise levels)

I For Gaussian noise, pL(ycr,t | f cr,t,θL) = N

(ycr,t

∣∣∣ f cr,t, (σcr)2), the

model is tractable in closed form.

P (f |Y,T,θK,θL) ∝N(f∣∣∣0, KT(θK)

∏c∈{A,B}

R∏r=1

N∏n=1

pL(ycr,tn | ftn ,θL),

I Noise model

(ycr,t

P (f |Y,T,θK,θL) ∝N(f∣∣∣0, KT(θK)

∏c∈{A,B}

R∏r=1

N∏n=1

pL(ycr,tn | ftn ,θL) ,

I Noise model

(ycr,t

P (f |Y,T, θK,θL ) ∝N(f∣∣∣0, KT(θK)

∏c∈{A,B}

R∏r=1

N∏n=1

I Noise model

(ycr,t

P (f |Y,T, θK,θL ) ∝N(f∣∣∣0, KT(θK)

∏c∈{A,B}

R∏r=1

N∏n=1

I Noise model

(ycr,t

Robustness With Respect to Outliers

I Outliers in the expression profilecan obscure the regressionresults.

I A mixture noise-model accountsfor outliers.

I Inference in this model is doneusing Expectation Propagation.

6 4 2 0 2 4 60.0

Application 2: differential gene expression Experimental Results on Arabidopsis

Illustration of the Model ComparisonA Differentially Expressed Gene

I Shared model.

I Independent model.

I Model comparison.

Predictive Performance (RECOMB09)

I Data:I 30,336 Arabidopsis thaliana gene probesI Biotic stress: fungus infectionI 24 time points, 4 biological replicates

I Evaluation of alternative methods on 2000 randomly chosenhuman-labeled probes:

I GP no robustI GP robustI F-Test (FT) (MAANOVA package)I Timecourse (TC) (Tai and Speed)

Predictive Performance (RECOMB09)

I Data:I 30,336 Arabidopsis thaliana gene probesI Biotic stress: fungus infectionI 24 time points, 4 biological replicates

I Evaluation of alternative methods on 2000 randomly chosenhuman-labeled probes:

I GP no robustI GP robustI F-Test (FT) (MAANOVA package)I Timecourse (TC) (Tai and Speed)

Predictive Performance (RECOMB09)ROC Curves

0 0.2 0.4 0.6 0.8 10

False positive rate

GP robust (AUC 0.985)GP standard (AUC 0.944)FT (AUC 0.859)TC (AUC 0.869)

Application 2: differential gene expression Detecting Temporal Patterns of Differential Expression

Time-local GPTwoSampleMotivation

I Differential expressionis not static over time.

I The responsedevelops over time.

I Different regulatorsmay be active atdistinct times andtrigger each other.

Time-local GPTwoSampleMotivation

I Differential expressionis not static over time.

I The responsedevelops over time.

I Different regulatorsmay be active atdistinct times andtrigger each other.

Time-local GPTwoSampleModel

I The shared andindependent model areinterleaved over time.

I Indicator variables ztnswitch between bothmodels.

I Inference is done usingGibbs sampling (later).

Time-local GPTwoSampleExample Results

The example from before

Time-local GPTwoSampleExample Results

The example from before Periodic differential expression

Smooth Time-local GPTwoSample

I Transitions betweennon-differential anddifferential expressioncan occur at unobservedtime points and aresmooth.

I Extending the time-localmodel with a GP prior asgating network on theswitch variables.

I Inference using Gibbssampling.

Smooth Time-local GPTwoSample

I Transitions betweennon-differential anddifferential expressioncan occur at unobservedtime points and aresmooth.

I Extending the time-localmodel with a GP prior asgating network on theswitch variables.

I Inference using Gibbssampling.

Smooth Time-local GPTwoSampleGibbs Sampling

I Gibbs sampling exploits tractableconditional distributions.

I Individual indicators zti are resampledin turn

P(zti = s | z\ti ,T,Y,θS,θI,θG

)∼ P

(Y | zti = s, z\ti ,T,θI,θS

(zti = s | z\ti ,θG

I Data likelihood from GP experts

I Predictive distribution from gating network

)∼ P

Smooth Time-local GPTwoSampleExample Results

Early differential expression Transient differential expression

Detecting Transition Points in Arabidopsis Microarray Time SeriesStart/Stop Times

I Studying the temporaldistribution of differentialexpression across genes.

I Considered were the top 6000differentially expressed genes.

I Differential expression appearsto occur in two waves with starttimes at 20h an 25h afterinfection.

I Only very few genes stopdifferential behavior within themeasured time interval.

Distribution of differential start/stop time

Detecting Transition Points in Arabidopsis Microarray Time SeriesStart Times for Gene Categories

I This distribution can bebroke down into genecategories.

I WRKY Family oftranscription factors isknown to be involved instress response.

Distribution of differential start time

Detecting Transition Points in Arabidopsis Microarray Time SeriesStart Times for Gene Categories

I This distribution can bebroke down into genecategories.

I WRKY Family oftranscription factors isknown to be involved instress response.

Distribution of differential start time

Outline

Summary

Motivation

I This part of the course is inspired and based on results of apublication by Neil D. Lawrence et al.Modelling transcriptional regulation using Gaussian processesftp://ftp.dcs.shef.ac.uk/home/neil/gpsim.pdf.

Motivation

I Microarray technologies allow tomeasure mRNA levels.

I The functional proteins and theirconcentration levels remain unobserved.

I Motivation: Infer the hidden proteinconcentrations?

Motivation

A single gene model

I The change in gene expression abundance yi for a gene i isapproximately described by a differential equation model of the form

dyi(t)

dt= Bi + Sif(t)−Diyi(t)

I f(t) regulatory transcription factor.I Bi basal transcription rate.I Si sensitivity of the gene to the transcription factor.I Di decay rate of the mRNA.

I Goal: infer the unobserved activation f(t) from mRNA measurementsof multiple target genes.

A single gene model

I The change in gene expression abundance yi for a gene i isapproximately described by a differential equation model of the form

dyi(t)

I f(t) regulatory transcription factor.I Bi basal transcription rate.I Si sensitivity of the gene to the transcription factor.I Di decay rate of the mRNA.

I Goal: infer the unobserved activation f(t) from mRNA measurementsof multiple target genes.

Derivative observations

I The key to solving this problem are derivative observations.

I Given knowledge about the derivative of a function f we would like toinfer its function values:

dy =∂f(t)

I We wish to find the joint probability of function values and functionderivatives

cov(dyi, yj) =∂

∂ticov(yi, yj)

cov(dyi, dyj) =∂2

∂ti∂tjcov(yi, yj)

I Using these covariance functions we can combine functionobservations and derivatives as training data in GP regression.

Derivative observations

I The key to solving this problem are derivative observations.

I Given knowledge about the derivative of a function f we would like toinfer its function values:

dy =∂f(t)

I We wish to find the joint probability of function values and functionderivatives

cov(dyi, yj) =∂

∂ticov(yi, yj)

cov(dyi, dyj) =∂2

∂ti∂tjcov(yi, yj)

I Using these covariance functions we can combine functionobservations and derivatives as training data in GP regression.

Derivative observationsSquared exponential kernel

I For the squared exponential kernel we obtain:

cov(yi, yj) = k(ti, tj) = A2e−0.5(ti−tj)

cov(dyi, yj) = −cov(yi, yj)(ti − tj)L2

cov(dyi, dyj) = cov(yi, yj)1

[δi,j −

L2(ti − tj)2

Derivative observationsExample

(From E. Solak et al.

Derivative observations in Gaussian Process models of dynamic systems)

Back to the ODE model for gene regulation

dyi(t)

I An explicit solution of the ODE system can be derived (standard ODEtechniques)

yi(t) =BiDi

+ kie−Dit + Sie

−Dit

f(u) exp(Diu)du

yi(t) =BiDi

+ Li[f ](t)

I Realizing that Li is a linear operator (like taking the derivative), wecan again evaluate the covariance between f(t) and Li[f ](t).

ODE model for gene regulationInference results

(From N. D. Lawrence et al.

Modelling transcriptional regulation using Gaussian processes)

Summary

Outline

Summary

I The design and choice of covariance functions allows for flexiblemodeling tasks.

I Prior on heart rateI Derivative observations, ODE systems

I Model comparison using Gaussian processes.I Testing for differential gene expression.

Oliver Stegle and Karsten Borgwardt - ETH Z

Documents

Transcript of Oliver Stegle and Karsten Borgwardt - ETH Z