Clustering Techniques - Lehigh CSElopresti/Courses/2007-08/CSE308... · Clustering Techniques. CSE...

CSE 308-408 · Bioinformatics: Issues and AlgorithmsLopresti · Fall 2007 · Lecture 16 - 1 -

Bioinformatics:Issues and Algorithms

CSE 308-408 • Fall 2007 • Lecture 16

Clustering Techniques


Administrative notes

I'll send you feedback on your proposal by the middle of the following week – then you're off and running!

Your final project / paper proposal is due on Friday, November 9 at 5:00 pm.

The proposal just needs to be a couple paragraphs telling me the problem area you plan to work on and some of the references you'll probably use.

If there's a possible connection between the work you'd like to do and the topics you've heard Professor Marzillier talk about, I'll discuss your proposal with her to get her feedback and suggestions (e.g., other papers you might read, datasets you might use for testing code you develop, etc.).


Outline

• DNA Microarrays• Hierarchical Clustering• K-Means Clustering• Conservative & Greedy K-Means Clustering• Corrupted Cliques Problem• CAST Clustering Algorithm

http://www.bioalgorithms.info


Applications of clustering

Motivation for clustering (from a general perspective):• Viewing and analyzing vast amounts of biological data in its

unstructured entirety can be perplexing.

• It is easier to interpret data if it is organized into clusters that combine similar (i.e., related) data points.

From a biological perspective, applications include:

• Analyzing data from DNA microarray experiments (expression analysis – i.e., determining which genes are switched “on” or “off” under certain conditions of interest).

• Building and understanding phylogenetic (evolutionary) trees based on genomic or other data.



Inferring gene functionality

What's the problem?

• Biologists want to know functions of newly-sequenced genes.

• Simply comparing new gene sequence to known DNA sequences often does not reveal function of new gene.

• For 40% of sequenced genes, functionality cannot be ascertained by comparing to sequences of known genes.

• Microarrays allow biologists to infer gene function even when sequence similarity alone is insufficient to infer it.



Life: a recipe for making proteins

http://www.cbs.dtu.dk/courses/thaiworkshop/programme.php


Recall the Central Dogma



Hybridization is central



Microarrays: the concept


Measure level of transcription for a very large number of genes in a single experiment.


Microarrays and expression analysis


Microarrays measure activity (expression level) of genes under varying conditions and/or points in time.

• A gene is active if it is being transcribed.

• More mRNA usually indicates more gene activity.

Expression level is estimated by measuring amount of mRNA for that particular gene:


Microarrays: how?



Stanford microarrays: production





Coating:

1. Rinse of slides: NaOH and EtOH (2 h - shaking).

2. Wash with water.

3. Coat slides: poly-L-lycine (1 h - shaking).

4. Wash and dry.

Attach probes:

1. Produce probes (oligos, cDNA library, PCR products).

2. Print by the use of a robot.




Spotting – mechanical deposition of probes:



Microarrayer



Microarray experiments

Steps:

• Produce cDNA from mRNA (DNA is more stable).

• Attach phosphor to cDNA to see when gene is expressed.

• Different color phosphors are available to compare many samples at once.

• Hybridize cDNA over microarray.

• Scan microarray with phosphor-illuminating laser: illumination reveals transcribed genes.

• Scan microarray multiple times for different color phosphors.



Microarray experiments

http://www.affymetrix.com

Phosphors canbe added hereinsteadThen instead of

staining, laser illumination can be used


Using microarrays

• Track sample over period of time to see how gene expression changes.

• Track two different samples under same conditions to see differences in gene expression.

Each box represents one gene’s expression over time



Using microarrays

Interpreting colors:

• Green: expressed only from control.

• Red: expressed only from experimental cell.

• Yellow: equally expressed in both samples.

• Black: NOT expressed in either control or experimental cells.



Microarray data

What does biologist do with microarray data?

• Microarray data usually transformed into an intensity matrix.

• Intensity matrix allows biologists to make correlations between different genes (even if they are dissimilar) and to understand how gene functions might be related.

321Gene 5387Gene 438.64Gene 39010Gene 2

10810Gene 1Time ZTime YTime XTime:

Intensity (expression level) of gene at measured time


Similar behavior?Clustering comes into play!


Clustering microarray data

• Plot each sample as data point in N-dimensional space.

• Build matrix for distances between every two gene points.

• Genes with a small distance share same expression patterns and might be functionally related or similar.

• Clustering reveal groups of functionally related genes.From “Cluster analysis and display of genome-wide expression patterns” by Eisen, Spellman, Brown, and Botstein, Proc. Natl. Acad. Sci. USA, Vol. 95, pp. 14863–14868, December 1998.

Different genes that express similarly


Clustering microarray data

Three different clusters

Intensity matrix Pairwise distance matrix

Expression patterns as points in 3-D space



Homogeneity and Separation Principles

All approaches to clustering guided by two basic principles:

• Homogeneity: elements within a given cluster are close.

• Separation: elements in different clusters are further apart.

Not that clustering is not an easy task! (Don't be mislead by simple illustrative examples.)

Given these points, a clustering algorithm might make two distinct clusters as follows ...



Bad clustering

Close distances from points in separate clusters

This clustering violates both Homogeneity and Separation Principles:

Far distances from points in same cluster



Good clustering

This clustering satisfies both Homogeneity and Separation Principles:



Clustering techniques

• Hierarchical: organize elements into a tree, leaves represent genes and length of the paths between leaves represents distances between genes. Similar genes lie within same subtrees.


• Agglomerative: start with every element in its own cluster, and iteratively join clusters together.

• Divisive: start with one cluster and iteratively divide it into smaller clusters.


Hierarchical clustering



Hierarchical clustering

Hierarchical Clustering often used to reveal evolutionary history:



Hierarchical clustering algorithm

Hierarchical Clustering (d, n)Form n clusters each with one elementConstruct graph T by assigning one vertex to each clusterwhile there more than one cluster

find two closest clusters C1 and C2

merge C1 and C2 into new cluster C of size |C1| +|C2|

compute distance from C to all other clustersadd a new vertex C to T and connect to vertices C1 and C2

remove rows and columns of d corresponding to C1 and C2

add a row and column to d corrsponding to new cluster Creturn T

Algorithm takes a n x n distance matrix d of pairwise distances between points as input.


Hierarchical clustering algorithm

Hierarchical Clustering (d, n)Form n clusters each with one elementConstruct graph T by assigning one vertex to each clusterwhile there more than one cluster

find two closest clusters C1 and C2

merge C1 and C2 into new cluster C of size |C1| +|C2|

compute distance from C to all other clustersadd a new vertex C to T and connect to vertices C1 and C2

remove rows and columns of d corresponding to C1 and C2

add a row and column to d corrsponding to new cluster Creturn T

Different ways to define distances between clusters may lead to different clusterings!


Computing distances

dmin(C, C*) = min d(x,y) for all elements x in C and y in C*

Distance between two clusters is smallest distance between any pair of elements.

davg(C, C*) = (1 / |C*||C|) ∑ d(x,y) for all elements x in C and y in C*

Distance between two clusters is average distance between all pairs of elements.


Squared-error distortion

Given a data point v and a set of points X, define distance from v to X:

d(v, X)

as (Eucledian) distance from v to closest point from X.

Given set of n data points V = {v1…vn} and set of k points X, define squared-error distortion as:

d(V, X) = ∑d(vi, X)2 / n 1 < i < n


Clustering microarray data: k-means clustering

K-means clustering is one way to organize this data:

• Given set of n data points and an integer k.

• We want to find set of k points that minimizes mean-squared distance from each data point to its nearest cluster center.

Sketch of algorithm:

• Choose k initial center points randomly and cluster data.

• Calculate new centers for each cluster using points in cluster.

• Re-cluster all data using new center points.

• Repeat last two steps until no data points are moved from one cluster to another or some other convergence criterion is met.


Formal definition of K-Means Clustering

The K-Means Clustering Problem.

Output: A set X consisting of k points (cluster centers) that minimizes squared-error distortion d(V, X) over all possible choices of X.

Input: A set, V, consisting of n points along with a parameter k.

A (trivially) simple variation, 1-means clustering:

The 1-Means Clustering Problem.

Output: A single point x (cluster center) that minimizes squared-error distortion d(V, x) over all possible choices of x.

Input: A set, V, consisting of n points.

1-means clustering is easy.General k-means clusteringis NP-complete, however.



From “Data Analysis Tools for DNA Microarrays” by Sorin Draghici.

• Pick k = 2 centers at random.• Cluster data around these

center points.

• Re-calculate centers based on current clusters.



From “Data Analysis Tools for DNA Microarrays” by Sorin Draghici.

• Re-cluster data around new center points.

• Repeat last two steps until no more data points are moved into a different cluster.


K-means clustering: Lloyd's algorithm

K-Means Clustering (k)arbitrarily assign k cluster centerswhile cluster centers keep changing

assign each data point to cluster Ci corresponding to closestcluster representative (center) (1 ≤ i ≤ k)

after assignment of all data points, compute new clusterrepresentatives according to cluster centers of gravityI.e., new cluster representative is ∑v / |C| for all v in C

output final cluster centers

Note that this may only leadto a locally optimal clustering.


K-means clustering: another example

0

1

2

3

4

5

0 1 2 3 4 5

expression in condition 1

expr

essi

on in

con

ditio

n 2

x1

x2

x3



0

1

2

3

4

5

0 1 2 3 4 5


expr

essi

on in

con

ditio

n 2

x1

x2

x3



0

1

2

3

4

5

0 1 2 3 4 5


expr

essi

on in

con

ditio

n 2

x1

x2x3



0

1

2

3

4

5

0 1 2 3 4 5


expr

essi

on in

con

ditio

n 2

x1

x2 x3


Conservative k-means clustering

Observations:

• This algorithm, known as Lloyd's algorithm, is fast, but in each iteration it moves many data points, not necessarily causing better convergence.

• A more conservative method would be to move one point at a time only if it improves the overall clustering cost.

• The smaller the clustering cost of a partition of data points, the better that clustering is.

• Different methods (e.g., squared-error distortion) can be used to measure this clustering cost.


Greedy k-means clustering

ProgressiveGreedyK-Means(k)select an arbitrary partition P into k clusterswhile forever

bestChange ← 0for every cluster C

for every element i not in C if moving i to cluster C reduces its clustering cost

if (cost(P) – cost(Pi C) > bestChange

bestChange ← cost(P) – cost(Pi C)

i* ← i, C* ← Cif bestChange > 0

Change partition P by moving i* to C*else

return P


Clique graphs

A more structured view of clustering:

• A clique is a graph with every vertex connected to every other vertex.

• A clique graph is a graph where each connected component is a clique.


Clique of size 3

Clique of size 5

Clique of size 6

Clique graph with 3connected components


Transformation into a clique graph

Any graph can be transformed into a clique graph by adding or removing edges.What can we do here?

1

7

2

6

3

5 4

1

7

2

6

3

5 4

Delete 2 edges


Transformation into a clique graph

As with edit distance we studied earlier, there many possible transformations:

1

3

2

4

5

1

3

2

4

5

1

3

2

4

5

1

3

2

4

5

Add 2 edges

Delete 4 edgesOr:


Formal definition of Corrupted Cliques Problem

The Corrupted Cliques Problem.

Output: The smallest number of additions and removals of edges that will transform G into a clique graph.

Input: A graph, G.

Our ultimate goal is to have:

• Vertices represent data points.

• Edges represent relationship between data points.

• Cliques represent meaningful groupings (i.e., clusters).


Distance graphs

Transform a distance matrix into a distance graph:

• Genes are represented as vertices in graph.

• Choose a distance threshold θ.

• If distance between two vertices is below θ, draw an edge between them.

• Resulting graph may contain cliques.

• These cliques represent clusters of similar data points!


Transforming distance graph into clique graph

Distance graph for θ = 7 Clique graph

Distance matrix d

Distance graph for is not quite a clique graph. However, it can be transformed into one by removing edges (g

1,g

10) and (g

1,g

9).


Heuristics for Corrupted Cliques Problem

Corrupted Cliques Problem is NP-Hard, some heuristics exist to approximately solve it. For example,

• CAST (Cluster Affinity Search Technique) is a practical and fast algorithm for CCP.

• CAST is based on notion of genes close to given cluster C, or distant from cluster C.

• Distance between gene i and cluster C defined as:d(i,C) = average distance between i and each gene in C

• Gene i is close to cluster C if d(i,C) < θ, distant otherwise.


CAST algorithm

CAST(S, G, θ) P ← Ø while S ≠ Ø V ← vertex of maximal degree in distance graph G C ← {v} while a close gene i not in C or distant gene i in C exists Find nearest close gene i not in C and add it to C Remove farthest distant gene i in C Add cluster C to partition P S ← S – C Remove vertices of cluster C from distance graph G return P

S = set of elements, G = distance graph, θ = distance threshold


Wrap-up

Remember:• Come to class having done the readings.• Check Blackboard regularly for updates.

Readings for next time:• BBP Chapters 17-18 and 20 (tools, datasets, and

applications).

Clustering Techniques - Lehigh CSElopresti/Courses/2007-08/CSE308... · Clustering Techniques. CSE...

Documents

Transcript of Clustering Techniques - Lehigh CSElopresti/Courses/2007-08/CSE308... · Clustering Techniques. CSE...