Approximate Substring Matching over Uncertain Strings

Tingjian Ge Zheng LiUniversity of Kentucky

Motivation• As a consequence of the burgeoning growth of data and the

cost/technology constraints against producing completely clean data, there is much uncertainty in the data itself.

Computational biology

Signal processing

Text retrieval

Motivation• The amount of text data increases in an unprecedented rate. Managing

the sheer amount of (often noisy) text data has become more challenging than ever.

• Approximate substring matching has many applications. – The deterministic case is well studied.– But approximate pattern matching over uncertain texts is largely

an unexplored problem in the past.

Example : Pattern Matching in DNA Sequence

• Suppose a certain DNA pattern, say AAATTT and it’s variations ( with small edit distance ) are known to be the cause of a low or nonfunctioning protein. Thus we want to do approximate pattern matching in DNA sequences.

• Challenges are :– A single DNA sequence can be a few million to a few hundred million

characters. – It has uncertainty due to a number of factors in the high-throughput

sequencing technologies.

Example : Holter Monitor Application

• For each heartbeat, the annotation software gives a symbol such as N (Normal beat), L (Left bundle branch block beat), etc. Quite often, the ECG signal of each beat may have ambiguity.

• A doctor might be interested in locating a pattern such as “NNAV”, in order to verify a specific diagnosis.

Outline

• Motivation• Preliminaries• A new semantics• Multilevel filtering index• Two verification algorithms• Experiments

Q-gram Index• The most frequently used indexing method for approximate matching is

the q-gram indexes.• To use the q-gram index, partition pattern into k+1 pieces, where k is the

edit distance threshold. Since the number of errors is no more than k, it must be true that at least one piece must have an exact match in the text.

Q-gram1 pos1 pos2 …Q-gram2 …

… …

Position list of q-gram1

Pattern String …

k+1 pieces

Verification Algorithm

• A DP algorithm that computes edit distance

0 1 2 3 41 1 1 2 32 2 2 2 33 3 3 3 2

t h i s

h a s

Ins.

Sub.

[ , ] min{ [ , 1] 1,[ 1, ] 1,[ 1, 1] ( [ ], [ ])}

d i j d i jd i jd i j c p i x j

0, [ ] [ ]( [ ], [ ])

1, [ ] [ ]if p i x j

c p i x jif p i x j

The edit distance between “has” and “This” is 2 in this example. The value in each cell is the edit distance between the corresponding pattern and text characters read so far.

Outline

• Motivation• Preliminaries• A new semantics• Multilevel filtering index• Two verification algorithms• Experiments

(k, τ)-Matching Query • We propose (k, τ)-matching query, which is based on a pattern string p, a

set of uncertain text strings {Xi} (1 ≤ i ≤ r), and threshold parameters k, τ, and asks for all substrings X of Xi’s such that Pr[d(p, X) ≤ k] > τ.

• An other semantics is the EED ( expected edit distance ), which computes the expected edit distance between p and X and check if it’s < k’.

Semantics• (k, τ)-matching query v.s. EED

– EED first summarizes possible worlds and then apply the threshold, hence many algorithms developed for the deterministic case are inapplicable.

– More importantly, EED may either miss real matches or have an unduly big threshold so that many false positives may mix in.

0 2 4 6 8x 10

6

100

101

102

103

104

Text string size

Num

ber o

f mat

ches

DNA: (k, )DNA: EED

Example : Semantics

T T T T T T T T T

A A A A A A G G G

G G A G A G T G T G A G T A T A G A A G

G G A G A G T G T G A G T A T A G A A G

Pattern p, Length = 20

Consider this approximate pattern matching in DNA sequence :

Substring X1 has an exact match, but 9 characters are uncertain. EED(p, X1) = 6

T T

A G

A G G G G G T G T A A G T A T A G A A T

Substring X2 has no exact match, with 5 errors and 2 uncertain characters. EED(p,X2)=6

To find X1, with EED, the threshold should be at least 6, which means we can’t avoid X2.But with (k, τ)-matching query, we can use a ( 2, 0 )-matching query to select X1 and avoid X2 at the same time.

Outline

• Motivation• Preliminaries• Semantics• Multilevel filtering Index• Two verification algorithms• Experiments

Left/Right Signature of a Q-gram

• The edit distance of signatures is no more than the edit distance of original strings if we treat each two-bit value in the signature as a character when computing the edit distance.

G 01HashRATGS

pos pos+q

q-gramleft signature right signature

Text string x

Left signature and right signature of a q-gram.The hash function maps a character to a two-bit value.

Multilevel Filtering Index

4-byte tag

4-byte tag

4-byte tag

. . . . .

. . . . .

28 72 short position list

next level directory

99 short position list

A multilevel filtering Index for a q-gram based on measuring signature distance.

4-byte tag in the figure contains left/right signature of the q-gram.

Best Matching Prefix

px

G

A

P

G G A P P

0

01

1

2

2

3

3 4 5

1

1

2

1

2

2

1

2

3

2

1

4

3

2

“1” is the smallest in the last row. Hence, 1 is x’s prefix distance from p and GGAP is the best matching prefix of x for p.

Dynamic Programing Verification Using Signatures

• The value on each diagonal in the DP table form a non-decreasing sequence, as shown in the figure.

• Let be prefix distance from , The verification requires :

We need to do this mapping

to accommodate the certain length of signatures in the index tag.

x’[l]

x’[m’+k]p’[m’]

x’[m’]

x’[m’−k]

p’[l]p’[l−k]

p’[l+k]

x’[1]p’[1]rl dd /

kdd rl

spp rl '/

rl xx /

Dynamic Programming Verification Using Signatures

• When pattern is exhausted. We do mapping as the following.

p’[m’] x’[m’−

k]x’[m’+k]

x’[l]

Example : Using the Index

• Input: X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]K = 2

PatternText

lDP

PatternText

rDP

Q-gram “BEST” matches. We then use this q-gram’s multi-level signature index for filtering.

Example : Using the Index

• Input: X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]K = 2

lDP

PatternText

rDP

B’ C’

B’

A’

C’

D’

0 1 2

1 0 1

2 1 1

3 2 1

4 3 2

PatternR’ O’

N’

R’

O’

B’

0 1 2

1 1 2

2 1 2

3 2 1

4 3 2

Text

kdd rl 211Need to expand to next level of the index.

Use the first level signatures in the index tag.

Using the Index-Example

• Input: X: … … [DC][CB][BEST][RO][DO]… …

P: [DC][AB][BEST][NR][OB]K = 2

lDP

PatternText

rDP

B C

B

A

C

D

0 1 2

1 0 1

2 1 1

3 2 1

4 3 2

PatternR O

N

R

O

B

0 1 2

1 1 2

2 1 2

3 2 1

4 3 2

Text C D

3 4

2 3

2 3

1 2

2 1

D O

3 4

3 4

3 4

2 3

2 3

kdd rl 21This candidate position is filtered out.

Use the second level signatures in the index tag.

Pattern is exhausted in this case.

Outline

• Motivation• Preliminaries• Semantics• Multilevel filtering index• Two verification algorithms

– Bounds based on CDF– Bounds based on local perturbation

• Experiments

Verification Algorithms• The goal of verification is to conclude whether a candidate

position selected by an index is a true match.

• We present two algorithms, each of which gives an upper and a lower bound of the probability that .kxpd ),(

Outline

• Motivation• Preliminaries• Semantics• Multilevel filtering index• Two verification algorithms

– Bounds based on CDF– Bounds based on local perturbation

• Experiments

Bounds Based on Cumulative Distribution Functions (CDF)

• The basic verification consists of two symmetric runs of a DP algorithm. We describe how we change such a DP algorithm to accommodate uncertain characters.

• Our key idea is to compute (at most) k+1 pairs of values in each cell, i.e., where D denotes the edit distance.

}0|])[],[{( kjjFjF ul ],[]Pr[][ jFjDjF ul

A Basic Step

1D 2D

3D D

Consider a basic step: how do we get D from its 3 neighbors?

•

argmini Di returns the index value i that minimizes Di; the minimization is defined as the Di (1 ≤ i ≤ 3) that has the greatest ( i.e. the one that has a small distance value with the highest prob. ).

)1],1[min(][][

]1[][][

1]Pr[

3

1

)(2

)1(1

)min(arg2

)1(1

12

1

i

iuuu

Dlll

jFpjFpjF

jFpjFpjF

ppcCp

ii

]0[)(ilF

//probability of a match at cell DPick one fixed neighbor cell to use.

Use the union of upper bounds from the three neighbors

Example : Bounds Based on CDF• Consider p = “CAT” and X is “C” followed by four characters, each of which

has the same distribution G.1A.4T.5, denoting that it is G (A, T) with probability 0.1 (0.4, 0.5). K = 2.

• Take the cell at the 3rd row and the 4th column as an example. How do we compute Fl[j] & Fu[j]?

p x

C

A

T

C G.1A.4T.5 G.1A.4T.5 G.1A.4T.5 G.1A.4T.5

(0, 0)(.7, .7)(1, 1)

(0, 0)(.64, .64)

(1, 1)

(.2, .2)(.7, .7)(1, 1)

(0, 0)(0, 0)

(.784, .784)

(0, 0)(.42, .42)(.85, 1)

(0, 0)(0, 0)

(.602, .602)

(.4, .4)(1, 1)

(1, 1)

(0, 0)(1, 1)

(0, 0)(0, 0)(1, 1)

(0, 0)(1, 1)

(0, 0)(0, 0)(1, 1)

argmini Di is D3 Mixture of the upper bound in D1 and the union of those in all three neighbours.

Outline

• Motivation• Preliminaries• Semantics• Multilevel filtering technique based on

measuring signature distance• Two verification algorithms

– Bounds based on CDF– Bounds based on local perturbation

• Experiments

Bounds Based on Local Perturbation

• Adjacent and remote possible worlds: Give a (k, τ)-matching query on a pattern p and on an uncertain text X, we say that a p.w. w of X, denoted x(w) is adjacent to p if . We say that it is remote to p if .

kwxpd ))(,(kwxpd ))(,(

G/A/T A/T … A/T

An initial adjacent\remote p.w.

Perturbation G/A/T A/T … A/T

G/A/T A/T … A/T

G/A/T A/T … A/T

G/A/T A/T … A/T

… … … …

More adjacent\remote p.w. after perturbation.

How to Get Initial Adjacent/Remote Possible World.

Get a closest p.w. ( an adjacent p.w. with the smallest distance ) on the optimal path in DP table.

C G.1\A.4\T.5 G.1\A.4\T.5 G.1\A.4\T.5

Use a randomized algorithm to get a remote p.w. :

K=1

1D 2D

3D D

If D1 and D contain the same distance value, the corresponding variable in the test string is called a crucial variable of this p.w.

0 1 2 3 4

1 0 1 2 3

2 1 0 1 2

3 2 1 0 1

px C G.1A.4T.5 G.1A.4T.5 G.1A.4T.5

C

A

T

Perturbation – How?

T/.5 T/.5 T/.5 T/.5 …

A/.4 A/.4 A/.4 A/.4 … G G/.1 G/.1 G/.1 G/.1 …

Suppose there’re c crucial variables in this closest/remote p.w.

Let be the difference between k and the edit distance of this p.w and the pattern.

For an closest p.w. :Pr(at most crucial variables ( of the total c curial variables ) change their values as in the optimal path ) is a lower bound of .

]),(Pr[ kXpd

For a remote p.w. :We could get a upper bound similarly.

Text string x

Outline

• Motivation• Preliminaries• Semantics• Multilevel filtering index• Two verification algorithms• Experiments

Setup of Experiment• We examine the behaviors of (k, τ)-pattern matching query using signature

filtering and verification bounds with the following dataset.• The DNA dataset.

– Raw datasets of sequencing runs of Escherichia coli 536 from the NCBI SRA (Sequence Read Archive) database.

– Use Bowtie to align the short DNA sequences with the complete Escherichia coli genome reference. The mapping reports output by Bowtie show positions within the DNA that have more than one possible value.

• The protein dataset. • Synthetic datasets.

– We generate a few synthetic datasets based on the two real datasets above.

– Vary the parameter values of data (such as the uncertainty ratio θ) or the size of the data.

Experiments

0 2 4 6 8

x 106

10-1

100

101

102

103

Text string size

Exe

cutio

n tim

e (s

econ

ds)

sig.; boundsno sig.; boundsno sig.; bounds 2sig.; no boundsno sig.; no bounds

0.1 0.2 0.3 0.4 0.510

0

101

102

103

104

Exe

cutio

n tim

e (s

econ

ds)

0 2 4 610

1

102

103

104

Text string size (in GB)

Exe

cutio

n tim

e (s

econ

ds)

with signatureswithout signatures

2GB(i) 2GB(ii) 4GB(i) 4GB(ii)0

200

400

600

800

1000

(i) with signatures (ii) without signatures

Exe

cutio

n tim

e (s

econ

ds)

I/O timeCPU time

Running time for various settings.

Varying θ. Using larger synthetic data.

Breakdown of I/O & CPU costs.

Experiments

# of positions to be verified.

Varying |p| (DNA). Varying |p| (protein).

Varying threshold k.

0 2 4 6 8

x 106

102

103

104

105

Text string size

Num

ber o

f pos

ition

s to

be

verif

ied

no sig.sig.no sig., after boundssig., after bounds

10 20 30 40 5010

0

101

102

103

104

|p|

Exe

cutio

n tim

e (s

econ

ds)

10 20 30 40 5010

0

101

102

103

|p|

Exe

cutio

n tim

e (s

econ

ds)

signatures; boundsno signatures; boundssignatures; no boundsno signatures; no bounds

0 1 2 3 410

0

101

102

103

k

Exe

cutio

n tim

e (s

econ

ds)

Conclusions and Future Work

• We study a real and unsolved problem of approximate substring matching over uncertain texts.– Proposed a novel semantics and demonstrate its advantages over an

alternative one introduced by previous work.– Developed a q-gram based index to handle uncertain texts.– Proposed two efficient verification algorithms.

• As future work, we plan to study the matching problem under correlated uncertainty to address a wider range of applications.

Thank You!

• Questions?