Tokenization
21
description
Tokenization. types. He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would." "And then?" asked Sam, as the old miser paused to take a bite of bread and meat. expressions. Co-occurrence Vectors. Tokenization. - PowerPoint PPT Presentation
Transcript of Tokenization
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 224 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 13 ALPHAawful 2 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 225 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 13 ALPHAawful 1 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 225 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 13 ALPHAawful 1 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1acted 12 121 2
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 225 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 14 ALPHAawful 1 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1acted 12 121 2
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 225 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 14 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1acted 12 121 2awful 12 121 3
…
Co-occurrence Vectors
TypeCoun
t Classand 909 ALPHAof 1,221 ALPHA
" 2,000 PUNCThe 225 ALPHA
, 898 PUNCTa 709 ALPHAi 565 ALPHA
acted 14 ALPHAawful 2 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Tokenization
He acted awful mysterious like, and finally he asks me if I'd like to own half of a big nugget of gold. I told him I certainly would."
"And then?" asked Sam, as the old miser paused to take a bite of bread and meat.
TokenDoc ID
Expr ID Index
he 12 121 1acted 12 121 2awful 12 121 3mysterious 12 121 4like 12 121 5, 12 121 6and 12 121 7
…
Co-occurrence Vectors
TypeCoun
t Classand 912 ALPHAof 1,224 ALPHA" 2,003 PUNCT
he 227 ALPHA, 890 PUNCTa 711 ALPHAi 567 ALPHA
acted 14 ALPHAawful 1 ALPHA
…
expressions
types
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysisexpressions
TokenDoc ID
Expr ID Index
he 12 121 1acted 12 121 2awful 12 121 3mysterious 12 121 4like 12 121 5, 12 121 6and 12 121 7
…121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to
take a bite of bread and meat .
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
121. awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
Surrounds Count Center Surrounds Doc Expr Indexcenterssurrounds
<start> actedhe
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
Surrounds Count<start>_acted 1
Center Surrounds Doc Expr Indexhe <start>_acted 3 121 1
centerssurrounds
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
121. <start> mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
Surrounds Count<start>_acted 1
Center Surrounds Doc Expr Indexhe <start>_acted 3 121 1
centerssurrounds
he awfulacted
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
Surrounds Count<start>_acted 1he_awful 1
…
Center Surrounds Doc Expr Indexhe <start>_acted 3 121 1
acted he_awful 3 121 2….
centerssurrounds
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
Surrounds Count<start>_acted 1he_awful 1
…
Center Surrounds Doc Expr Indexhe <start>_acted 3 121 1
acted he_awful 3 121 2….
centerssurrounds
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take
bread and meat .a ofbite
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Polygram Analysis
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
Surrounds Count<start>_acted 1he_awful 1
…a_of 1
Center Surrounds Doc Expr Indexhe <start>_acted 3 121 1
acted he_awful 3 121 2….bite a_of 3 124 12
centerssurrounds
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Collocation Discovery
Rank Surrounds Count1 the_of 77,611 2 <start>_the 42,114 3 the_and 37,105 4 <start>_was 22,769
…21 to_and 8,633 22 to_a 8,471 23 the_in 8,433 24 <start>_said 8,072 25 a_. 7,993
…57 of_. 4,966
…71 a_of 4,493
υ [%]
surroundsfrequency
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Collocation Discovery
Rank Surrounds Count1 the_of 77,611 2 <start>_the 42,114 3 the_and 37,105 4 <start>_was 22,769
…21 to_and 8,633 22 to_a 8,471 23 the_in 8,433 24 <start>_said 8,072 25 a_. 7,993
…57 of_. 4,966
…71 a_of 4,493
surrounds
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Collocation Discovery
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold .
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
a_.
.
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
Collocation Discovery
121. <start> he acted awful mysterious like , and finally he asks me if i'd like to own half of a big nugget of gold
122. <start> i told him i certainly would . 123.“ <start> and then “ ? 124.<start> asked sam , as the old miser paused to take a bite of
bread and meat .
a_.
.
Center Surrounds
…
he <start>_actedacted he_awful… …bite a_of… …
centers
big nugget of gold
Tokenization Polygram Analysis
Collocation Discovery
Co-Occurrence Matrix ConstructionSplittingTerm Clustering
AnnotationSynthesis
Text
n>1
term clusters
Occurrence Vectors
expressions types centerssurrounds
types, surrounds, centers
Collocations
