Special Topics in Computer ScienceSpecial Topics in Computer Science

The Art of Information RetrievalThe Art of Information Retrieval

Chapter 7: Text Operations Chapter 7: Text Operations

Alexander Gelbukh

www.Gelbukh.com

2

Previous chapter: ConclusionsPrevious chapter: Conclusions

Modeling of text helps predict behavior of systemso Zipf law, Heaps’ law

Describing formally the structure of documents allows to treat a part of their meaning automatically, e.g., search

Languages to describe document syntaxo SGML, too expensive

o HTML, too simple

o XML, good combination

3

Text operationsText operations

Linguistic operations Document clustering Compression Encription (not discussed here)

4

Linguistic operationsLinguistic operations

Purpose: Convert words to “meanings” Synonyms or related words

o Different words, same meaning. Morphology

o Foot / feet, woman / female

Homonymso Same words, different meanings. Word senses

o River bank / financial bank

Stopwordso Word, no meaning. Functional words

o The

5

For good or for bad?For good or for bad?

More exact matchingo Less noise, better recall

Unexpected behavioro Difficult for users to graspo Harms if introduces errors

More expensiveo Adds a whole new technologyo Maintenance; language dependentso Slows down

Good if done well, harmful if done badly

6

Document preprocessingDocument preprocessing

Lexical analysis (punctuation, case)o Simple but must be careful

Stopwords. Reduces index size and pocessing time Stemming: connected, connection, connections, ...

o Multiword expressions: hot dog, B-52

o Here, all the power of linguistic analysis can be used

Selection of index termso Often nouns; noun groups: computer science

Construction of thesauruso synonymy: network of related concepts (words or phrases)

7

StemmingStemming

Methodso Linguistic analysis: complex, expensive maintenance

o Table lookup: simple, but needs data

o Statistical (Avetisyan): no data, but imprecise

o Suffix removal

Suffix removalo Porter algorithm. Martin Porter. Ready code on his website

o Substitution rules: sses s, s o stresses stress.

8

Better stemmingBetter stemming

The whole problematics of computational linguistics POS disambiguation

o well adverb or noun? Oil well.

o Statistical methods. Brill tagger

o Syntactic analysis. Syntactic disambiguation

Word sense disambiguatiuono bank1 and bank2 should be different stems

o Statistical methods

o Dictionary-based methods. Lesk algorithm

o Semantic analysis

9

ThesaurusThesaurus

Terms (controlled vocabulary) and relationships Terms

o used for indexingo represent a concept. One word or a phrase. Usually nounso sense. Definition or notes to distinguish senses: key (door).

Relationshipso Paradigmatic:

Synonymy, hierarchical (is-a, part), non-hierarchical

o Syntagmatic: collocations, co-occurrences WordNet. EuroWordNet

o synsets

10

Use of thesurusUse of thesurus

To help the user to formulate the queryo Navigation in the hierarchy of words

o Yahoo!

For the program, to collate related termso woman female

o fuzzy comparison: woman 0.8 * female. Path length

11

Yahoo! vs. thesaurusYahoo! vs. thesaurus

The book says Yahoo! is based on a thesaurus.

I disagree Tesaurus: words of language organized in hierarchy Document hierarchy: documents attached to hierarchy This is word sense disambiguation I claim that Yahoo! is based on (manual) WSD Also uses thesaurus for navigation

12

Text operationsText operations

Linguistic operations Document clustering Compression Encription (not discussed here)

13

Document clusteringDocument clustering

Operation on the whole collection Global vs. local Global: whole collection

o At compile time, one-time operation

Localo Cluster the results of a specific query

o At runtime, with each query

Is more a query transformation operationo Already discussed in Chapter 5

14

Text operationsText operations

Linguistic operations Document clustering Compression Encription (not discussed here)

15

CompressionCompression

Gain: storage, transmission, search Lost: time on compressing/decompressing

In IR: need for random access. o Blocks do not work

Also: pattern matching on compressed text

16

Compression methodsCompression methods

Statistical Huffman: fixed size per symbol.

o More frequent symbols shorter

o Allows starting decompression from any symbol

Arithmetic: dynamic codingo Need to decompress from the beginning

o Not for IR

Dictionary Pointers to previous occurrences. Lampel-Ziv

o Again not for IR

17

Compression ratioCompression ratio

Size compressed / size decompressed

Huffman, units = words: up to 2 bits per charo Close to the limit = entropy. Only for large texts!

o Other methods: similar ratio, but no random access

Shannon: optimal length for symbol with probability p is - log2 p

Entropy: Limit of compressiono Average length with optimal coding

o Property of model

18

ModelingModeling

Find probability for the next symbol Adaptive, static, semi-static

o Adaptive: good compression, but need to start frombeginning

o Static (for language): poor compression, random access

o Semi-static (for specific text; two-pass): both OK

Word-based vs. character-basedo Word-based: better compression and search

19

Huffman codingHuffman coding

Each symbol is encoded, sequentially More frequent symbols have shorter codes No code is a prefix of another one

How to buildthe tree: book

Byte codesare better

Allow forsequentialsearch

20

Dictionary-based methodsDictionary-based methods

Static (simple, poor compression), dynamic, semi-static. Lempel-Ziv: references to previous occurrence

o Adaptive

Disadvantages for IRo Need to decode from the very beginning

o New statistical methods perform better

21

Comparison of methodsComparison of methods

22

Compression of inverted filesCompression of inverted files

Inverted file: words + lists of docs where they occur Lists of docs are ordered. Can be compressed Seen as lists of gaps.

o Short gaps occur more frequently

o Statistical compression

Our work: order the docs for better compressiono We code runs of docs

o Minimize the number of runs

o Distance: # of different words

o TSP.

23

Research topicsResearch topics

All computational linguisticso Improved POS tagging

o Improved WSD

Uses of thesauruso for user navigation

o for collating similar terms

Better compression methodso Searchable compression

o Random access

24

ConclusionsConclusions

Text transformation: meaning instead of stringso Lexical analysis

o Stopwords

o Stemming POS, WSD, syntax, semantics Ontologies to collate similar stems

Text compressiono Searchable

o Random access

o Word-based statistical methods (Huffman)

Index compression

25

Thank you!Till compensation

lecture