The Haystack SKA/LOFAR Performance Simulator Feb 13, 2004 Ramesh Bhat MIT/Haystack.
SE2016 Java Alexey Tokar "To find a needle in a haystack"
Transcript of SE2016 Java Alexey Tokar "To find a needle in a haystack"
![Page 1: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/1.jpg)
Найти иг[ло]ку в стоге сена
![Page 2: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/2.jpg)
Information retrieval
●○
●○
●○
![Page 3: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/3.jpg)
Computer vision
![Page 4: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/4.jpg)
Parametric search
http
://m
arke
t.yan
dex.
ua
![Page 5: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/5.jpg)
Text search
![Page 6: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/6.jpg)
Basic data structures
![Page 7: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/7.jpg)
Simple matching
String needle = getUserInput();
for ( String title : dictionary ) {
if ( title.equals( needle ) ) {
System.out.println( “Gotcha!” );
}
}
![Page 8: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/8.jpg)
Simple matching: better
String needle = getUserInput().toLowerCase();
for ( String title : dictionary ) {
if ( title.startsWith( needle ) ) {
System.out.println( “Gotcha!” );
}
}
![Page 9: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/9.jpg)
Simple matching: awesomest !
String needle = “транс”;начало сравнений
...терминатор транзит трансамерика трансильвания трансформеры...
![Page 10: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/10.jpg)
Simple matching. Results.
![Page 11: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/11.jpg)
Hash-based search
.hashCode() % buckets.length buckets
Терминатор 1207003689 00
Альф 32058595 01
Поймай меня если сможешь 824411401 02
Умри, но не сейчас 1792373633 03
Зловещие мертвецы 1534549542 04
Терминал 1534215135 05
09...
![Page 12: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/12.jpg)
Hash-based search
05
ТерминалАльф ...
![Page 13: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/13.jpg)
Hash-based search
![Page 14: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/14.jpg)
Radix trie
ме
с довый месяц
т яц в деревне
о ь в
сына встречи уду ампира илли
...Медовый месяцМесяц в деревнеМесть вампира
Месть вуду
Месть виллиМесто сынаМесто встречи...
![Page 15: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/15.jpg)
Radix trie
![Page 16: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/16.jpg)
Advanced approaches
![Page 17: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/17.jpg)
Fuzzy search factors
Трансформеры
перестановка трансфромеры
вставка транссформеры
удаление тр_нсформеры
замена трансформиры
![Page 18: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/18.jpg)
Levenshtein distance
Ученый-математик, доктор физико-математических наук.
Функцию его имени можно найти в любом языке программирования.
http
://et
hw.o
rg/im
ages
/8/8
3/Le
vens
htei
n.jp
g
![Page 19: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/19.jpg)
Jaro-Winkler distance
s - длина словаm - кол-во совпавших сомволовt - половина транспозицийl - длина общего префиксаp - коэф.сглаживания (~0.1)
![Page 20: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/20.jpg)
Jaro-Winkler distance
m = 5
W1W2
d
хорошошорох
0.7
пробкакоробка
0.78
звездазвездный
0.89
![Page 21: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/21.jpg)
Jaro-Winkler distance
![Page 22: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/22.jpg)
Depth-first search (DFS)
A
B C
DC A C
DB D E F
Needle = ABCD
ABCDABC(B→D)AB(+C)DA(C→B)CDA(C→B)C(E→D)A(C→B)C(F→D)A(+B)C(A→D)
![Page 23: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/23.jpg)
Depth-first search (DFS)
![Page 24: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/24.jpg)
N-gram
АЛО начАЛО, русАЛОчка, волшебное зеркАЛО ...
АЧА нАЧАло, удАЧА, нАЧАльники ...
НАЧ НАЧало, НАЧальники, геркулес НАЧало ...
ЧАЛ наЧАЛо, мгновения пеЧАЛи, наЧАЛьники ...
Начало (Inception), 2010 год
![Page 25: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/25.jpg)
N-gram
![Page 26: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/26.jpg)
N-gram
●○○
●○○
●○
*по моему субъективному опыту
![Page 27: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/27.jpg)
N-gram
![Page 28: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/28.jpg)
Lookup improvements
![Page 29: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/29.jpg)
Template hashing
T R A N S F O R M E R S
1 0 1 0 0 0 1 1 1 1 0 0 0
A C E G I K M O Q S U W Y
B D F H J L N P R T V X Z
![Page 30: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/30.jpg)
Template hashing
![Page 31: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/31.jpg)
Bloom filter
0 1 0 0 0 0 1 1 0 0
Needle
![Page 32: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/32.jpg)
Bloom filter
![Page 33: SE2016 Java Alexey Tokar "To find a needle in a haystack"](https://reader031.fdocuments.us/reader031/viewer/2022030313/58eca78b1a28ab2a598b457d/html5/thumbnails/33.jpg)
Thank you! Questions? :)