New Advanced(usage(of(indexes(in(...
Transcript of New Advanced(usage(of(indexes(in(...
![Page 2: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/2.jpg)
Presentation(overview
� Structure4of4Coherence4index� How4IndexAwareFilterworks� Multiple4indexes4in4same4query� Custom4index4provider4API4(since43.6)� Embedding4Apache4Lucene4into4data4grid
![Page 3: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/3.jpg)
Creation(of(index
QueryMap.addIndex(ValueExtractor extractor, boolean ordered, Comparator comparator)
Attribute(extractor,(used(to(identify(index(later
Index(configuration
![Page 4: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/4.jpg)
Using(of(query(API
public interface QueryMap)extends Map){Set)keySet(Filter)f)3Set)entrySet(Filter)f)3Set)entrySet(Filter)f, Comparator)c)3...}
public interface InvocableMap)extends Map){Map)invokeAll(Filter)f, EntryProcessor)agent)3Object)aggregate(Filter)f, EntryAggregator)agent)3...}
![Page 5: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/5.jpg)
Indexes(at(storage(node
extractor indexextractor index
extractor index
Indexes
Backing map
Named cache backend
SimpleMapIndexReverse map Forward map
valkeykeykeykey
val keykey
val keykey
key valkey valkey val
![Page 6: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/6.jpg)
Indexes(at(storage(node
� All4indexes4created4on4cache4are4stored4in4map
� Reverse4map4is4used4to4speed4up4filters� Forward4map4is4used4to4speed4up4aggregators
Custom(extractors(should(obey(equals/hashCode contract!
QueryMap.Entry.extract���
is(using(index,(if(available
![Page 7: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/7.jpg)
Indexes(at(storage(node
� Index4structures4are4stored4in4heap� and4may4consume4a4lot4of4memory
� For4partitioned4scheme� keys4in4index4are4binary(blobs,� regular(object,4otherwise
� Indexes4will4keep4your4key4in4heap4even4if4you4use4off4heap4backing4map
� Single4index4for4all4primary(partitions of4cache4on4single4node
![Page 8: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/8.jpg)
How(filters(use(indexes?
interface IndexAwareFilter extends EntryFilter {int calculateEffectiveness(Map)im, Set)keys)3Filter)applyIndex(Map)im, Set)keys)3
}
� applyIndex��������� ����������������������� � �� ��� calculateEffectiveness������������ ��������������� ������
nested4filters� each4node4executes4index4individually� For(complex(queries(execution(plan(is(calculated(ad(hoc,(
each(compound(filter(calculates(plan(for(nested(filters
![Page 9: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/9.jpg)
Example:(equalsFilter
Filter4execution4(call4to4applyIndex()4)� Lookup4for4matching4index4using4extractor instance4as4key
� If4index4found,4� lookup4index4reverse4map4for4value� intersect4provided4candidate(set with4key(set from4reverse4map
� return4null � candidate4set4is4accurate,4no4object4filtering4required
� else4(no4index4found)� return4this � all4entries4from4candidate(set should4be4deserialized and4evaluated4by4filter
![Page 10: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/10.jpg)
Multiple(indexes(in(same(query
Example:4ticker=IBM4&4side=Bnew AndFilter(
new EqualsFilter��getTicker�����������new EqualsFilter��getSide��������
Execution4plan� call4applyIndex������������������ ��
� only4entries4with4ticker4IBM4are4retained4in4candidate4set
� call4applyIndex������������������� ��� only4entries4with4side=B4are4retained4in4candidate4set
� return4candidate4set
![Page 11: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/11.jpg)
Index(performance
PROs� using4of4inverted4index� no4deserialization4overheadCONs� very4simplistic4cost4model4in4index4planner� candidate4set4is4stored4in4hash4tables4(intersections/unions4may4be4expensive)
� high4cardinality4attributes4may4cause4problems
![Page 12: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/12.jpg)
Compound(indexes
Example:4ticker=IBM4&4side=B� Index4per4attribute
new AndFilter(new EqualsFilter��getTicker���������, new EqualsFilter��getSide�������)
� Index4for4compound4attributenew EqualsFilter(new MultiExtractor��getTicker, getSide��,
Arrays.asList�� ����� ����������������
���� ���������������������������������������������extractor(used(to(create(index!
![Page 13: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/13.jpg)
Ordered(indexes(vs.(unordered
19.23
1.63 1.37
0.61 0.721.19
0.1
1
10
100
Term4count4=4100k Term4count4=410k Term4count4=42k
Filte
r(executio
n(tim
e((m
s)
Unordered Ordered
![Page 14: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/14.jpg)
Custom(indexes(since(3.6
interface IndexAwareExtractor
extends ValueExtractor {
MapIndex createIndex(boolean ordered,Comparator)comparator,Map)indexMap,BackingMapContext bmc)3
MapIndex destroyIndex(Map)indexMap)3}
![Page 15: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/15.jpg)
Ingredients(of(customs(index
� Custom4index4extractor� Custom4index4class4(extends4MapIndex)� Custom4filter,4aware4of4custom4index+� Thread4safe4implementation� Handle4both4binary4and4object4keys4gracefully� Efficient4insert4(index4is4updates4synchronously)
![Page 16: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/16.jpg)
Why(custom(indexes?
Custom4index4implementation4is4free4to4use4any4advanced4data4structure4tailored4for4specific4queries.
� NGram index4� fast4substring4based4lookup� Apache4Lucene index4� full4text4search� Time4series4index4� managing4versioned4data
![Page 17: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/17.jpg)
Using(Apache(Lucene in(grid
Why?� Full4text4search4/4rich4queries� Zero4index4maintenancePROs� Index4partitioning4by4Coherence� Faster4execution4of4many4complex4queries4CONs� Slower4updates� Text4centric
![Page 18: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/18.jpg)
Lucene example
Step41.4Create4document4extractor//)First,)we)need)to)define)how)our)object)will)map)
//)to)field)in)Lucene)document)
LuceneDocumentExtractor)extractor)= new LuceneDocumentExtractor()3extractor.addText("title", new ReflectionExtractor("getTitle"))3extractor.addText("author", new ReflectionExtractor("getAuthor"))3extractor.addText("content", new ReflectionExtractor("getContent"))3extractor.addText("tags", new ReflectionExtractor("getSearchableTags"))3
Step42.4Create4index4on4cache//)next)create)LuceneSearchFactory)helper)class)
LuceneSearchFactory)searchFactory)= new LuceneSearchFactory(extractor)3//)initialize)index)for)cache,)this)operation)actually)tells)coherence)
//)to)create)index)structures)on)all)storage)enabled)nodes)
searchFactory.createIndex(cache)3
![Page 19: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/19.jpg)
Lucene example
Now4you4can4use4Lucene4queries//)now)index)is)ready)and)we)can)search)Coherence)cache)
//)using)Lucene)queries)
PhraseQuery)pq)= new PhraseQuery()3pq.add(new Term("content", "Coherence"))3pq.add(new Term("content", "search"))3//)Lucene)filter)is)converted)to)Coherence)filter)
//)by)search)factory)
cache.keySet(searchFactory.createFilter(pq))3
![Page 20: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/20.jpg)
Lucene example
You4can4even4combine4it4with4normal4filters//)You)can)also)combine)normal)Coherence)filters
//)with)Lucene)queries)
long startDate)
= System.currentTimeMillis() : 1000 * 60 * 60 * 243//)last)day)
long endDate)= System.currentTimeMillis()3BetweenFilter)dateFilter)
= new BetweenFilter("getDateTime", startDate, endDate)3Filter)pqFilter)= searchFactory.createFilter(pq)3//)Now)we)are)selecting)objects)by)Lucene)query)and)apply)
//)standard)Coherence)filter)over)Lucene)result)set)
cache.keySet(new AndFilter(pqFilter, dateFilter))3
![Page 21: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/21.jpg)
Lucene search(performance
0.72
0.71
1.10
1.09
3.30
1.80
1.16
1.18
4.38
4.39
1.93
1.96
2.38
2.38
0.67
7.23
1.49
7.77
8.81
8.75
1.53
8.66
15.96
15.96
11.15
11.12
52.59
8.74
0.5 5 50
A1=x4&4E1=y
E1=x4&4A1=y
D1=x4&4E1=y
E1=x4&4D1=y
E1=x4&4E2=y
E1=x4&4E2=Y4&4E3=z
D1=w4&4E1=x4&4E2=Y4&4E3=z
E1=x4&4E2=Y4&4E3=z4&4D1=w
A24in4[n..m]4&44E1=x4&4E2=Y4&4E3=z
E1=x4&4E2=Y4&4E3=z4&4A24in4[n..m]
�#������#����#" �!���#&��!��$&��!��%&�
�#&��!��$&��!��%&��!��#������#����#"
H1=a4&4E1=x4&4E2=Y4&4E3=z
E1=x4&4E2=Y4&4E3=z4&4H1=a
Filter(execution(time((ms)
Lucene
Coherence
![Page 22: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/22.jpg)
Time(series(index
Special4index4for4managing4versioned4data
Getting4last4version4for4series4kselect * from versions)where series=k and version)=
(select max(version) from versions)where key=k)
Series key Entry id Timestamp Payload
Entry key Entry value
Cache entry
![Page 23: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/23.jpg)
Time(series(index
Series inverted index
Series key
Series key
Series key
Series key
Series key
HA
SH
TAB
LE
Timestamp Entry ref
Timestamp Entry ref
Timestamp Entry ref
Timestamp inverted subindex
ORDER
![Page 24: New Advanced(usage(of(indexes(in( Oracle(Coherencecoherence.java.net/assets/pdf/2011-11-04-Alexey-Ragozin... · 2020. 7. 1. · Advanced(usage(of(indexes(in(Oracle(Coherence Alexey](https://reader033.fdocuments.us/reader033/viewer/2022060600/6053f4779249891ac45c7639/html5/thumbnails/24.jpg)
Thank4you
Alexey [email protected]
http://aragozin.blogspot.come my4articleshttp://code.google.com/p/gridkite my4open4source4code