Batch Indexing & Near Real Time, keeping things fast

Marc SturleseSoftware engineer @ Trovit

About me...

• Marc Sturlese – @sturlese

• Software engineer @Trovit. R&D focused

• Responsible for search and scalability

Agenda

• Who we are

• Batch architecture. Hadoop & Hive

• Near real time architecture. Storm & stuff

• Putting it all together

• Alternatives and Future directions

• Questions

Who we are

Trovit, a search engine for classifieds

Who we are

Batch Layer

• Hadoop based

• Documents are crunched by a pipeline of MR jobs

• Hive to save stats of each phase

Batch LayerPipeline overview

Incoming data

Deployment

Lucene Indexes

Ad Processor Diff Matching Expiration Deduplication Indexing

t – 1

External Data

Hive Stats

Hadoop Cluster

Batch LayerThe good things!

• Index always built from scratch. Small number of big segments

• Multicast deployment allows to send indexes to all slaves at the same time.

• Backups convenient on HDFS

Batch LayerThat was cool but...

• Not even close to real time

• Crunch documents in batch means to wait until all is processed. This can take a few hours

• We want to show the user fresher results!

Near real time LayerStorm and stuff to the rescue

Near real time LayerStorm properties

• Distributed real time computation system

• Fault tolerance

• Horizontal scalability

• Low latency

• Reliability

Near real time LayerStorm in action

Slave

Slave

Solr prod replicas

SlaveXML feed

XML feed

Kafka partition

Kafka partition

Storm topologySources

Kafka spout

Kafka spout

XML spout Doc Manager bolt Indexer bolt

SHUFFLEGROUPING GROUPING

FIELD

Near real time LayerStorm in action

• Spouts just read and send

• Doc Manager Bolt processes and classifies

• Indexer Bolt adds documents to Solr

• Replicated logic with different implementation

• Careful not to overload Solr slaves...

Near real time LayerStorm in action

Near real time LayerStorm in action. But...

Near real time LayerStorm in action. But...

• Now Solr has to handle user queries and storm inserts

• Field grouping on Indexer Bolt for politeness

• Small bulks to reduce insert requests

• Committing on many cores, same host, same time can be painful

Near real time LayerStorm in action - Committing

Indexer Bolt Cars US

Real state UK R1 Cars US R1 Cars US R2 Jobs BR R1 Jobs BR R2 Real state ES R1

Indexer Bolt Jobs BR

ZooKeeper Locker

Slave 1 Slave 2 Slave N

. . .

Near real time LayerStorm in action

• Adding documents now is fast

• Keep number of segments small

• Avoid merges on big segments

• Just add new docs (no deletes or updates)

Mixed ArchitecturePutting it all together

15

Slave

Slave

Solr prod replicas

SlaveXML feed

XML feed

Kafka partition

Kafka partition

Storm topologySources

Hbase doc info

Bulk addExists?

MR Pipeline

zk

Mixed ArchitectureSwapping indexes

• NRT docs might not be contained in the new batch index (even fresher than the “being built” batch index)

• This can lead to inconsistencies...

Mixed ArchitectureSwapping indexes. Time jumps!

Mixed ArchitectureSwapping indexes

HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer

Mixed ArchitectureSwapping indexes

HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer

Mixed ArchitectureSwapping indexes

HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer

NRT t+1

NRT t+2

Mixed ArchitectureSwapping indexes

HBase

XML feed t

Slave t+1

Slave t

Pipeline t

Pipeline t+1

XML feed t+1

XML feed t+2

NRT indexerBatch indexer

NRT t+1

NRT t+2

Mixed ArchitectureSwapping indexes

• NRT indexed docs must be stored in a temporary storage

• Fetch missing docs from the storage and add them before the next deploy

• This avoids time jumps

Mixed ArchitectureStorm and Hadoop

• Near real time inserts, low latency

• Hadoop handles deletes and updates. No rush on those

• No merges on big segments so optimal query response times

• Tolerant to human errors

• Temporary lost of accuracy on the NRT layer

AlternativesSolrCloud - Why not?

• Good for the vast majority of use cases

• Incremental inserts/updates/deletes oriented. Pay segment merges per real time

• Need to deploy full indexes fast (faster that rsync or http replication)

• Now full deploy easier with aliases

Future linesLucene real time feature

• Allows to see docs in the index before they are committed

• Good but not a must right now for the use case

• Very easy to integrate on the current architecture

??

Thanks for your attention!

Marc Sturlesemarc@trovit.com

Lucene/Solr Revolution 2013, San Diego, May 1 2013

CONFERENCE PARTYThe Tipsy Crow: 770 5th AveStarts after Stump The ChumpYour conference badge gets you in the door

TOMORROW Breakfast starts at 7:30Keynotes start at 8:30