Managing a large scale SolrCloud cluster via APIsAnshum GuptaSr. Software Engineer, IBM WatsonApache Lucene/Solr Committer & PMC member

About me

• Anshum Gupta, Apache Lucene/Solr committer and PMC member, IBM Watson Search team.

• Interested in search and related stuff.• Apache Lucene since 2006 and Solr since 2010.

© 2014 International Business Machines Corporation 2

Apache Solr is the most widely-used search solution on the planet.

Solr has tens of thousands of applications in production.

You use everyday.

8,000,000+Total downloads

Solr is both establishedand growing.

250,000+Monthly downloads

2,500+Open Solr jobs and the largest

community of developers.

A little bit of history

• SolrCloud• Introduced in 4.0• Not Solr in the Cloud but set of distributed features for Solr

• No APIs to manage distributed clusters• We’ve come a long way since then

© 2014 International Business Machines Corporation 4

01SolrCloud - Physical Architecture

ZooKeeper

Node 1 Node 2

LoadBalancer

Client

Client

Client

Client

Client

Client

Client

Client

Client

Lots

Of

Interaction

Coins by Creative Stall from the Noun Project

SolrCloud Terminology

• Node: JVM process that hosts the Solr web application• Collection: Logic index distributed across multiple nodes• Shard: Logical slice of a collection. Each shard has a name, hash range

(optional), leader, and replicationFactor.• Replica: Physical Solr index that can accept updates or queries. Implemented as

a Solr core.• ReplicationFactor: Number of copies of each shard.• Leader: One replica per shard which assumes special duties like versioning.

They are elected using ZooKeeper.• Overseer: Special node that manages cluster level operations. Also elected

using Zookeeper.

© 2014 International Business Machines Corporation 6

CONFIDENTIAL

Exploring the APIs - Getting Started

© 2014 International Business Machines Corporation 7

Creating a collection

• HTTP API

• Response

© 2014 International Business Machines Corporation 8

http://localhost:8983/solr/admin/collections?action=CREATE&name=foo&numShards=2&replicationFactor=1

{"responseHeader":{ "status":0, "QTime":1848}, "success":{ "":{"responseHeader":{ "status":0, "QTime":1741}, "core":"foo_shard1_replica1"}}}

Add a replica

• HTTP API

• Response

© 2014 International Business Machines Corporation 9

http://localhost:8983/solr/admin/collections?action=ADDREPLICA&collection=foo&shard=shard1

{"responseHeader":{ "status":0, "QTime":1156}, "success":{ "":{"responseHeader":{ "status":0, "QTime":1147}, "core":"foo_shard1_replica4"}}}

+

We added more than what we need - Delete replica

• HTTP API

• Response

© 2014 International Business Machines Corporation 10

http://localhost:8983/solr/admin/collections?action=DELETEREPLICA&collection=foo&shard=shard1&replica=core_node3

{"responseHeader":{ "status":0, "QTime":111}, "success":{ "":{ "responseHeader":{ "status":0, "QTime":2}}}}

CONFIDENTIAL

Exploring the APIs - Rule based scaling out

© 2014 International Business Machines Corporation 11

Rule based Replica Assignment

• Use it during creation of:• Collection• Shard• Replica

• Rules based on:• Hostname• Free disk space• Number of cores• Node name• System property• Other Metrics

© 2014 International Business Machines Corporation 12

Rule based Replica Assignment

• How is a rule defined?

• When creating a new replica, apply to each node.• Supported Operators

© 2014 International Business Machines Corporation 13

f(shard, replica, tag)

default, > , < , !

Rule based Replica Assignment - Examples

• Example #1 - for any shard, limit number of replicas to 2 on any node

• Example #2 - For any replica, limit number of cores to 5 on a node

• Example #3 - Do not create any cores on host 198.21.2.19

© 2014 International Business Machines Corporation 14

shard:*,replica:<2,node:*

replica:*,cores:<5

host:!198.21.2.19

CONFIDENTIAL

Adjusting to growth - Moving data

© 2014 International Business Machines Corporation 15

Divide and Rule? - Let’s split a shard

• HTTP API

• Response

© 2014 International Business Machines Corporation 16

http://localhost:8983/solr/admin/collections?action=SPLITSHARD&collection=bar&shard=shard1

Too Verbose to Share!

Another way to move data - Migrate

• HTTP API

• Response

• What happens?• Move documents to target collection• Handle incoming updates and have temporary routing for original collection to

send updates to target

© 2014 International Business Machines Corporation 17

http://localhost:8983/solr/admin/collections?action=MIGRATE&collection=foo&split.key=a!&target.collection=bar

Too Verbose to fit in here!

CONFIDENTIAL

Monitoring your Cluster

© 2014 International Business Machines Corporation 18

LIST Collections - Let’s see what we have

• HTTP API

• Response

© 2014 International Business Machines Corporation 19

http://localhost:8983/solr/admin/collections?action=LIST

{"responseHeader":{ "status":0, "QTime":0}, "collections":["a", "bar", "foo"]}

CLUSTERSTATUS : Don’t we like details?

© 2014 International Business Machines Corporation 20

http://localhost:8983/solr/admin/collections?action=CLUSTERSTATUS

{"cluster":{ "collections":{ "foo":{ "replicationFactor":"1", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node1":{ "core":"foo_shard1_replica1", "base_url":"http://192.168.1.125:8983/solr", "node_name":"192.168.1.125:8983_solr", "state":"active", "leader":"true"}, "core_node2":{ "core":"foo_shard1_replica2", "base_url":"http://192.168.1.125:8983/solr", "node_name":"192.168.1.125:8983_solr", "state":"active"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"100", "autoAddReplicas":"false", "znodeVersion":14, "configName":"a"}}, "live_nodes":["192.168.1.125:8983_solr"]}}

• HTTP API

OVERSEERSTATUS - More than a sneak peek

© 2014 International Business Machines Corporation 21

http://localhost:8983/solr/admin/collections?action=OVERSEERSTATUS

• HTTP API

• Response - Verbose. Actually, more than that!• Stats for:

• Errors• Time taken• Much more

CONFIDENTIAL

Managing your Cluster

© 2014 International Business Machines Corporation 22

Set a specific node to be the Overseer

© 2014 International Business Machines Corporation 23

http://localhost:8983/solr/admin/collections?action=ADDROLE&role=overseer&node=192.168.1.125:8983_solr

• ADD Role

• Response - Nothing substantial but a 200 on success• Supported role:

• Only Overseer for now• Remember - It’s only a preference

Setting a preferred leader

© 2014 International Business Machines Corporation 24

http://localhost:8983/solr/admin/collections?action=ADDREPLICAPROP&shard=shard1&collection=foo&replica=core_node2&property=preferredLeader&property.value=true

• Using ADDREPLICAPROP

• Sets ‘preferredLeader’ to ‘true’. The only pre-defined property.

Setting preferred leaders for the entire collection

© 2014 International Business Machines Corporation 25

http://localhost:8983/solr/admin/collections?action=BALANCESHARDUNIQUE&collection=foo&property=preferredLeader

• Using BALANCESHARDUNIQUE

• Sets ‘preferredLeader’ to ‘true’ evenly across the collection

Rebalance the leaders

© 2014 International Business Machines Corporation 26

http://localhost:8983/solr/admin/collections?action=REBALANCELEADERS&collection=foo

• Using REBALANCELEADERS

• This allows preferredLeaders to take up leadership for shards.

I lost my leader! Where do I go now?

© 2014 International Business Machines Corporation 27

http://localhost:8983/solr/admin/collections?action=FORCELEADER&collection=foo&shard=shard1

• Force a leader election using FORCELEADER API

• This can ONLY be used when the shard has no leader.

CONFIDENTIAL

Recipe #1 - Shard Splitting

© 2014 International Business Machines Corporation 28

Splitting a shard - It’s more than just the single API call

• Monitor disk-size• CLUSTERSTATUS & Core STATUS

• SPLITSHARD• Make sure there’s enough spare disk space

• Add one more replica• Force leader election (DELETEREPLICA)• Delete old INACTIVE shard (Parent)

© 2014 International Business Machines Corporation 29

CONFIDENTIAL

Recipe #2 - Ensuring high availability

© 2014 International Business Machines Corporation 30

High availability through a monitoring application

• Auto-add replica• Monitor cluster status - CLUSTERSTATUS API• ADD Replica using replica placement strategy• Remove replicas when there are too many

© 2014 International Business Machines Corporation 31

CONFIDENTIAL

Recipe #3 - Migrating Cluster Infrastructure

© 2014 International Business Machines Corporation 32

Migrating cluster infrastructure

• CLUSTERSTATUS, and OVERSEERSTATUS• Overseer, Leaders, etc.

• Move the Overseer first OR last• Rebalance leaders• Force leader elections if required*

© 2014 International Business Machines Corporation 33

*The assumption is you know what you’re doing!

CONFIDENTIAL

What’s more?

© 2014 International Business Machines Corporation 34

There’s a lot more, a single session is too less!

• Version 2 APIs• SolrJ support for all the APIs - Easy to tool in Java• Security• Cross Data Center Support

© 2014 International Business Machines Corporation 35

Coming Soon - SolrCloud Backup and Restore!

• Committed just a few days ago• Back up and restore of

• Collections• ZK metadata• Configs

• Release with 6.1 (and may be 5.6)

© 2014 International Business Machines Corporation 36

http://www.twitter.com/anshumguptahttp://www.linkedin.com/in/anshumgupta/anshum@apache.org

Connect @