HowtosetupOrchestratortomanagethousandsofMySQLservers

SimonJMudd |3rd October2017

SessionSummary

• Whatisorchestratorandwhyuseit?• Whathappensasyoumonitormoreservers?• Featuresaddedtomakeitscaleandimproveusability• Usingorchestratoratsmallerscale• Wayforward

1

Booking.com

• Oneofthelargesttravele-commercesitesintheworld• part ofthePricelineGroup(NASDAQ:PCLN)• Weofferaccommodationin228countries• Ourwebsiteandcustomerservicein40languages• Morethan15,000employeesin204officesin70countries

• WeusethousandsofMySQLservers:• weuseorchestratortomanagethetopologyandhandlemasterandintermediatemasterfailures

2

WhatisOrchestratorandwhyuseit?

3

Orchestrator

4

Orchestrator

• WrittenbyShlomi Noach• hestartedonthisatoutbrain andisnowworkingatgithub.com• Heintroducedbooking.com toorchestratorabout3yearsagowhenwewerelookingforsomethingtohandlefailoversautomatically

5

Orchestrator

• PeriodicallymonitorsMySQLserversandcheckstheirhealth• Handlesmasterfailover,butalsodoesmuchmore…• GUItomanageandvisualise topology– veryhandy• CLItodothesametasks– usedforscripting• APIcallstorunatadistance• NeedsaDBbackendtostorestate.

• NormallyMySQLbutcanbeSQLite

• Writteningo

6

Orchestrator

Whatfailuresdoesithandle?• Masterfailures

• Optionalhookstoexternalsystemswhichneedtobeawareofthesefailures

• Intermediatemasterfailures• Doesnot careaboutleafslavesorapplications• WorkswithOracleorMariaDB GTID• WorkswithoutGTID:CanaddPseudo-GTID (eventsinjectedonthemasterareusedtofindamatch)sononeed tomigrate

• Handlesmulti-leveltopologies7

OrchestratorGUI

8

OrchestratorGUI

9

OrchestratorGUI

10

Orchestrator

TopologyManagement• DraganddropusingtheGUI

• Moveoneslaveabout• Moveallslaves

• ScriptablerelocationfromthecommandlineorusingAPIcalls

11

Whathappensasyoumonitormoreservers?

12

Whathappensasyoumonitormoreservers?

• Integrationneededwithinternalinfrastructure• Deployment:tellorchestratortodiscoverandforgetservers*• Determinecandidatemasters• Handlespecialcases:

• testMySQLversions,specialsetups(black- orwhite-listserversorclusters)

• MakeorchestratorHA• Monitororchestratorbehaviour andperformance• Providewideraccesstodifferenttypesofuser

13*Itcanautomaticallydetectnewserversinanexistingclusterbutnotnewdetectnewclusterswithouthelp

IntegrationwithInternalInfrastructure

• Populatethemetadatadb onthemasterto:• Maphostorinstancenamestomorefamiliarclusternames• Howtodeterminereplicationdelay• Configurationofacceptablelevelsofreplicationdelay

• Addandremovalofservers/instancesastheyaredeployedorremovedfromservice

• SetupofPseudo-GTID(ifnotusingGTID)

14

IntegrationwithInternalInfrastructure

• Addfailoverhooksformonitoring,notificationandtotakesite-specificactions(tellothersystemsaboutthenewmaster)

• Selectionofcandidatemasters• Blacklistingserverswhicharenotsuitable:backupservers,testservers,serversinthewrongnetworkareas…

15

BetterVisibility

• Improveorchestratordeploymentvisibility• Foreachrunningapp:showhost,version,uptime• Showtheactivenodeandhowlongit’sbeenactive

• AuditingofMySQLfailuresandrecoveryviatheGUIisgoodandimproving

• noneedtosearchthelogs

16

BetterVisibility

17

Featuresaddedtoscaleandimproveusability

18

Performance

Wefoundbottlenecksespeciallyonstartup• Trytodiscoverseveralthousandmysql serversatonceandupdatethebackendatthesametimeàmax_connections exceeded

• Multiplegoroutinestryingtopollthesamestuckserver

Solution:• FIFODiscoveryqueuewhichavoidsduplicatesandlimitsmaximumdiscoveryconcurrency

19

Performance

Howtofigureoutwhat’sgoingon?• Understandingloggingishardatthisscale– toomuchnoise• Nodiscoverymetricstoseeproblemsatserveroraggregatelevel

Solution:• CollectdiscoverymetricsandkeepforNseconds• Logdiscoverytimesindebugmode• Provideinterfacetoretrieveraworaggregatevaluestouseinmonitoringsystems

20

Performance

Discovery(Poll)times

21

Performance

Discovery(Poll)counters

22

Performance

• Aclientupgrademightupgradethedatabasewhichotherolderappswerestillusing

Solution:• Makeauto-upgradeofthedatabaseoptionalsotheDBAcontrolsthis

23

Performance

• Crosszone(dc)accesschangesperformanceprofilesignificantlyandcausedproblems

• orchestratorappsaresupposedtobeeasytoreplaceandlocationshouldnotmatter

• latencycanbearealenemySolution:• Batchupdatesofsomedataintosmallernumberoflargerinserts• Collectmetricsonthesetimings• Catchdiscoverieswhichtaketoolong(internalcodebottlenecks)• Visibilityofthemetricsmadeiteasiertolocatecauses

24

Performance

• Specialconnectionssettings• "MySQLOrchestratorMaxPoolConnections":controlgopoolsize• "MySQLConnectTimeoutSeconds":1

• don’twastetimewaitingtoconnecttoadeadserver

25

Performance

golang specific-isms• Orchestratorbydefaultusesdatabase/sql andbydefaultsendsaquerywithparametersusingMySQL’sPrepare/Executesyntax

• Thisgenerates2rtt’s andonslowerconnectionscanaffecttheelapsedtimetocompleteaquery

• OptionstodisablethisbyinterpolatingparametervaluespriortosendingSQL

• Go(orchestratorcode)isquitehappytotry topoll10,000serversatonce

• Sometimesthatisnotsensible• Throttlingtoavoidthunderingherdisnecessary

26

OrchestratorHA

• Morethenoneorchestratorserverperzone/dc• Someupgradesreallyeasy– justrestartwithnewbinaries

• Commonendpointvialoadbalancer• Simplerforusers• worksforapi callsandmaysimplifyfirewallrules

27

28

OrchestratorHA

LoadBalancer

app1 app2 app3 app4

nginx1 nginx2 nginx3 nginx4

backend

Zone1 Zone2

OrchestratorHA

MightIhavemorethanoneorchestratorcluster?• Yesforactivedevelopment

• asaside-effectgivesusextraredundancy• Developmentloadistoosmalltocatchmanyissues• Recoveriesdisabledglobally onthisclusterbutmonitoringworksthesame

• Complianceregulationsmayrequiresegregationofdifferentnetworks

29

OrchestratorHA

Solution• MovefromusingorchestratorbinarytouseclusterAPIinterface

• Recentlymigratedtouseneworchestrator-clientcommandwhichsolvesthesameproblemandwasneededfororchestrator/raftaccess

• Simplifiesconfiguration• Allowseasyaccesstomorethenoneorchestratorcluster• Orchestratorupgradeswithdb backendchangesareeasier

30

OrchestratorAPI

EnhancementstoAPIcalls• Bulkretrievalofinstanceinformationandpromotionrules• Asynchronousdiscoverycall(e.g.bootstrapnewcluster)• Moremonitoringinformationavailable

• Discoverytimingmetrics• Discoveryqueuemetrics• Backendwritemetrics

31

SpecialCases

• TestingMySQL8.0orMariaDB 10.3?• “Let’snotpromotetothisbox”• Sameapplieswhiletestingnewminorversionsofcourse

• Sometopologieshaveslaveswithaggregatedata• Donot treatthemasanormalbox– shouldnot becandidatemasters

• OrchestratorcannothandleGRormulti-sourcereplicationyet• Besttoavoid theseboxes(forautomaticfailover)untilwehavesolutions• Patcheswelcometosolvesuchmissingfunctionality

32

SpecialCases

HandlingTLSconnections• OrchestratorcouldhandleusingTLSornotusingitbut…• SomeserversneedtobeaccessedbyTLS,othersdon’t(ODBCaccessormoresecuritysensitive systems)

• Orchestratorcouldnothandlethis• Codeaddedtorecognise errorandautomaticallyswitchtoTLS:

• Error 3159: Connections using insecure transport are prohibited while --require_secure_transport=ON

• GlobalOFF button– givesyoupeaceofmind

33

ProvideWiderUserAccess

• Orchestratorfanclub• Differentgroupsofuserslikeorchestrator• DBAs,Developers,Sysadmins,Auditors,Managers

• Usenginx (orsimilar)• Providesauthentication• ProvidesTLS• Thecombinationcanbeusedwithunix groupstoallowuser oradmin accesstoorchestrator

• Combinedwithaloadbalancerprovideseasyaccessforusersandalsoforapplications(usingapi calls)

34

Monitoring

Somethingstomonitor• Orchestratorprocess(andnginx)• Orchestratorclusterendpoint• SuccessfulorfailedDiscoveriesperminute• Discoveryqueuesizes• Discoverytimings

• aggregatedatagivesmean,medianandpercentiles• DiscoveriesexceedingInstancePollSeconds• Whenchangingactiveorchestratornodethesevaluesmay change

35

Booking.com contributions

Commitstopublicorchestratorrepo• Simon:170• Dmitry:40• Mauro:15• Daniël:8• Shlomi:many(whileworkingatbooking)

36

Usingorchestratoratsmallerscale

37

Usingorchestratoratsmallerscale

Notmentionedherebut• ConsideruseofSqlite – goodstartingpoint– singlebinary• ConsideruseofSqlite/raft

• providesHA• allnodesmonitorallMySQLservers

• Onlydifferenceisthedb backend• Notsurewherescalinglimits

38

Configurationsettings

Settingstobeconsidered,brokendownbyfunction

39

Configurationsettings

• MySQLbackend• "MySQLOrchestratorCredentialsConfigFile":"/path/.my-orchestratordb.cnf"• "MySQLOrchestratorDatabase":"orchestrator”• "MySQLOrchestratorHost":"orchestratordb.example.com"• "MySQLOrchestratorPort":3306• "MySQLOrchestratorMaxPoolConnections":100• "MySQLConnectTimeoutSeconds":1

• Sqlite backend• "BackendDB":"sqlite”• "SQLite3DataFile":"/var/lib/orchestrator/orchestrator.db"

40

Configurationsettings

• Psuedo-GTIDSettings(ifusingpseudo-gtid)• PseudoGTIDPattern• PseudoGTIDMonotonicHint• DetectPseudoGTIDQuery

41

Configurationsettings

• Clusterandhostsettings• Querymetadatadb (populatedexternally)todetectclusters• DetectClusterAliasQuery• DetectClusterDomainQuery

42

Configurationsettings

• Recoverysettings• Regexp filters– verysitedependent• RecoverMasterClusterFilters – white-listmastersbyclustername• RecoverIntermediateMasterClusterFilters• PromotionIgnoreHostnameFilters – ignoreserversfrombeingpromoted*• RecoveryIgnoreHostnameFilters – ignorespecialserversfromrecovery

43*Doesnotscalewell

Configurationsettings

• Failoversettings• OnFailureDetectionProcesses – whattodowhenafailureisdetected• PreFailoverProcesses – whattodopriortostartingrecovery• PostFailoverProcesses – whattodoaftercompletingrecovery• PostUnsuccessfulFailoverProcesses – whattodoifrecoveryfails• PostMasterFailoverProcesses – whattodoafterIMrecovery• PostIntermediateMasterFailoverProcesses – whattodoafterMasterrecovery

44

Configurationsettings

• Authenticationsettings(e.g.ifusingnginx withLDAP)• "AuthenticationMethod":"proxy",• "HTTPAuthUser":”user1",• "HTTPAuthPassword":”pass1",• "AuthUserHeader":”SomeHeader",• “PowerAuthUsers":["api-user1","api-user2",”realuser1"]• PowerAuthGroups":[”special_sysadmins”,“dbas”],

45*Doesnotscalewell

Configurationsettings

• Environmentsettings(e.g.shorten/simplifyhostnames)• “DataCenterPattern”• “PhysicalEnvironmentPattern”:• “RemoveTextFromHostnameDisplay”:“:.example.com:3306”

46

WayForward

47

WayForward

• Improvementsneededtotackleproblemsatbothendsofthescale• Smallerinstallations– forgettingonboard• Largerinstallations– toallowforfurtherscaling

48

WayForward

• Simplifyconfigurationandentrytoorchestrator• Shlomi isdoingaverygoodjobwithsqlite andraftsetups• Configurationcouldbesimplerandmoreautomaticformostpeople• Needtostandardise orchestratorsetupsmore?

• ExtendfunctionalitytocovermoreoftheMySQLeco-system• AWSandothercloudsystems• GroupReplicationorGalera• Multi-source

49

WayForward

• Distributionofdiscoveriesamongstallorchestratornodes• Orchestrator/raft:allnodesmonitorallMySQLservers

• Raftusagerecommendshavingseveralnodes• Orchestrator/MySQL:onenodemonitorsallMySQLservers• Better:distributemonitoringamongstavailablenodes

• Avoidsunnecessaryloadonmonitoredservers• reducesworkonbusyorchestratorapps• Usefulforsmallandlargeinstallations

• efficientbalancingisharder

50

WayForward

• Reducerecoverytime• Speedingupdetectiontorecoverytime wouldbegoodasreducesdowntime• Shouldbepossibletoreacttofailureevent(knowingstateofotherservers)immediately

• statecurrentlystoredinbackenddb• analysisanddetectionphasehappensindependentlyofserverpolling

• Withreduceddefaultpolltimeof5secondsrecoveryislikelytobetriggeredwithin10seconds

• notcriticalformostpeople?

51

WayForward

• Furtherworkneededtoscalemore• bottlenecksstillexist• Largerinstallationskeepgrowing

• Improvemonitoring• ExternalAPIcalls• Addinternalmetrics

52

Conclusion

53

Doesitwork?

Icheckedforfailuresoverarecentperiod• 6masterfailures• About40intermediatemasterfailures• No-onecalledup• Noharmwasdone

54

Questions?

Thanks

Simon J Muddsimon.mudd@booking.com