Hotel Search, Scalability, and Apache Ignite · 2018-11-15 · Hotel Search, Scalability, and...
19
Hotel Search, Scalability, and Apache Ignite Musaul Karim • Senior Consultant • June 2018
Transcript of Hotel Search, Scalability, and Apache Ignite · 2018-11-15 · Hotel Search, Scalability, and...
Hotel Search, Scalability, and Apache Ignite
Musaul Karim • Senior Consultant • June 2018
Introduction
A G E N D A
In-Memory Computing Summit, London • 25-26 June 2018
HotelSearchSystems
Architecture
Successes&Challenges
Questions
AboutMe
InitialCareer
§ 2000- StartedasaC++developer
§ 2003- TookabreaktodomyMSc
§ 2005- BackintoworldofworkatDeloitte
In-MemorySystems
2007- Fidessa
§ HighTransactionOrder&ExecutionManagementSystem
§ In-housedevelopedDistributedCacheSystemsforTradeData
2010- Barclays
§ MigratedDBMSbasedRiskCalculationenginetoanIn-MemoryCache&Computesystem
§ HybridIn-housetech+SolaceSystems+OracleCoherence
2013- CreditSuisse
§ OracleCoherencebasedPrimeServicesRiskSystem
SoftwareConsultant● In-memory&DistributedSystemsSpecialist ● MScDistributedComputing
In-Memory Computing Summit, London • 25-26 June 2018
CGConsultancy
§ ITSystemMigrationProjects
§ TechnologyAssessment
§ OptionswithintheModernLandscape
§ ProofofConcepts
§ LeadingFollowupDevelopmentWork
§ OverallTechnicalArchitecture
Travelsectorclients
§ JacTravel
§ OAG
§ Recentlystartedworkingwithoneofthelargesttraveloperators
Hotel Search Systems
In-Memory Computing Summit, London • 25-26 June 2018
HotelSearchSystemOverview§ HandlesHotel/RoomSearchrequestsviaaB2BAPI
§ ReceivesupdatesintradayasstreamsaswellasbatchesfromBookingSystemsandotherThirdPartySupplierSystems
§ ReturnsPricedRoomsmatchingtheSearchCriteria
§ MatchesHotelsbasedonlocationssearched(Canalsosearchforspecifichotels)
§ MatchesRoomsbasedonStayDateAvailabilityandOccupancyrequirementsetc.
§ Excludesroomsbasedonanydistributionrules
§ Calculatespricesforalltheroomoptions
§ TypicallymoreI/OboundthanCPU
§ ItrequiresalargenumberofqueriesagainstDatabaseTables(orCaches)ateachstage
§ Largenumberofcalculationstobeperformed.i.e.theyneedtobedoneforeachroom/specialoffer/room-extrasetc.
In-Memory Computing Summit, London • 25-26 June 2018
SearchJourney
In-Memory Computing Summit, London • 25-26 June 2018
Hotel&RoomSelection
SelectRooms- Roomcriteria- OccupancyRules
SelectHotels- Location- Contracts- DistributionRules
FilterbyAvailability- Availability- StayPeriod+restrictions
Cost&PriceCalculation(PerRoom,Dynamic)
ApplySpecialOffers&Supplements
CalculateCost
ApplyMargin
ApplyTax
FinaliseResult
BuildandReturnResponse
DeduplicateRooms
Architecture
In-Memory Computing Summit, London • 25-26 June 2018
PreviousInfrastructure atJacTravel§ TwoPlatforms
§ Oneretainedasabookingplatform(iVector)
§ Theotherbeingdecommissioned(TravelSudio).
§ BuiltonMicrosoftSqlServerandIIS(VB.NETandC#)
§ Over100SQLServer+IISInstances
§ Handled typicaltrafficof~140millionsearchesperday
§ AverageResponseTimeof2.5Seconds
§ Hardwareupgradedasmuchaspossible(e.g.SSDs)
§ Variousdatabaseoptimisations considered
§ Search-specific“cache”tables
§ In-memoryTablesinSQLServer.
§ Infrastructurecosttoohighandreachingdiminishingreturns
In-Memory Computing Summit, London • 25-26 June 2018
NewSearch-Grid Overview§ Server/CacheNodes
§ ApacheIgniteembeddedinSpringMVCservice
§ ClusterwithFullyReplicatedCaches
§ MostCachesOff-Heap
§ Processconsumesaround60GBmemory,includinga20GBJVMheap.
§ LoadedfromSQLDatabase(withnofurtherDBat“Search-Time”)
§ RequestsreceivedviaEmbeddedJettyandprocessedbyanIgniteService
§ 20nodeshandling~300millionsearches
§ UpdateClientNodes
§ SubscribestoaMessageQueue
§ ~200kupdatesintraday
§ UpdatesCachesusingacombinationofServicesandIgniteDataStreamers
§ UpdateswithnovisibleimpactonSearchProcess
In-Memory Computing Summit, London • 25-26 June 2018
Jetty
IgniteCaches
SearchRequestsoverHTTP
IgniteUpdateClient
MessageBus
UpdatesforAvailability,Rates,StaticDataetc
OverallArchitecture
In-Memory Computing Summit, London • 25-26 June 2018
Search-Grid Internals§ ~50Caches
§ FullyReplicated
§ MostareOff-Heap
§ CacheQueries
§ Directkeybasedaccesswherepossible
§ SQLFieldsandIndexesonlywhenSQLQueriesarenecessary
§ SearchRequest
§ ProcessedbyanIgniteService
§ SQLFieldsandIndexesonlywhenSQLQueriesarenecessary
§ ThreadsmanagedbyIgniteServicesPool
§ SearchprocessedusingaSinglethreadonaSingleNode
§ Thisallowsthesystemtobescaleduplinearly
In-Memory Computing Summit, London • 25-26 June 2018
Deployment§ Deploymenttestedon
§ PhysicalHosts
§ VM/CloudProviders:AWS,Azure,Rackspace
§ Zerodown-timeClusterdeployment&restart
§ Startingnewnodesonaseparatecluster(blue/green)
§ Fullyautomated– orchestratedusingAnsible
§ AdjustingClustertomatchTrafficVolume
§ CacheNodescanbeaddedorremovedtomatchTrafficVolume
§ Cacheswillrebalanceontonewnodes
§ TheEventmechanismcanbeusedtodeterminewhenallcachesarerebalanced
In-Memory Computing Summit, London • 25-26 June 2018
Successes & Challenges
In-Memory Computing Summit, London • 25-26 June 2018
Performance§ LoadTeston4Nodes
§ AWSm4.4xlarge
§ 16vCPU(2.3GHzXeonE5-2686)
§ RequestInjection
§ 8JMeterInjectornodes
§ 320requests/secateachstep
§ MeasurementsOverview
§ Cansustain960requests/secondwithoutbreaching1-secondSLAredlinefor99th %
§ Averageresponsetime:~20ms
§ 99th Percentile:~270ms
§ Requestsstartqueuingupbeyondthisrate
In-Memory Computing Summit, London • 25-26 June 2018
MigrationGains§ 90%reductionininfrastructure
§ 90%reductioninResponseTime
§ FasterResponse-Timeenablesnewuse-casestobeconsideredforthesearchprocess
§ LinearlyScalablebyaddingnewnodes
§ Predictabilitymakesinfrastructure/capacityplanningeasier
§ OpenSourcegrid-technologyrunningonLinux
§ Aidesquickandeasyprovisioningofad-hocDev/Testenvironments
§ MakesiteasiertohaveaDevOpsprocess
§ NewDevelopmentProcesses(BDD,TDD,CI/CD)
§ Visiblecorrelationbetweenuserstoriesandcode
§ Testcoverageprovidesmoreconfidencewhenmakingcomplexchanges
In-Memory Computing Summit, London • 25-26 June 2018
Migration Pains
§ Needformaintainingmultiplesystemsintheinterimperiod
§ NeedstoreplicatetheCalculationLogic,aspricesmustbeidenticaltoBookingSystem
§ ImplicitRounding basedonDatabaseFieldprecision– MultipleTempTables
§ Existingalgorithmsoptimised forDatabaseQueries/StoredProcedures
§ APIClientschangetheirSearchpattern/behaviorafternoticingtheimprovedperformance
§ IncreaseSearchRate
§ Increaseinlargerregion/citysearches
§ Introducingnewtechnologyrequirednewtoolsets&processesforauxiliaryfunctions
§ Replacingdatabasebasedmonitoring&reportingtools
§ Manyoptions.Neededabitofdiscoveryprocess.
In-Memory Computing Summit, London • 25-26 June 2018
SupportingServices§ 3rd-partySupplierCacheing
§ AmoreclassicalimplementationofaRead-throughcache
§ Reducingloadon3rd partypartners
§ Smartersearchestopartnersbasedonmostcommonsearchtypes
§ NativePersistence
§ Real-TimeStatistics/Analytics
§ Typesofsearchesbyclients
§ Locationsbeingsearched
§ SpikesinrequestsbyClients/Location
§ Integrationwith3rd partyproductsfordetailedanalytics/visualisation
In-Memory Computing Summit, London • 25-26 June 2018
TechnicalConsiderations§ WorkingwithLargeJVMHeaps
§ GarbageCollectorBenchmarking/Comparison/Tuning
§ Developmentconsiderationstoavoidlong“Stoptheworld”pauses
§ InitialRebalancingcantakealongtime
§ Needtomakeconsiderationsforzero-downtimedeployments
§ Igniteisproductwithalotofactivedevelopment
§ Greatforgettinglotsofnewusefulfeatures
§ Sometimesweneededhelpwithnewfeatures,sometimesthefeaturesneedsomeoptimisations
§ Whenwefoundbugs,GridGainhavehelpedbycreatingversionsforuscontainingthefixes
§ Professionalsupportontheseissues
§ Developerskillsetcanbemorebusinessfocusedcomparedtobuildingaplatformin-house.
In-Memory Computing Summit, London • 25-26 June 2018
