Building High Scalable Distributed Framework on Apache Mesos
-
Upload
rahul-kumar -
Category
Documents
-
view
117 -
download
0
Transcript of Building High Scalable Distributed Framework on Apache Mesos
![Page 1: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/1.jpg)
BuildingHighScalableFramework onApache Mesos
RAHULKUMAR
TECHNICALLEAD,SIGMOID
![Page 2: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/2.jpg)
“A distributed system is a collection of independent computers that appears to its
users as a single coherent system.”
![Page 3: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/3.jpg)
A distributed system application works independently and communication through messages.
q Resource Sharingq Opennessq Concurrencyq Scalabilityq Fault Toleranceq Transparency
![Page 4: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/4.jpg)
MesosIntro
“Apache Mesos abstracts CPU, memory, storage, and other compute resources away from machines (physical or virtual), enabling fault-tolerant and elastic distributed systems to easily be built and
run effectively.”
![Page 5: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/5.jpg)
![Page 6: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/6.jpg)
![Page 7: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/7.jpg)
![Page 8: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/8.jpg)
SoftwareProjectsBuiltonMesos
q Aurora isaserviceschedulerthatrunsontopofMesos,enablingyoutorunlong-runningservicesthattakeadvantageofMesos'scalability,fault-tolerance,andresourceisolation.
q Marathon isaprivatePaaSbuiltonMesos.Itautomaticallyhandleshardwareorsoftwarefailuresandensuresthatanappis“alwayson”.
q Spark isafastandgeneral-purposeclustercomputingsystemwhichmakesparalleljobseasytowrite.
q Chronos isadistributedjobschedulerthatsupportscomplexjobtopologies.Itcanbeusedasamorefault-tolerantreplacementforCron.
q ElasticSearch isadistributedsearchengine.Mesosmakesiteasytorunandscale.
![Page 9: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/9.jpg)
CreateownFrameworkWhyweneedtowriteownframework:
q Customscheduling
q Autoscaling
q AdvancedTaskManagement
![Page 10: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/10.jpg)
WhyMesosq TaskDistribution
q Launching,Monitoring,failuredetection
q Resourceisolation
q Containersupport
qMessagingbetweenTasks
qMakeStatedistributed
![Page 11: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/11.jpg)
LanguageSupport
![Page 12: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/12.jpg)
ProtocolBuffer
Protocolbuffersare usedextensivelyformessagingandserializationinsideMesosandwhendevelopingMesosframeworks.- TasksaredescribeinProtocolbuffermessage- Ithelpsittomakelanguageindependent
![Page 13: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/13.jpg)
messageFrameworkInfo {requiredstringname=2;optionalFrameworkID id=3;optionaldoublefailover_timeout=4[default=0.0];optionalboolcheckpoint=5[default=false];optionalstringrole=6[default="*"];optionalstringhostname=7;optionalstringprincipal=8;optionalstringwebui_url =9;messageCapability{enum Type{UNKNOWN=0;REVOCABLE_RESOURCES=1;TASK_KILLING_STATE=2;}optionalTypetype=1;}repeatedCapabilitycapabilities=10;optionalLabelslabels=11;}
![Page 14: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/14.jpg)
importorg.apache.mesos.Protos.FrameworkInfo
val framework=FrameworkInfo.newBuilder.setName(“SigApp”).setUser(”rahul").setRole("*").setCheckpoint(false).setFailoverTimeout(0.0d).build()
![Page 15: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/15.jpg)
TheSchedulerThescheduleristhecomponentthatinteractsdirectlywiththeleadingMesosmaster
q Launchtasksonthereceivedoffersq Handlestatusupdatescheckingfortaskfailureandrestartq StatePersistandmanagefailovers
![Page 16: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/16.jpg)
Yourframeworkschedulershouldinheritfromthe Scheduler class
schedulershouldcreateaSchedulerDriver ,whichwillmediatecommunicationbetweenyourschedulerandtheMesosmaster.
![Page 17: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/17.jpg)
classSigScheduler() extendsScheduler {overridedef error(driver: SchedulerDriver, message:String) {}overridedef executorLost(driver:SchedulerDriver, executorId:ExecutorID,slaveId:SlaveID,status:Int){}overridedef slaveLost(driver:SchedulerDriver, slaveId:SlaveID){}overridedef disconnected(driver: SchedulerDriver) {}overridedef frameworkMessage(driver: SchedulerDriver, executorId:ExecutorID,slaveId:SlaveID,data:Array[Byte]){}overridedef statusUpdate(driver:SchedulerDriver, status:TaskStatus){println(s"received statusupdate$status")}overridedef offerRescinded(driver: SchedulerDriver, offerId:OfferID){}overridedef resourceOffers(driver: SchedulerDriver, offers:java.util.List[Offer]){//code}def submitTasks(tasks:String*)={this.synchronized {this._tasks.enqueue(tasks:_*)}}overridedef reregistered(driver: SchedulerDriver, masterInfo:MasterInfo){}
overridedef registered(driver: SchedulerDriver, frameworkId:FrameworkID,masterInfo:MasterInfo){}
![Page 18: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/18.jpg)
![Page 19: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/19.jpg)
TheExecutor
q ExecuterExecutingtasksasrequestedbythescheduler
q Keepingtheschedulerinformedofthestatusofthosetasks
q TaskManagement
![Page 20: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/20.jpg)
val exec= new Executor {
override def launchTask(driver: ExecutorDriver, task: TaskInfo): Unit ={}
override def killTask(driver: ExecutorDriver, taskId: TaskID): Unit ={}
override def shutdown(driver: ExecutorDriver, taskId: TaskID): Unit ={}
}
![Page 21: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/21.jpg)
MesosEndpoints
qHTTPendpointsavailableforagivenMesosprocess
http://master.com:5050/files/browse
http://master.com:5050/files/browse/files/debug
http://master.com:5050/files/browse
http://master.com:5050/api/v1/scheduler
http://master.com:5050/health
![Page 22: Building High Scalable Distributed Framework on Apache Mesos](https://reader034.fdocuments.us/reader034/viewer/2022042611/58f1ba6b1a28abe00d8b4591/html5/thumbnails/22.jpg)
Thank YouQ/A