Apache Ka)a - files.ondemand.cloudera.com · § Web site acYvity tracking – Web applicaIon sends...

ApacheKa)aChapter1

201601

©Copyright2010-2016Cloudera.Allrightsreserved.Nottobereproducedorsharedwithoutpriorwri@enconsentfromCloudera. 01-2

Inthischapteryouwilllearn

§ WhatKa8aisandwhatadvantagesitoffers

§ Aboutthehigh-levelarchitectureofKa8a

§ WhatseveralusecasesforKa8aare

§ Howtocreatetopics,publishmessages,andreadmessagesfromthecommandlineandinJavacode

ApacheKa)a


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases

§ Messages,Topics,andParIIons

§  ProducersandConsumers

§ MessageOrderingGuarantees

§  UsingtheJavaAPI

§  EssenIalPoints

§  Hands-OnExercise:UsingKa)afromtheCommandLine


§ ApacheKa8aisadistributedcommitlogservice– Widelyusedfordataingest– Offersscalability,performance,reliability,andflexibility– Conceptuallysimilartoapublish-subscribemessagingsystem

§ OriginallycreatedatLinkedIn,butnowanopensourceApacheproject– DonatedtotheApacheSoXwareFoundaIonin2012– GraduatedfromtheApacheIncubatorin2013– IncludedaspartofClouderaLabsin2014– SupportedbyClouderaforproducIonusewithCDHin2015

WhatisApacheKa)a?

Apache Kafka


§ Scalable– Ka)aisadistributedsystemthatsupportsmulIplenodes

§ Fault-tolerant– Dataispersistedtodiskandreplicatedthroughoutthecluster

§ Highthroughput– Eachbrokercanprocesshundredsofthousandsofmessagespersecond

§ Lowlatency– DataisdeliveredinafracIonofasecond

§ Flexible– DecouplestheproducIonofdatafromitsconsumpIon

CharacterisIcsofKa)a

*

*Usingmodesthardware,withmessagesofatypicalsize


§ Messagesrepresentarbitraryuser-definedcontent– Forexample,applicaIoneventsorsensorreadings

§ AnoderunningtheKa8aserviceiscalledabroker– AproducIonclustertypicallyhasmanyKa)abrokers– Ka)aalsodependsontheZooKeeperserviceforcoordinaIon

§ Producerspushmessagestoabroker– Theproducerassignsatopic,orcategory,toeachmessage

§ ConsumerspullmessagesfromaKa8abroker– Theyreadonlymessagesinrelevanttopics

High-LevelArchitecture:Terminology


High-LevelArchitecture:Example

Producer #1 Producer #2

Consumer #1 Consumer #2

Kafka Cluster

Broker

Broker

Broker

Broker

Broker

Broker

login_failure

Producer #3

call_placed

login_failure

login_failure call_placed


ChapterTopics

ApacheKa8a

§ Overview

§ UseCases








§ Ka8aisusedforavarietyofusecases,suchas– LogaggregaIon– Messaging– WebsiteacIvitytracking– OperaIonalmetrics– Streamprocessing– Eventsourcing

§ AsubsetofthesecouldalsobedonewithFlume– Forexample,aggregaIngWebserverlogdataintoHDFS

§ Ka8aoUenbecomesabeVerchoiceasusecasecomplexitygrows

WhyKa)a?


§ Distributedmessagebus/centraldatapipeline– EnableshighlyscalableEAI,SOA,CEPandmicroservicearchitectures– DecouplesserviceswithastandardizedmessageabstracIon– SupportsmulIplemessageclientlanguageswithhighthroughput

§ LogaggregaYon– Ka)acancollectlogsfrommulIpleservices– LogscanbemadeavailabletomulIpleconsumers,suchasHadoopandApacheSolr

CommonKa)aUseCases(1)

EAI: EnterpriseApplicaIonIntegraIonSOA:Service-OrientedArchitectureCEP:ComplexEventProcessing


§ WebsiteacYvitytracking– WebapplicaIonsendseventssuchaspageviewsandsearchestoKa)a– Eventsbecomeavailableforreal-Imeprocessing,dashboards,andofflineanalyIcsinHadoop

§ AlerYngandreporYngonoperaYonalmetrics– Ka)aproducersandconsumersoccasionallypublishtheirmessagecountstoaspecialKa)atopic– AservicecomparescountsandsendsanalertupondetecIngdataloss

§ Streamprocessing– AframeworksuchasSparkStreamingreadsdatafromatopic,processesit,andwritesprocesseddatatoanewtopicwhereitbecomesavailableforusersandapplicaIons– Ka)a’sstrongdurabilityhelpstofacilitatethisusecase

CommonKa)aUseCases(2)


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases

§ Messages,Topics,andParYYons







§ MessagesinKa8aarevariable-sizebytearrays– AllowsforserializaIonofdatainanyformatyourapplicaIonrequires– Commonformatsincludestrings,JSON,andAvro

§ Thereisnoexplicitlimitonmessagesize– OpImalperformanceusuallyoccurswithmessagesofafewKBinsize– Werecommendthatyoudonotexceed1MBpermessage

§ Ka8aretainsallmessagesforadefinedYmeperiod– Thisperiodcanbesetonglobalorper-topicbasis– Messageswillberetainedregardlessofwhethertheywereread– TheyarediscardedautomaIcallyaXertheretenIonperiodisexceeded

Messages


§ Thereisnoexplicitlimitonthenumberoftopics– Ka)aworksbe@erwithafewlargetopicsthanmanysmallones

§ Atopiccanbecreatedexplicitlyorsimplybypublishingtothetopic– Controlledbytheauto.create.topics.enableproperty– Werecommendthattopicsbecreatedexplicitly

Topics


§ EachtopicisdividedintosomenumberofparYYons*– ParIIoningimprovesscalabilityandthroughput

§ AtopicparYYonisanorderedandimmutablesequenceofmessages– NewmessagesareappendedtotheparIIonastheyarereceived– EachmessageisassignedauniquesequenIalIDknownasanoffset

TopicParIIoning

*NotethatthisisunrelatedtoparIIoninginHDFS,MapReduce,orSpark

Older messages Newer messages

Partition 1

Partition 2

0 1 2 3 4 5 6 7 8 9

Time

Partition 0

0 1 2 3 4 5 6 7

0 1 2 3 4 5 6 7 8

Producer A

Producer B


§ EachparYYoncanbereplicatedacrossaconfigurablenumberofbrokers*– Doingsoisrecommended,asitprovidesfaulttolerance

§ EachbrokeractsasaleaderforsomeparYYonsandafollowerforothers– Followerspassivelyreplicatetheleader– Iftheleaderfails,afollowerwillautomaIcallybecomethenewleader

ReplicaIon

*NotethatthisisunrelatedtoHDFSreplicaIon

Broker B

Partition 0

Partition 2

Broker C

Partition 0

Partition 1

Partition 0

Broker A

Partition 1

Partition 2

Partition 1

Partition 2

Leader

Follower

Legend


§ InproducYon,youwilllikelystartKa8aviaClouderaManager– Inthisclass,wemuststartitmanuallyontheVM

§ SinceKa8adependsonZooKeeper,wemuststartthatservicefirst

§ WecanthenstarttheKa8aservice

StarIngtheKa)aBroker

$ sudo service zookeeper-server start

$ sudo service kafka-server start


§ Ka8aincludesaconvenientsetofcommandlinetools– ThesearehelpfulforexploringandexperimentaIon

§ Thekafka-topicscommandoffersasimplewaytocreateKa8atopics– Providethetopicnameofyourchoice,suchasdevice_status – YoumustalsospecifytheZooKeeperconnecIonstringforyourcluster

CreaIngTopicsfromtheCommandLine

$ kafka-topics --create \ --zookeeper localhost:2181 \ --replication-factor 1 \ --partitions 1 \ --topic device_status


§ Usethe--listparametertolistalltopics

DisplayingTopicsfromtheCommandLine

$ kafka-topics --list \ --zookeeper localhost:2181


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases








§ ProducerspublishmessagestoKa8atopics– Theycommunicatewithabroker,notaconsumer

ProducerRecap

Producer #1 Producer #2

Kafka ClusterBroker

Broker

Broker

Broker

Broker

Broker

login_failure

Producer #3

call_placed

login_failure


§ AproducerisresponsibleforselecYngparYYonsformessagesitpublishes– ThisisprimarilydonetobalancetheloadacrossallparIIons– TheproducerwritesmessagestoaparIIoninorder– ApluggablePartitionerclassselectstheparIIonforeachmessage

SelecIngtheParIIon


Partition 1

Partition 2

0 1 2 3 4 5 6 7 8 9

Time

Partition 0

0 1 2 3 4 5 6 7

0 1 2 3 4 5 6 7 8

Producer A

Producer B


§ ProducerscancollectmulYplemessagestowritetoaparYYon– Thisreducesthenumberofrequestsmadetobrokers– SuchrequestssenttobrokerscontainonebatchperparIIon

§ BatchingiscontrolledthroughproperYessetfortheproducer– Thedefaultistosendmessagesimmediately– Batchsizeisconfigurable,asisthemaxImetowaitbeforesending

Aside:MessageBatchesIncreaseThroughputandLatency


Partition 1

Partition 2

0 1 2 3 4 5 6 7 8

Time

Partition 0

0 1 2 3 4 5 6 8

0 1 2 3 4 5 6 7 Producer B

9

Producer A7

8

9 10

9 10


§ Theproducerisconfiguredwithalistofoneormorebrokers– ItasksthefirstavailablebrokerfortheleaderofthedesiredparIIon

§ Theproducerthensendsthemessagetotheleader– Theleaderwritesthemessagetoitslocallog– Eachfollowerthenwritesthemessagetoitsownlog– AXeracknowledgementsfromfollowers,themessageiscommi@ed

MessagesareReplicated

Broker B

Partition 0

Partition 2

Broker C

Partition 0

Partition 1

Broker A

Partition 1

Partition 2

Partition 1

Partition 2

1 Partition 02

3Producer


§ Youcancreateaproducerusingthekafka-console-producertool

§ Specifyoneormorebrokersinthe--broker-listopYon– Eachbrokerconsistsofahostname,acolon,andaportnumber– IfspecifyingmulIplebrokers,separatethemwithcommas– Inourcasethereisonebroker:localhost:9092

§ Youmustalsoprovidethenameofthetopic– Wewillpublishmessagestothetopicnameddevice_status

CreaIngaProducerfromtheCommandLine(1)

$ kafka-console-producer \ --broker-list localhost:9092 \ --topic device_status


§ YoumayseeafewlogmessagesintheterminalaUertheproducerstarts

§ Itwillthenacceptinputintheterminalwindow– Eachlineyoutypewillbeamessagesenttothetopic

§ UnYlyouhaveconfiguredaconsumerforthistopic,you’llseenootheroutputfromKa8a

CreaIngaProducerfromtheCommandLine(2)


§ AconsumerreadsmessagesthatwerepublishedtoKa8atopics– Theycommunicatewithabroker,notaproducer

§ ConsumeracYonsdonotaffectotherconsumers– Forexample,issuingtheKa)acommandlinetoolto"tail"thecontentsofatopicdoesnotchangewhatisconsumedbyotherconsumers

§ Theycancomeandgowithoutimpactontheclusterorotherconsumers

ConsumerRecap

Consumer #1 Consumer #2

login_failure call_placed

Kafka ClusterBroker

Broker

Broker

Broker

Broker

Broker


§ Youcancreateaconsumerwiththekafka-console-consumertool

§ ThisrequirestheZooKeeperconnecYonstringforyourcluster– UnlikecreaIngaproducer,whichinsteadrequiredalistofbrokers

§ Thecommandalsorequiresatopicname– Inourcase,wewillusedevice_status

§ Youcanuse--from-beginningtoreadallavailablemessages– Otherwise,itwouldreadonlynewmessages

CreaIngaConsumerfromtheCommandLine

$ kafka-console-consumer \ --zookeeper localhost:2181 \ --topic device_status \ --from-beginning


§ UsingUNIXpipesorredirecYon,youcanreadinputfromfiles– Thedatacanthenbesenttoatopicusingthecommandlineproducer

§ Thisexampleshowshowtoreadinputfromafilenamedalerts.txt – Eachlineinthisfilebecomesaseparatemessageinourtopic

§ ThistechniquecanbeaneasywaytointegratewithexisYngprograms

WriIngFileContentstoTopicsviatheCommandLine

$ cat alerts.txt | kafka-console-producer \ --broker-list localhost:9092 \ --topic device_status


§ MessaginghastwotradiYonalmodels– Queuing– Publish-subscribe

§ Withqueuing,apoolofconsumersmayreadfromaserverandeachmessagegoestooneofthem

§ Inpublish-subscribe,themessageisbroadcasttoallconsumers

§ AKa8aconsumergroupisaconsumerabstracYonthatgeneralizesbothofthesemodels

HowdoesKa)adifferfromtradiIonalmessagemodels?


§ Eachmessagepublishedtoatopicisdeliveredtooneconsumerinstancewithineachsubscribingconsumergroup

§ Consumerinstancescanbeinseparateprocessesoronseparatemachines

§ ThediagrambelowdepictsaKa8aclusterwithtwobroker(servers)– ThebrokersarehosIngfourparIIons,P0-P3– ConsumergroupAhastwoconsumerinstancesandgroupBhasfour

Ka)aConsumerGroupOperaIon

Ka8aCluster P0 P3 P1 P2

ConsumerGroupA

C1 C2

ConsumerGroupB

C3 C4 C5 C6

Broker1 Broker2


§ Ka8afuncYonslikeatradiYonalqueuewhen– Allconsumerinstancesbelongtothesameconsumergroup– Inthiscase,agivenmessageisreceivedbyoneconsumer

§ Ka8afuncYonsliketradiYonalpublish-subscribewhen– Eachconsumerinstancebelongstoadifferentconsumergroup– Inthiscase,allmessagesarebroadcasttoallconsumers

Ka)aConsumerGroupConfiguraIons


§ Inbetweenthetwoextremesofqueuingorpublish-subscribeliesabalancedsoluYon– Atopiccanhaveoneconsumergroupforeach“logicalsubscriber”

§ Inthisapproach,eachconsumergroupiscomposedofmanyconsumerinstances– Thisprovidesscalabilityandfaulttolerance– Amountstopublish-subscribesemanIcswherethesubscriberisaclusterofconsumersinsteadofasingleprocess

Using“LogicalSubscribers”


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases








§ AtradiYonalqueueretainsmessagesinorderontheserver– Theserverhandsoutmessagestoconsumersintheordertheyarestored

§ Insomemessagesystems,messagesdeliveredtoconsumersasynchronouslymayarriveoutoforderatdifferentconsumers– MessageorderiseffecIvelylostinthepresenceofparallelconsumpIon

§ Theworkaroundistoallowonlyoneprocesstoconsumefromaqueue– Thisisthe"exclusiveconsumer"approach– Thereisnoparallelism

TradiIonalMessageOrdering


§ ParYYonswithinKa8atopicsmakeitpossibletoprovideaconsumergroupwith– Messageorderingguarantees– Loadbalancing

§ ParYYonsareassignedtoconsumersinaconsumergroup– EachparIIonisconsumedbyexactlyoneconsumerinthegroup– TheconsumerofaparIIonistheonlyreaderofthatparIIonandconsumesthedatainorder

§ ThenumberofconsumerscannotexceedthenumberofparYYons

Ka)aOrdering


§ Ka8aonlyprovidesatotalorderovermessageswithinaparYYon,notbetweendifferentparYYonsinatopic

§ Per-parYYonorderingcombinedwiththeabilitytoparYYondatabykeyissufficientformostapplicaYons

§ SomeapplicaYonsrequiretotalorderingforagiventopic– AccomplishthisbycreaIngjustoneparIIonforthetopic– Notethatthismeansonlyoneconsumerprocessisallowed

Ka)aOrderingTip


§ MessagessentbyaproducertoaparYculartopicparYYonwillbeappendedintheordertheyaresent– Forexample,ifmessageM1issentbythesameproducerasmessageM2,andM1issentfirst,then– M1willhavealoweroffsetthanM2– M1willappearearlierinthelogthanM2

§ Aconsumerseesmessagesintheorderinwhichtheyarestoredinthelog

§ ForatopicwithreplicaYonfactorN,uptoN-1serverfailurescanoccurwithoutlosinganymessagescommiVedtothelog

Ka)aGuarantees


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases




§ UsingtheJavaAPI




§ Ka8a’sJavaAPIallowsyoutoeasilycreateproducersandconsumers– Yourcodecansendmessagestoatopicusingaproducer– Yourcodecanalsoreadmessagessenttoatopicusingaconsumer

§ Thenextthreeslidesshowsamplecodeforasimpleproducerthatsendsamessagetoatopic

Ka)aJavaAPI:Producer


SimpleProducer(1):ImportStatementsandClassDeclaraIon

package com.loudacre.example; import java.util.Properties; import java.util.concurrent.Future; import org.apache.kafka.clients.producer.KafkaProducer; import org.apache.kafka.clients.producer.Producer; import org.apache.kafka.clients.producer.ProducerConfig; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.RecordMetadata; import org.apache.kafka.common.serialization.StringSerializer; public class ProducerExample { public static void main(String[] args) {

Note:filecon0nuesonnextslide


SimpleProducer(2):ProducerProperIesConfiguraIon

Properties props = new Properties();

// This is a comma-delimited list of brokers to contact props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");

// This specifies that the write will only be committed // after all brokers with replicas have acknowledged it props.put(ProducerConfig.ACKS_CONFIG, "all");

// # of bytes to collect in message batch before sending props.put(ProducerConfig.BATCH_SIZE_CONFIG, 16384); // Specifies classes used for message serialization props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());



SimpleProducer(3):MessageCreaIonandPublicaIon

// Create a Producer using our configuration properties Producer<String, String> producer =

new KafkaProducer<String, String>(props); // Specify the topic and value for the message String topic = "app_events"; String value = "CART_ADD,alice,0872584";

// Create and send the message ProducerRecord<String, String> message = new ProducerRecord<String, String>(topic, value); producer.send(message); // Close the producer once we no longer need it producer.close(); } }


§ Thenextfewslidesprovidesamplecodeforasimpleconsumer– Thisconsumerreadsmessagespostedtotheselectedtopic

Ka)aJavaAPI:Consumer


High-LevelConsumer(1):ImportsandClassDeclaraIon

package com.loudacre.example; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Properties; import kafka.consumer.Consumer; import kafka.consumer.ConsumerConfig; import kafka.consumer.ConsumerIterator; import kafka.consumer.KafkaStream; import kafka.javaapi.consumer.ConsumerConnector; import kafka.serializer.Decoder; import kafka.serializer.StringDecoder; public class ConsumerExample { public static void main(String[] args) {



High-LevelConsumer(2):PropertyConfiguraIon

// Define required properties and configure the consumer Properties props = new Properties(); props.put("zookeeper.connect", "localhost:2181"); props.put("group.id", "example"); ConsumerConfig cfg = new ConsumerConfig(props); ConsumerConnector consumer = Consumer.createJavaConsumerConnector(cfg); // Prepare to subscribe to app_events with one thread String topic = "app_events"; Map<String, Integer> tpx=new HashMap<String, Integer>(); tpx.put(topic, Integer.valueOf(1)); // Set up the message decoder and subscribe to the topic Decoder<String> dec = new StringDecoder(null); Map<String, List<KafkaStream<String, String>>> sm = consumer.createMessageStreams(tpx, dec, dec);



High-LevelConsumer(3):MessageProcessing

// Get our topic's stream and iterate over its messages for (KafkaStream<String, String> str : sm.get(topic)) { ConsumerIterator<String, String> i = str.iterator();

// Process each incoming message while (i.hasNext()) { String message = i.next().message(); System.out.println("Message was: " + message); } } } }


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases





§  EssenYalPoints



§ NodesrunningtheKa8aservicearecalledbrokers

§ Producerspublishmessagestocategoriescalledtopics

§ Messagesinatopicarereadbyconsumers– MulIpleconsumerinstancescanbelongtoaconsumergroup– Ka)aretainsmessagesforadefined(butconfigurable)amountofIme– Consumersmaintainanoffsettotrackwhichmessagestheybeenread

§ TopicsaredividedintoparYYonsforperformanceandscalability– TheseparIIonsarereplicatedforfaulttolerance

EssenIalPoints


ThefollowingoffermoreinformaYonontopicsdiscussedinthischapter

§ TheApacheKa8aWebsite– http://kafka.apache.org/

§ Real-TimeFraudDetec:onArchitecture– http://tiny.cloudera.com/kmc01a

§ Ka8aReferenceArchitecture– http://tiny.cloudera.com/kmc01b

§ TheLog:WhatEverySoDwareEngineerShouldKnow…– http://tiny.cloudera.com/kmc01c

Bibliography


ChapterTopics

ApacheKa8a

§ Overview

§  UseCases






§  Hands-OnExercise:UsingKa8afromtheCommandLine


§ Inthisexercise,youwilluseKa8a’scommandlineuYliYestocreateanewtopic,publishmessagestothetopicwithaproducer,andreadmessagesfromthetopicwithaconsumer– PleaserefertotheHands-OnExerciseManualforinstrucIons

Hands-OnExercise:UsingKa)afromtheCommandLine

Integra(ngFlumeandKa0aChapter2

201601

©Copyright2010-2016Cloudera.Allrightsreserved.NottobereproducedorsharedwithoutpriorwriEenconsentfromCloudera. 02-2

Inthischapteryouwilllearn

§ WhattoconsiderwhenchoosingbetweenFlumeandKa<aforausecase

§ HowFlumeandKa<acanworktogether

§ HowtoconfigureaFlumesourcethatreadsfromaKa<atopic

§ HowtoconfigureaFlumesinkthatpublishestoaKa<atopic

Integra(ngFlumeandKa0a


ChapterTopics

IntegraBngFlumeandKa<a

§ Overview

§  UseCases

§  Configura(on

§  TipsforDeployment

§  Essen(alPoints

§  Hands-OnExercise:UsingKa0aasaFlumeSink

§  Hands-OnExercise:UsingKa0aasaFlumeSource


§ BothFlumeandKa<aarewidelyusedfordataingest– Althoughthesetoolsdiffer,theirfunc(onalityhassomeoverlap– SomeusecasescouldbeimplementedwitheitherFlumeorKa0a

§ HowdoyoudeterminewhichisabeGerchoiceforyourusecase?

ShouldIUseKa0aorFlume?


§ FlumeisefficientatmovingdatafromasinglesourceintoHadoop– ItofferssinksthatwritetoHDFS,anHBasetable,oraSolrindex– Easilyconfiguredtosupportcommonscenarios,withoutwri(ngcode– Canalsoprocessandtransformdataduringtheingestprocess

Characteris(csofFlume


§ Ka<aisapublish-subscribemessagingsystem– Itoffersmoreflexibilityforconnec(ngmul(plesystems– ProvidesbeEerdurabilityandfaulttolerancethanFlume– Typicallyrequireswri(ngcodeforproducersand/orconsumers– NodirectsupportforprocessingmessagesorloadingintoHadoop

Characteris(csofKa0a

Apache Kafka


§ BothsystemshavestrengthsandlimitaBons

§ Youdon’tnecessarilyhavetochoosebetweenthem– Itispossibletousebothwhenimplemen(ngyourusecase

§ Fla<aistheinformalnameforFlume-Ka<aintegraBon– ItusesaFlumeagenttoreadfromorwritemessagestoKa0a

§ ItisimplementedasaKa<asourceandsinkforFlume– ThesecomponentsshipwithFlume,star(ngwithCDH5.2.0– AKa0achannelalsonowshipswithFlume,star(ngwithCDH5.3.0

Fla0a=Flume+Ka0a


ChapterTopics


§ Overview

§ UseCases

§  Configura(on






§ ByusingtheKa<asink,Flumecanpublishmessagestoatopic

§ Inthisexample,anapplicaBonusesFlumetopublishapplicaBonevents– Theapplica(onsendsdatatotheFlumesourcewheneventsoccur– Theeventdataisbufferedinthechannelun(litistakenbythesink– SinceweuseaKa0asink,theeventsarepublishedtoaspecifiedtopic– AnyKa0aconsumercanthenreadmessagesforapplica(onevents

UsingFlumeasaKa0aProducer

Application

Source (netcat)

Channel (Memory)

Sink (Kafka)

Flume Agent Kafka Cluster

Broker

Broker

Broker

Consumer


§ ByusingtheKa<asource,Flumecanreadmessagesfromatopic– Itcanthenwritethemtoyourdes(na(onofchoiceusingaFlumesink

§ Inthisexample,theProducersendsmessagestoKa<abrokers– TheFlumeagentusesaKa0asource,whichactsasaconsumer– TheKa0asourcereadsmessagesinaspecifiedtopic– Themessagedataisbufferedinthechannelun(litistakenbythesink– ThesinkthenwritesthedataintoHDFS

UsingFlumeasaKa0aConsumer

Kafka Cluster

Broker

Broker

Broker

Producer

Source (Kafka)

Channel (Memory)

Sink (HDFS)

Flume Agent Hadoop Cluster


ChapterTopics


§ Overview

§  UseCases

§  ConfiguraBon






§ ThetablebelowdescribesseveralproperBesoftheKa<asink

Configura(on:UsingFlumeasaKa0aProducer(1)

Application

Source (netcat)

Channel (Memory)

Sink (Kafka)

Flume Agent Kafka Cluster

Broker

Broker

Broker

Consumer

Name DescripBon

type Mustbesettoorg.apache.flume.sink.kafka.KafkaSink

brokerList Comma-separatedlistofbrokers(formathost:port)tocontact

topic ThetopicinKa0atowhichthemessageswillbepublished.

batchSize Howmanymessagestoprocessinonebatch


§ ThisistheFlumeconfiguraBonfortheexampleonthepreviousslide


# Define names for the source, channel, and sink agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 # Define the properties of our source, which receives event data agent1.sources.source1.type = netcat agent1.sources.source1.bind = localhost agent1.sources.source1.port = 44444 agent1.sources.source1.channels = channel1 # Define the properties of our channel agent1.channels.channel1.type = memory agent1.channels.channel1.capacity = 10000 agent1.channels.channel1.transactionCapacity = 1000

Note:filecon.nuesonnextslide


§ TheremainingporBonoftheconfiguraBonfilesetsuptheKa<asink


# Define our Kafka sink, which publishes to the app_event topic agent1.sinks.sink1.type = org.apache.flume.sink.kafka.KafkaSink agent1.sinks.sink1.topic = app_events agent1.sinks.sink1.brokerList = localhost:9092 agent1.sinks.sink1.batchSize = 20 agent1.sinks.sink1.channel = channel1


§ ThetablebelowdescribesseveralproperBesoftheKa<asource

Configura(on:UsingFlumeasaKa0aConsumer(1)

Name DescripBon

type org.apache.flume.source.kafka.KafkaSource

zookeeperConnect ZooKeeperconnec(onstring(e.g.,localhost:2181)

groupId UniqueIDtousefortheconsumergroup(default:flume)

topic NameofKa0atopicfromwhichmessageswillberead

Kafka Cluster

Broker

Broker

Broker

Producer

Source (Kafka)

Channel (Memory)

Sink (HDFS)

Flume Agent Hadoop Cluster



§ ThisistheFlumeconfiguraBonfortheexampleonthepreviousslide– ItdefinesasourceforreadingmessagesfromaKa0atopic

# Define names for the source, channel, and sink agent1.sources = source1 agent1.channels = channel1 agent1.sinks = sink1 # Define a Kafka source that reads from the calls_placed topic # The "type" property line wraps around due to its long value agent1.sources.source1.type = org.apache.flume.source.kafka.KafkaSource agent1.sources.source1.zookeeperConnect = localhost:2181 agent1.sources.source1.topic = calls_placed agent1.sources.source1.groupId = flume agent1.sources.source1.channels = channel1

Note:filecon.nuesonnextslide



§ ThisistheFlumeconfiguraBonfortheexampleonthepreviousslide

# Define the properties of our channel agent1.channels.channel1.type = memory agent1.channels.channel1.capacity = 10000 agent1.channels.channel1.transactionCapacity = 1000 # Define the sink that writes call data to HDFS agent1.sinks.sink1.type=hdfs agent1.sinks.sink1.hdfs.path = /user/training/calls_placed agent1.sinks.sink1.hdfs.fileType = DataStream agent1.sinks.sink1.hdfs.fileSuffix = .csv agent1.sinks.sink1.channel = channel1


ChapterTopics


§ Overview

§  UseCases

§  Configura(on






§ Ka<ahasasignificantlysmallerproducerandconsumerecosystem– UseKa0aifyou’repreparedtoimplementproducersandconsumers

§ UseFlumeifitssourcesandsinksmatchyourrequirements– Flumehasmanybuilt-insourcesandsinksfromwhichtochoose– Usingthemrequiresonlyconfigura(on,notwri(ngcode

UseKa0aforCustomProducersandConsumers


§ Flumecanprocessdatain-flightusinginterceptors– Thesecanbeveryusefulforfilteringortransformingdata

§ Ka<arequiresanexternalstreamprocessingsystem– SparkStreamingisapopularchoice

UseFlumeforFilteringandTransformingData


§ BothKa<aandFlumearereliablesystemsthatcanguaranteenodataloss

§ However,Flumedoesnotreplicateevents– Asaresult,ifanodewiththeFlumeagentcrashes,youwillloseaccesstotheeventsinthechannelun(lyourecoverthedisks– Thisistrueevenwhenusingthefilechannel

§ UseKa<aifyouneedaningestpipelinewithveryhighavailability

UseKa0aforHighAvailability


§ YoucanconfigureaFlumeagenttousemulBplechannels– Eachchannelsendsdatatoanassociatedsink

§ ThiscanbeusedtowritedatatoHDFSandKa<asimultaneously

AFlumeAgentCanWritetoMul(pleSinks

Flume Agent

Channel Kafka Sink

Source

Channel HDFS Sink

Write data to HDFS

Publish toKafka topic

Application

Broker

Broker

Broker


ChapterTopics


§ Overview

§  UseCases

§  Configura(on


§  EssenBalPoints




§ FlumeandKa<aaredisBnctsystemswithdifferentdesigns– Youmustweighttheadvantagesanddisadvantagesofeachwhenselec(ngthebesttoolforyourusecase

§ FlumeandKa<acanbecombinedwithFla<a– ThisistheinformalnameforFlumecomponentsforKa0aintegra(on– YoucanreadmessagesfromatopicusingaKa0asource– YoucanpublishmessagestoatopicusingaKa0asink

Essen(alPoints


ThefollowingoffermoreinformaBonontopicsdiscussedinthischapter

§ Fla<a:ApacheFlumeMeetsApacheKa<aforEventProcessing– http://tiny.cloudera.com/kmc02a

§ DesigningFraud-DetecBonArchitectureThatWorksLikeYourBrainDoes– http://tiny.cloudera.com/kmc02b

Bibliography


ChapterTopics


§ Overview

§  UseCases

§  Configura(on



§  Hands-OnExercise:UsingKa<aasaFlumeSink



§ Inthisexercise,youwilluseFlume’sKa<asinktowritedatathatwasreceivedbyaFlumeagentintoaKa<asink– PleaserefertotheHands-OnExerciseManualforinstruc(ons

Hands-OnExercise:UsingKa0aasaFlumeSink(Fla0a)


ChapterTopics


§ Overview

§  UseCases

§  Configura(on




§  Hands-OnExercise:UsingKa<aasaFlumeSource


§ Inthisexercise,youwilluseFlume’sKa<asourcetoreaddatapublishedtoaKa<atopicandwriteittoadirectoryinHDFS– PleaserefertotheHands-OnExerciseManualforinstruc(ons

Hands-OnExercise:UsingKa0aasaFlumeSource(Fla0a)

Apache Ka)a - files.ondemand.cloudera.com · § Web site acYvity tracking – Web applicaIon sends...

Documents

Transcript of Apache Ka)a - files.ondemand.cloudera.com · § Web site acYvity tracking – Web applicaIon sends...