Luke Young Rhee: Hashtag-Hashtag Presentation

Click here to load reader

  • date post

    08-Feb-2017
  • Category

    Technology

  • view

    31
  • download

    0

Embed Size (px)

Transcript of Luke Young Rhee: Hashtag-Hashtag Presentation

ashtag#

ashtag#

Luke Young Rhee

ashtag#

The trending topics of trending topicsashtag#

Luke Young Rhee

Impact

Discovery

ashtag#

The trending topics of trending topicsashtag#

htht.techDEMO

YouTube

The Pipeline

The Pipeline

{"text": "RT @Pozzzzzzz"id": 5469180, time: 2016:09:30T...,"entities": {"user_mentions":... "hashtags": [{text: yum, indices: [32, 35] },{text: beer, indices: [32, 36] }] }, Tweet

The Pipeline

kafka-connect-twitter

Kafka ConnectData Formating

The Pipeline

Kafka Streams

ProcessFilter

The Pipeline

Druid

Low latencyIngestionAnalyticsScalable

The Pipeline

Query: Tabletimehashtags...[beer, yum]...[beer, Lagunitas]...[cats, cute, notCrazy]

The Pipeline

Query: Filtertimehashtags...[beer, yum]...[beer, Lagunitas]...[cats, cute, notCrazy]

...beer...yum...beer...Lagunitas

hashtags = beer

The Pipeline

Query: Counttimehashtagscount...beer2...yum1...Lagunitas1

The Pipeline

Query: Count

+TopNtimehashtagscount...beer2...yum1...Lagunitas1

The Pipeline

The Pipeline

Kafka ConnectorKafka Streams

Kafka Indexing Service

pydruid

ChallengesKafka Streams / Serdes+Druid

Druid Cluster

ChallengesKafka Streams / Serdes+DruidIngest: 1% Twitter ~ 1k - 2k /min

Query: ~ 92.6k rows

Success!

Thanks!Luke Young Rhee

University of California, IrvineMS Mathematics

Nintex, IrvineTest Analyst

Enjoy being near the ocean and getting lost in new cities