CS157B - Big Data Management

Flume with Twitter Integration

Date: 03/3/2014Professor: Thanh Tran

by Swathi Kotturu

ETL Using FlumeWhat is Flume?

Apache Flume is a distributed service for efficiently collecting, aggregating, and moving large amounts of log data.

Flume and it’s integration with Hadoop and can be used to capture streaming twitter data which can be filtered based on keywords and locations..

More About FlumeIt has a very simple architecture based on streaming data flows.Flume takes a source and processes it through a memory channel, where the data gets filtered and sinks into the HDFS.

Flume AgentsFlume can deploy any number of agents. An Agent is a container for Flume data flow. It can run any number of sources, sinks, and channels.

It must have a source, channel, and sink.

Flume SourcesSources are not Necessarily restricted to log data.

It is possible to use Flume to transport event data such as network traffic data, social-media-generated data, e-mail messages, etc…

The events can be HTTP POSTS, RPC calls, strings in stdout, etc….

After an event occurs, Flume sources write the event to a channel as a transaction.

Flume ChannelsChannels are internal passive stores with specific characteristics. This allows a source and a sink to run asynchronously.

Two Main Types of Channels

Memory Channels

- Volatile Channel that buffers events in memory only. If JVM crashes, all data is lost.

File Channels

- Persistant Channel that is stored to disk.

You can Run Multiple Agents and Servers to collect data in parallel.

Get Twitter Access

Flume in ClouderaDownload flume-sources-1.0-SNAPSHOT.jar and add it to the flume class path. http://files.cloudera.com/samples/flume-sources-1.0-SNAPSHOT.jar

In the Cloudera Manager, you can add the class path:

“Services” -> “flume1″ -> “Configuration” -> “Agent(Default)” -> “Advanced” -> “Java Configuration Options for Flume Agent”, add:

–classpath /opt/cloudera/parcels/CDH-4.3.0-1.cdh4.3.0.p0.22/lib/flume-ng/lib/flume-sources-1.0-SNAPSHOT.jar

Flume in Cloudera (cont.)

Flume in Cloudera (cont.)You also have to exclude the original file that came with Flume, pre-installed by renaming it .org. The file is search-contrib-1.0.0-jar-with-dependencies.jar and is in the /usr/lib/flume-ng/lib/ path.

mv search-contrib-1.0.0-jar-with-dependencies.jar search-contrib-1.0.0-jar-with-dependencies.jar.org

Using Hue, create user Flume and give them access to read and write in hdfs.

Flume in Cloudera (cont.)

Flume in Cloudera (cont.)From the Cloudera Manager, go to

“Services” -> “flume1″ -> “Configuration” -> “Agent(Default)” -> “Agent Name”.

Set the Agent Name to Twitter Agent

Flume in Cloudera (cont.)

Flume in Cloudera (cont.)Also set the Configuration File to the following and make sure to replace the ConsumerKey, ConsumerSecret, AccessToken, AccessTokenSecret

Also set the Configuration File to the following and make sure to replace the ConsumerKey, ConsumerSecret, AccessToken, AccessTokenSecret

TwitterAgent.sources = TwitterTwitterAgent.channels = MemChannelTwitterAgent.sinks = HDFS

TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSourceTwitterAgent.sources.Twitter.channels = MemChannelTwitterAgent.sources.Twitter.consumerKey = <consumer key>TwitterAgent.sources.Twitter.consumerSecret = <consumer secret>TwitterAgent.sources.Twitter.accessToken = <access token>TwitterAgent.sources.Twitter.accessTokenSecret = <access token secret>

Flume in Cloudera (cont.)TwitterAgent.sources.Twitter.keywords = flu, runny nose, tissue, sick, ill, cough

TwitterAgent.sinks.HDFS.channel = MemChannelTwitterAgent.sinks.HDFS.type = hdfsTwitterAgent.sinks.HDFS.hdfs.path = hdfs://localhost:8020/user/flume/tweets/TwitterAgent.sinks.HDFS.hdfs.fileType = DataStreamTwitterAgent.sinks.HDFS.hdfs.writeFormat = TextTwitterAgent.sinks.HDFS.hdfs.batchSize = 1000TwitterAgent.sinks.HDFS.hdfs.rollSize = 0TwitterAgent.sinks.HDFS.hdfs.rollCount = 10000

TwitterAgent.channels.MemChannel.type = memoryTwitterAgent.channels.MemChannel.capacity = 10000TwitterAgent.channels.MemChannel.transactionCapacity = 100

Flume in Cloudera (cont.)

Flume in Cloudera (cont.)

Restart Flume Agent

Flume in Cloudera (cont.)

Flume in Cloudera (cont.)

Example TweetWe loaded raw tweets into HDFS which are represented as chunks of JSON

Next StepsTell Hive how to read the data

You will need Hive-serdes-1.0-SNAPSHOT.jar

http://files.cloudera.com/samples/hive-serdes-1.0-SNAPSHOT.jar

As Hive is setup to read delimited row format but in this case needs to read json.

Flume ResourcesLearn More

https://dev.twitter.com/docs/streaming-apis/parameters

https://cwiki.apache.org/confluence/display/FLUME/Home

http://blog.cloudera.com/blog/2012/09/analyzing-twitter-data-with-hadoop/

Thank you!Q/A