Big Data analysis:Introducing Cygnus and Cosmos

francisco.romerobueno@telefonica.com

Introduction

2

• Big Data– What is it?– How much data is

there?– Data growing forecast

• How to deal with Big Data– What is MapReduce– Hadoop architecture

• Cosmos Big Data– What is cosmos– Cosmos architecture– What can be done

with Cosmos

• Cluster operation– Using the RESTful API– Using the Python CLI

• How to exploit the data– I/O

• Hadoop file systemcommands

• WebHDFS/HttpFSRESTful API

– Querying• Local Hive CLI• Remote Hive client

– MapReduceapplications• Java programming• Oozie

• Cosmos place in FIWARE– General IoT platform– Real-time context data

persistence

• Roadmap

Example-driven explanation (1)

3

Wirecloud(Alice)

Orion Cygnus CosmosWirecloud

(Bob)

subscribe(“mychat”)

subscribe(“mychat”)

update(“mychat”,”hi bob”)

notify(“mychat”,“hi bob”)

notify(“mychat”,“hi bob”)

subscribe(*)

write(“mychat”,”hi bob”)

update(“mychat”,”hi alice”)

notify(“mychat”,”hi alice”)

notify(“mychat”,“hi alice”)

write(“mychat”,”hi alice”)

mychathi bob

mychathi bobhi alice

run(“mychat”)

run(“mychat”)

Example-driven explanation (2)

4

Wirecloud(Charles)

Orion Cygnus Cosmos

subscribe(“mychat”)

read(“mychat”,[“hi bob”,”hi alice”])

mychathi bobhi alice

subscribe(“other”)

Wirecloud(David)

subscribe(“other”)

update(“other”,”hi charles”)

notify(“other”,”hi charles”)

notify(“other”,“hi charles”)

write(“other”,”hi charles”)

otherhi

charles

run(“mychat”)

run(“other”)

update(“mychat”,”hi all”)

notify(“mychat”,“hi all”)

notify(“mychat”,“hi all”)

write(“mychat”,”hi all”)

mychathi bobhi alice

hi all

run(“other”)

Wirecloud

5

• Builds user interfaces based on basicbuilding blocks– Widgets (maps, text boxes, tables…)

– Operators (widgets without graphics)

• The building blocks can be wired

• Able to read from– Orion Context Broker

– Cosmos HDFS

– CKAN

Orion Context Broker

6

• Manages context information– Context information is that of interest for an application

• Implements the OMA NGSI model– Entities having each one several attributes– E.g. entity=mychat, attributes=msg

• Regarding an entity’s attribute, it only stores the lastvalue– mychat.msg=hi bob (t0)– mychat.msg=hi alice (t1)– mychat.msg=hi all (t2)

• Synchonous operation– UpdateContext and QueryContext

• Asynchronous operation– SubscribeContext and NotifyContext

Cygnus connector

7

• Builds the historic regarding Orion entities– chat.msg [(t0,hi bob),(t1,hi alice),(t2,hi all)]

• Subscribes to Orion for the entity’s attributes aimedto be persisted– Orion sends notifications when a subscribed entity’s

attribute changes

• Persistence can be made based on:– HDFS files (Hadoop, Cosmos)– MySQL tables– CKAN resources– Mongo documents (upcoming)

• Regarding HDFS, a RESTful API is used for writing(WebHDFS)

Cosmos Big Data

8

• Provides Big Data storage capabilities– Based on Hadoop Distributed File System, running on top of a

shared cluster– Cluster lifecycle: forever– Data isolation capabilites

• Provides Big Data computing capabilities– Based on Hadoop MapReduce engine, running on top of

private clusters– In the future: Spark clusters, other…– Clusters lifecycle: created, used and destroyed– Access to the shared HDFS

• Hadoop ecosystem– Hive/Shark for querying the data– Oozie for remote scheduling of tasks– WebHDFS/HttpFS for RESTful I/O API

Hands-on: creating the environment (1)

9

• Create an account in Cosmos (log by using your FIWARE credentials)http://cosmos.lab.fi-ware.org/cosmos-gui/

• Create a new fresh VM at FIWARE cloudhttp://cloud.lab.fi-ware.org

• Create the FIWARE yum repo (/etc/yum.repos.d/fiware.repo)[testbed-fi-ware]

name=Fiware Repository

baseurl=http://repositories.testbed.fi-ware.org/repo/rpm/x86_64/

gpgcheck=0

enabled=1

• Create the MongoDB yum repo (/etc/yum.repos.d/mongodb.repo)[mongodb]name=MongoDB Repositorybaseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/gpgcheck=0enabled=1

Hands-on: creating the environment (2)

10

• Install and run MongoDB (orion uses it)$ yum install mongodb-org$ mkdir –p /data/db$ nohuo mongod &

• Install and run Orion Context Broker$ yum install contextBroker$ service contextBroker start

• Install Cygnus connector$ yum install cygnusEdit the configuration files (see next slide)$ service cygnus start

Hands-on: creating the environment (3)

11

• agent_chatrooms.conf (showing relevant conf)chatrooms.sources = http-source

chatrooms.sinks = hdfs-sink

chatrooms.channels = hdfs-channel

chatrooms.sources.http-source.port = 5050

chatrooms.sources.http-source.handler.notification_target = /notify

chatrooms.sources.http-source.handler.default_service = hamburg

chatrooms.sources.http-source.handler.default_service_path = chatrooms

chatrooms.sinks.hdfs-sink.channel = hdfs-channel

chatrooms.sinks.hdfs-sink.cosmos_host = 130.206.80.46

chatrooms.sinks.hdfs-sink.cosmos_port = 14000

chatrooms.sinks.hdfs-sink.cosmos_default_username = frb

chatrooms.sinks.hdfs-sink.cosmos_default_password = xxxxxxxxxxxxx

chatrooms.sinks.hdfs-sink.hdfs_api = httpfs

chatrooms.sinks.hdfs-sink.attr_persistence = column

chatrooms.sinks.hdfs-sink.hive_host = 130.206.80.46

chatrooms.sinks.hdfs-sink.hive_port = 10000

chatrooms.sinks.hdfs-sink.krb5_auth = false

• Cygnus_instance_chatrooms.conf (showing relevant conf)CYGNUS_USER=cygnus

CONFIG_FOLDER=/usr/cygnus/conf

CONFIG_FILE=/usr/cygnus/conf/agent_chatrooms.conf

AGENT_NAME=chatrooms

LOGFILE_NAME=cygnus_chatrooms.log

ADMIN_PORT=8081

Hands-on: creating the environment (4)

12

• Subscribe to Orion on behalf of Cygnus(curl $ORION_HOST:$ORION_PORT/v1/subscribeContext -s -S --header 'X-Auth-Token: '$TOKEN'' --

header 'Content-Type: application/json' --header 'Accept: application/json' -d @-) <<EOF

{

"entities": [

{

"type": "ChatMessageLastOnly",

"isPattern": "true",

"id": ".*"

}

],

"reference": "http://$CYGNUS_HOST:$CYGNUS_PORT/$CYGNUS_TARGET",

"duration": "P5Y",

"notifyConditions": [

{

"type": "ONCHANGE",

"condValues": [

"msg"

]

}

]

}

EOF

Hands-on: using the scripts

13

• Download the scripts from herehttps://github.com/frbattid/fiware-chatrooms-demo

• Content– orion_append_context.sh

• Creates a chatroom entity with initial message

– orion_update_context.sh• Updates the message of an existent chatroom

– orion_query_context.sh• Gets the last message of an existent chatroom

– cosmos_read.sh• Reads from Cosmos the whole conversation regarding a chatroom

• Other– orion_subscribe_context.sh

• Makes a subscription to all the chatrooms

– Idm_get_token.sh• Gets an authentication token, if needed

Further reading

14

• Cosmos@FIWARE catalogue– http://catalogue.fiware.org/enablers/bigdata-

analysis-cosmos

• Cosmos presentation• http://es.slideshare.net/FranciscoRomeroBuen

o/fiwarecosmosv7tech

• Cygnus@github– https://github.com/telefonicaid/fiware-

connectors

Thanks!Thanks!