CREATING A REPO

SITORY

FOR

DATA CRAW

LED FR

OM MULTIPLE

SOCIAL N

ETWORKS

A T H E N S UN I V

E R S I TY O

F EC O N O M I C

S AN D

B U S I NE S S

Konstantinos Christofilos

OK. What is the problem ?In order to get data from each service you have to speak its language (API)

What can we do about that ?We can create a repository of mixed services data and query that to produce more complex results

How are we going to do that ?

How are we going to do that ?Step 1 – Generate endpoints

How are we going to do that ?Step 2 – Load data

Step 1 – Generate endpointsImporter interface

Example (Facebook page):

The command that generates endpoints takes as input a text file with each name in a single line

Step 2 – Load dataInput

Output

Step 2 – Load data (Input)Example (Facebook page)

Step 2 – Load data (Output)Neo4j

Apache Jena

ExampleA simple example running three names: Kostas Christofilos, Iannis Kotidis, Vasilis

Spiropoulos

Example

Neo4j database size: 240 KB

Apache Jena database size: 75.7 KB

Scaling ?In order to be easily scaled, the application is designed to handle Inputs and Outputs as APIs. That approach gives the ability for a horizontal scale.

ProcessGather Data

• We took the names of world’s greatest leaders from Fortune magazinehttp://fortune.com/worlds-greatest-leaders/

• Application queried the APIs for accounts that are related to these names

• It build a list of endpoints for those names• Data was fetched from those endpoints• Entities were recognized• Data was saved into two different databases (Apache Jena,

Neo4j)

ProcessThe names that were used are the following:

Jeff Bezos, Angela Merkel, Aung San Suu Kyi, Pope Francis, Tim Cook, John Legend, Christina Figueres, Paul Ryan, Ruth Bader Ginsburg, Sheikh Hasina, Nick Saban, Huateng "Pony" Ma, Sergio Moro, Bono, Stephen Curry, Steve Kerr, Bryan Stevenson, Nikki Haley, Lin-Manuel Miranda, Marvin Ellison, Reshma Saujani, Larry Fink, Scott Kelly, Mikhail Kornienko, David Miliband, Anna Maria Chavez, Carla Hayden, Maurizio Macri, Alicia Garza, Patrisse Cullors, Opal Tometi, Chai Jing, Moncef Slaoui, John Oliver, Marc Edwards, Arthur Brooks, Rosie Batty, Kristen Griest, Shaye Haver, Denis Mukwege, Christine Lagarde, Marc Benioff, Gina Raimondo, Amina Mohammed, Domenico Lucano, Melinda Gates, Susan Desmond-Hellman, Arvind Kejriwal, Jorge Ramos, Michael Froman, Mina Guli, Ramon Mendez, Bright Simons, Justin Trudeau, Clare Rewcastle Brown, Tshering Tobgay

ProcessThe list of the previous names after it was parsed from the application produced a list of more than 11.000 endpoints in Facebook, Instagram and Twitter with the following distribution.

Names Facebook Page

Endpoints

Twitter Endpoints

Instagram Endpoints

Total Endpoints

56 437 866 9903 11206

ProcessThe parse of those endpoints in a single workstation* took about 14 days for the period 2016-06-04 to 2016-06-18.Most of the time was consumed in the entity recognition process

*Workstation specs: AMD FX 2-Core CPU, 4GB RAM, 120GB SSD, Linux OS with 5 concurrent processes of the application running

ResultsThe application run a single pass over the generated endpoints witch took about 14 days in a single workstation and the generated nodes were 126,395.

Endpoints Machines Generated

Nodes

Time Average nodes/day/machine

11206 1 126395 14 days

9028

Results

Neo4j Apache Jena60

65

70

75

80

85

90

95

100

Import (%)NER (%)

Data import time distribution (Empty database)

Results

Neo4j Apache Jena80

82

84

86

88

90

92

94

96

98

100

Import (%)NER (%)

Data import time distribution (100,000+ nodes)

Results

Jesus

Lebron

Stephen

Steph

Curry

0 5000 10000 15000 20000 25000

Persons mentions

Person Mentions

Results

Padre

GSW

Santo

Cavs

NBA

0 1000 2000 3000 4000 5000 6000 7000 8000

Organization Mentions

Organization Mentions

Results

Jordan

America

Hermosa

Venezuela

Cleveland

0 500 1000 1500 2000 2500 3000 3500 4000

Location Mentions

Location Mentions

Conclusion

• Data are generated in vast amounts every moment. • We created an approach of linking heterogeneous data in a

single repository.• Generated data can be analyzed and produce combined

results.• Patterns can be identified from that repository.

Future extensions

• New Inputs can be implemented (new APIs)• New Outputs can be implemented (new storage engines)• Name list can be saved in a database and accessed from all

nodes. Now it’s a local file• Queue for entity recognizer can be implemented and remove

the blocking code• Centralized logging for cluster monitoring. Now log goes to

STDOUT

REST APIs• REST stands for Representational State Transfer• REST ignores the details of component implementation • REST is an architecture that enables applications to

communicate without knowing the underlying technology• REST APIs promote easy reusability

REST APIS

Graph Databases• Graph database is a database type that uses graph

structures with nodes, edges and properties to represent and store data.

• Graph databases are based on graph theory and employs nodes, edges and properties

• Nodes represent entities such as people, businesses, accounts, or any other item you might want to keep track of

• Edges, also known as graphs or relationships, are the lines that connect nodes to other nodes and represent relationship between them

Graph Databases

Graph DatabasesResource Description Framework (RDF)

• RDF is a standard model for data interchange on the Web and was specified by W3C

• Web is a graph, created by nodes, edges and relations

Graph DatabasesProperty Graphs

• Property Graph databases are graph databases that contains connected entities, which can cold, any number of attributes

• Nodes can be tagged with labels representing their different roles

• Labels may also serve to attach metadata to certain nodes

Graph DatabasesProperty Graphs

Natural Language Processing (NLP)• Natural language processing (NLP) is an area of research

and application that explores how computers can be used to understand and manipulate natural language text or speech to do useful things

• NLP lie in a number of disciplines, information sciences, linguistics, mathematics, electrical and electronic engineering, artificial intelligence and robotics, psychology, etc

• Applications of NLP include a number of fields of studies, such as machine translation, natural language text processing and summarization, user interfaces, multilingual and cross language information retrieval, speech recognition, artificial intelligence and expert systems

Natural Language Processing (NLP)Named Entity Recognition (NER)

Named entity recognition (NER) is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of time etc. NER systems use linguistic grammar-based techniques as well as statistical models, i.e. machine learning.

We used Stanford Named Entity Recognizer that was created by The Stanford Natural Language Processing Group.http://nlp.stanford.edu/software/CRF-NER.shtml

Natural Language Processing (NLP)Named Entity Recognition (NER)