GTRI Proprietary / Limited Distribution

Quick Start Guide

Analytics Framework

Ashley Scripka Beavers

GTRI Proprietary / Limited Distribution

Architecture

File System

DataLoader API Analytics API Visualization API

MongoDB

Resource Management Layer

Python Algorithms

SmallK

Sentiment Stats

Easy Vis

Special Vis

Custom Dashboard (DiamondEye, MINT)

Ingest Modules

CSV/XLSX

MongoDB

Text Documents

Filters

DateTime

GTRI LEAN

Text Processing

GTRI Proprietary / Limited Distribution

Documentation

• All interfaces: docs/• INGEST_INTERFACE.py

• FILTER_INTERFACE.py

• ANALYTIC_INTERFACE.py

• VISUALIZATION_INTERFACE.py

• Utility Command-Line application:• configure.py

• Usage: python configure.py

--api {ingest, filters, analytics, visualization}

--filename [filename of filter without .py extension]

--mode {add, remove}

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

Interface documentation and templates

Sample data optionally used during installation

Apache web server configuration file for default installation

Command-line application for adding/removing modules fro the APIs

DataLoader API

Analytics API

Visualization API

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

Location for storing source files and matrix files

Python files for the associated API

WSGI configuration file

JavaScript files – used for DetectDatatype exploration of MongoDBjs

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

js

DataLoader

DataLoaderAPIv01.py

Python Module for DataLoader functionality

Flask file for controlling API endpoints

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

js

DataLoader

DataLoaderAPIv01.py

filters

ingest

Location of all Filter modules;when adding a new Filter, the file goes here

Location of all Ingest modules;when adding a new Ingest, the file goes here

Various other helper files

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

Location for storing results files

Python files for the associated API

WSGI configuration file

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

Analytics

AnalyticsAPIv01.py

Python Module for Analytics functionality

Flask file for controlling API endpoints

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

Analytics

AnalyticsAPIv01.py

algorithms Location of all Analytics modules;when adding a new analytic, the file goes here

Various other helper files

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

python

wsgi

Python files for the associated API

WSGI configuration file

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

python

wsgi

Visualization

VisualizationAPIv01.py

Python Module for Visualization functionality

Flask file for controlling API endpoints

GTRI Proprietary / Limited Distribution

Repository Structure

analytics-framework dataloader

conf

analytics

visualization

minimal

docs

configure.py

python

wsgi

Visualization

VisualizationAPIv01.py

vis Location of all Visualization modules;when adding a new vis, the file goes here

Various other helper files

GTRI Proprietary / Limited Distribution

Repositories

• Analytics-Framework• Main Analytics-Framework repository

• https://github.gatech.edu/ascripka3/Analytics-Framework-GTRI_PROPRIETARY

• Repository access for DiamondEye• https://github.gatech.edu/DiamondEye

• diamondeye• Main web UI

• analytics• Submodule in the Analytics API for additional analytics, including SmallK’s HierNMF

• lean• Submodule in the DataLoader API for text processing

• smallk• Supporting code for SmallK and Pysmallk (older version which supports Pysmallk)

• diamondeye-basic• Vagrant box for use in setting up a development environment

• Follow the “Developer Instructions” or the “User Instructions”

• diamondeye-bells• Vagrant box with all additional repos for use in setting up dev environment

• Follow the “User Instructions”

GTRI Proprietary / Limited Distribution

• Python 2.7 dev• Java 8• Mongo server

• Python libraries:• Numpy

• scipy

• pandas

• scikit-learn

• flask, flask-restful, flask-cors, flask-restplus, django

• xlrd

• arrow, parsedatetime, geocoder

• brewer2mpl, vincent, markdown

• feedparser

Available libraries

GTRI Proprietary / Limited Distribution

Repository Structure for Submodules

analytics-framework

dataloader

conf

analytics

visualization

minimal

docs

configure.py

data

python

wsgi

Analytics

AnalyticsAPIv01.py

algorithms

Various other helper files

smallk(separate git repo)

python

__init__.py

__init__.py

hiernmf.py

• When adding submodules, place the module repo within the module directory (‘algorithms’ in this case)

• The submodule must include __init__.py files all the way down• There must be a file that conforms to the appropriate interface• When using configure.py to register the new module, you include the relative path

to the file:• python configure.py

--api analytics--filename smallk.python.hiernmf--mode add