SF MicroStrategy Bay Area Meetup

December 4, 2013

LinkedIn Confidential ©2013 All Rights Reserved

Agenda

• About LinkedIn

• BI Challenges

• Administration Hacks

• Q & A

Our Mission Connect the world’s professionals to make them

more productive and successful.

Our Vision Create economic opportunity for every

professional in the world.

Members First!

©2013 LinkedIn Corporation. All Rights Reserved.

The world’s largest professional network

Over 65% of members are now international

Active Company Pages

>3M

Languages

>30M

>91% Fortune 100 Companies

use LinkedIn Talent Soln to hire

19

259M+ Members Worldwide

Growing at more than two members per second

Students

Technology Stack

Technology Stack

Internal User Groups

Data Analytics

Infrastructure

Biz-Ops

Data

Sciences

Product

Hiring Solutions

Executives

Marketing Solutions

Talent Solutions

Business Intelligence Team

Bhavani Akunuri Sabeer Thajudeen Kevin Simon Poorna Lankipalli

Sandeep Deviah Qun Li Sunil Shirguppi

Agenda

• About LinkedIn

• BI Challenges

• Administration Hacks

• Q & A

Challenges

• BIG DATA - 4 V’s :

• Data Quality & SLA

• Navigation to reports/dashboards

• Multiple sources for answering critical questions

Velocity , Volume , Variety & Veracity

Microstrategy Meetup

December 4, 2013

LinkedIn Confidential ©2013 All Rights Reserved

Metadata Driven Dashboards

by: Kevin Simon, Sr Data Warehouse Engineer

How much would you be benefited if you

had the ability to effect real time change to

your critical dashboards, with little to no IT

involvement?

FOR THE ENGINEERS:

How much would you be benefited if your

users could make changes to their critical

dashboards, without needing to engage you

to make simple changes?

FOR THE BUSINESS USERS:

THE OLD WAY …

The fixed layout of this

dashboard makes it

difficult to manage.

IT resources can be freed to work on other

projects.

Real-time changes can effectively be made to dashboards.

Metrics can be introduced into your production

dashboards, prior to ETL development.

HOW DID WE FIX THE PROBLEM?

With a Metadata driven design…

PROCESS FLOW AND TABLE DESIGN

METRIC_SK METRIC_NAME CAT_ID CAT_ORDER METRIC_ORDER DISPL_FACTOR

100 PAGE VIEWS 100 10 10

200 SUBSCRIBERS (M) 100 10 20 1000000

9000 PV / UNIQUE MBR 200 20 10 1000

Daily ETL

Aggregate

Process

Daily Custom

Metrics

Forecasting

Model (R)

Dashboard

View

DATE_SK METRIC_SK COUNTRY_ID LOCALE_ID METRIC_NAME METRIC_VALUE

2013-11-01 100 500 1000 PAGE VIEWS 1000

2013-11-01 200 500 1000 SUBSCRIBERS 2000

2013-11-01 9000 550 1000 PV / UNIQUE MBRS 1500

Aggregate

Look up

IN CONCLUSION…

A Metadata approach to dashboard

design has allowed for real time change

to critical dashboards.

IT Resources can focus on other things.

More power and control given to users.

Same message across multiple dashboards.

Time to market greatly reduced.

Next Play

Challenges

• BIG DATA - 4 V’s

• Data Quality & SLA

• Navigation to reports/dashboards

• Multiple sources for answering critical questions

Microstrategy Meetup

December 4, 2013

LinkedIn Confidential ©2013 All Rights Reserved

Data Quality & SLA

by: Sabeer Thajudeen, Senior Software Engineer

Dashboard Delivery

M 1 M 2 M n

Daily ETL

Aggregate

Process

Daily Custom

Metrics

Forecasting

Model (R)

Trigger Cube

Validator

Trigger

Internal

Trigger

External

Command Manager

Data

Validation

(BTEQ)

Metric Alert WoW No

Data

M1 Y 20% Y

M2 Y NA Y

M3 N NA NA

Command Manager

- Trigger Event

(Dashboard)

Command Manager

- Trigger Event

(Alert Report)

Workflow

Scheduler

ABORT

……

MSTR Workflow

Challenges

• BIG DATA - 4 V’s

• Data Quality & SLA

• Navigation to reports/dashboards

• Multiple sources for answering critical questions

Microstrategy Meetup

December 4, 2013

LinkedIn Confidential ©2013 All Rights Reserved

Navigation to reports/dashboards

by: Bhavani Akunuri, Senior Software Engineer

Too Many Clicks

Resolution

• Identified top used reports

• Create short links

• Landing page re-design

Demo

Sample Code

Last Updated Timestamp

<web:ifBeanValue bean="frame" property="getDesignMode" value="0">

<web:then>

<div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div>

</web:then>

<web:else>

<web:ifBeanValue bean="frame.rb" property="getXMLStatus" value="6">

<web:then>

<div class="mstrMenuLastUpdatedCustom"> <web:displayGuiComponent name="report_last_update" isContainer="true"/> </div>

</web:then>

<web:else>

<web:displayGuiComponent name="report_design_area" isContainer="true"/>

</web:else>

</web:ifBeanValue>

</web:else>

</web:ifBeanValue>

Challenges

• BIG DATA - 4 V’s

• Data Quality & SLA

• Navigation to reports/dashboards

• Multiple sources for answering critical questions

Q: How are my key products performing?

Q: What caused the variance on KPI?

Q: Does it have a global effect?

Demo

Sample Code

Overlay Effect

<style>

.overlay_effect-ingraphs,

{

display:none;

position:fixed;

opacity:0.7;

height:50%;

width:50%;

z-index:100;

}

</style>

Pop Up Code

<script type="text/javascript">

$(document).ready( function() {

var main = $(this);

$("#ingraphs").click(function() {

loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs");

});

function loadPopupBox(overlay,popup){

$(overlay).fadeIn("slow");

$(popup).fadeIn("slow");

}

});

</script>

Sample Code

Overlay Effect

<style>

.overlay_effect-ingraphs,

{

display:none;

position:fixed;

opacity:0.7;

height:50%;

width:50%;

z-index:100;

}

</style>

Pop Up Code

<script type="text/javascript">

$(document).ready( function() {

var main = $(this);

$("#ingraphs").click(function() {

loadPopupBox(".overlay_effect-ingraphs",".popupInGraphs");

});

function loadPopupBox(overlay,popup){

$(overlay).fadeIn("slow");

$(popup).fadeIn("slow");

}

});

</script>

Next Play

Agenda

• About LinkedIn

• BI Challenges

• Administration Hacks

• Q & A

Microstrategy Meetup

December 4, 2013

LinkedIn Confidential ©2013 All Rights Reserved

Administration Hacks

by: Sabeer Thajudeen, Senior Software Engineer

Swapping iServer names in User Bookmarks

Context:

▪ Disaster (DR) recovery is critical . MSTR infrastructure has been set up for DR

▪ Each time we test DR, VIP and base URL remains the same. However, iServer names change

Impact:

▪ Bookmarks (to access reports directly) include iServer names and will not work

Resolution:

▪ Use MSTR SDK to seamlessly swap incorrect iServer names in bookmarks (External Security

Module)

Link

Implementation Detail

▪ Create a Custom ESM (External Security Module) that extends ‘Abstract External

Security’

▪ Override the function ‘handlesAuthenticationRequest’

– 1) Read iServer names in incoming URL using regex. Matcher and

regex.Pattern libraries ▪ Pattern pat_type_1 = Pattern.compile("Server=.{4}-rpt..", 2);

▪ Pattern pat_type_2 = Pattern.compile("mstrWeb=-.{5}-rpt..", 2);

for(Matcher mat_type_2 = pat_type_2.matcher(requestURL); mat_type_2.find();)

{ //read iServer name in incoming URL

String iServerName = mat_type_2.group();

….

– 2) Replace if iServer name does not match expected server name Example:

//Does iServerName in URL match with expected Server Name

if(! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=-")).append(Location).append("*-

").append(Server1).toString()) && ! iServerName .equalsIgnoreCase((new StringBuilder("mstrWeb=-

")).append(Location).append("*-").append(Server2).toString()))

{ ……

//Correct Server Name

requestURL = mat_type_2.replaceAll((newStringBuilder("mstrWeb=-")).append(Location).append("*-

").append(Server1).toString());

….}

}

– 3) Resubmit corrected URL - response.sendRedirect(requestURL);

▪ Update MicroStrategy Config Settings – Application Settings to use Custom ESM

(Web Customization Editor)

Monitoring MSTR via Web

Context:

▪ 24/7 Operations team to monitor and support infrastructure

▪ Handles not just MSTR but Teradata, Hadoop & ETL pipleline

▪ Prefer a web based system for monitoring rather than a desktop client

Solution

▪ Built a Web GUI to monitor critical aspects of the MicroStrategy infrastructure

– Delivery Status of all our critical dashboards

– Cube Refresh Status

– Job monitor for currently active jobs

– User Connection Monitor

InSights Monitor

MSTR Monitor

(Web Application)

Statistics

Tables

MSTR

Application

(iServer)

SDK JDBC

Implementation Detail ▪ Dashboard Delivery Status and Cube Status

– IS_DOCUMENT_STATS and IS_CUBE_REP_STATS

▪ Instantiate WebObjectsFactory: factory = WebObjectsFactory.getInstance();

▪ Force connection to each iServer //Get cluster information

WebClusterAdmin wca = factory.getClusterAdmin();

WebClusters wcs = wca.getClusters();

for(int i=0;i<wcs.size();i++){

for(int j=0;j<wcs.get(i).size();j++){

//Set load balance factor to 0 to force cionnection to other iServer

wcs.get(i).get(j).setLoadBalanceFactor(0);

jobDetails+= getJobDetails(session,factory,httpReqId);

wcs.get(i).get(j).setLoadBalanceFactor(1);

}

}

▪ Inside getJobDetails: //Retrieve job source. Loop through each job to get job info

//Use the same logic for the source WebMonitorTypeUserConnection to obtain session info

strSessionID = session.getSessionID();

source = (JobSource) factory.getMonitorSource(EnumWebMonitorType.WebMonitorTypeJob);

results = source.getJobs();

for (int i=0; i < results.getCount(); i++) {

Job singleJob = results.get(i);

job_id= singleJob.getJobID();

duration = singleJob.getDuration();….

}

Next Play

Credits

▪ Vincent Kwong – Program Manager

▪ Anirudh Gupta & Crowe Bonnie – Meet up Organizers

▪ LinkedIn AV team, Communications & Security

▪ Data Science & Biz-Ops

▪ ETL & Platform Teams

▪ NOC, DS Operations & DBA

Agenda

• About LinkedIn

• BI Challenges

• Administration Hacks

• Q & A