Information Systems for Managers

Emerging Trends in IT

Mani Garlapati (Lead Strategist, Google)

(x- Sr. Data Scientist, WalmartLabs)

Proximity Advertising

Promotional Effectiveness - Detailed

Promotional Effectiveness - Detailed Cont...

Analytics

Store Planogram

Association Rule Analysis

An association rule problem is where you want to discover rules that describe large portions of your data, such as people that buy X also tend to buy Y.

Market Basket Analysis is one of the applications of Association Rule Analysis in the retail industry.

Apriori Algorithm

It is a classical algorithm in data mining used for mining frequent item sets and

relevant association rules

Apriori uses a "bottom up" approach, where frequent subsets are extended one

item at a time

● Both X and Y can be placed on the same shelf, so that buyers of one item would be prompted to buy the other.

● Promotional discounts could be applied to just one out of the two items.● Advertisements on X could be targeted at buyers who purchase Y.● X and Y could be combined into a new product, such as having Y in flavors of X.

Out of Scope

Association Rule Analysis Cont..

Let X be an itemset, X ⇒ Y an association rule and T a set of transactions of a given database.

Support: It is an indication of how frequently the itemset appears in dataset.

Confidence: It is an indication of how often the rule has been found to be true.

Lift: Likely item Y is purchased when item X is purchased,while controlling popularity itemY

Out of Scope

Association Rule Analysis Cont..

Link

Out of Scope

DDPP / DIPP Framework

Descri

ptive Diag

nosti

c Predict

ive Prescri

ptive

What happened?

Why did it happen?

What will happen?

How can we make it happen?

1

*Source: Gartner Analytic Ascendancy Model

23

4

Hindsight

In

sight

Foresight

Analytics High Level Landscape

1

32

Analytics

Prescriptive Analytics

Descriptive analytics Predictive analytics

Enabling smart decisions based on data

What should we do?

Mining data to provide business insights

What has happened?

Predicting the future based on historical

patternsWhat could happen?

How do grocery cashiers know to hand you

coupons you might actually use?

How does Netflix frequently recommend just the right

movie?

Why do airline prices change every hour?

Analytics Value Chain

What best can happen?

What will happen next?

What if these trends continue?

Why is this happening?

Where exactly is the problem?

What actions are needed?

How many, how often, where?

What happened?Standard Reports

Adhoc Reports

OLAP, Drilldown

Alerts, KPIs, Trends

Statistical Analysis

Forecasting

Predictive Modeling

Optimization

Bus

ines

s Va

lue

Degree Of Intelligence

Understanding Patterns

Understanding Relations

Data

Understanding Principles

AI ML DL

Foundational Data Science

Machine Learning

Machine Learning helps a machine learn the data and provides it capability to identify future outcomes. Machine Learning is extensively used for predictive modeling and identifying trends or patterns from the data.

It can classified into below categories:Supervised Learning - Learn using labelled data and predict future outcomes.

1. Regression - When output is a real number2. Classification - When output has specific number of categories

UnSupervised Learning - Identify patterns when data is neither classified nor labeled.

Reinforcement Learning - Learn how to behave in a environment by performing actions and seeing the results.

Semi-supervised Learning - Some data is labeled but most of it is unlabeled and a mixture of supervised and unsupervised techniques can be used

SupervisedLearning

ReinforcementLearning

UnSupervisedLearning

MachineLearning

ML for Managers

Link

Bamboolib: GUI Python in Kaggle

Pandas Profiling

AutoViz: A New Tool for Automated Visualization - Github

Dora

Lens - Github

HOLOVIEWS

Automated EDA word document with all the analysis

EDA Functions for quick visualizations

AutoWEKA

Auto-sklearn

TPOT

H2O AutoML

TransmogrifAI

MLBox

DataRobot

Retail Banking Value Chain

Customer Acquisition

Customer Segmentation &

Profiling

Market Response Modeling

Campaign Effectiveness

AnalysisApplication Scorecard - Mortgages

Underwriting Analytics

Asset Valuation Scoring – Loan

Origination

Customer Relationship Management

Customer Lifetime Value

Analysis

Customer Risk Profiling

Cross – Sell / Up – Sell Modeling

Customer Transaction AnalyticsCustomer Account

Management Strategy

Text, Web & Social Media

Analytics

Portfolio Performance Management

Attrition / Retention ModelingPortfolio

Performance Analytics

Credit Research & Trends

Attrition Modeling &

StrategyRetention

Modeling & Strategy

Default Management

Default Risk Prediction

Behavioral Scorecard

Delinquency / Pre-Delinquency

Analytics

Fraud Analytics & Scoring

Promise – to – Pay

Propensity – to – Pay

Prepayment Risk Prediction

Portfolio Risk Management

Basel Analytics

Economic Capital Modeling & Reporting

Stress testing & Scenario Analytics

Credit Loss Migration

Loss Forecasting

Corporate & Support

HR Analytics

Market Assessment &

Entry

Competitive Intelligence

M & A Evaluation

Pricing Strategy Research

Structured v/s Unstructured Data

• Well-defined arrangement, easy to understand structure and comprehensible hierarchy is considered a

structurally sound entity.

• Seamlessly added in a relational database and are easily searchable by simplest of search engine

operations or even algorithms; whereas, the unstructured data is a nightmare for the designers to

connect the random strands of data with the existing meaningful ones and present it as a structure.

• Structural data is closer to machine language than the unstructured data.

Retail Analytics

Retail Value Chain

Customer AnalyticsAcquisition, Retention, Loyalty, Churn, Renewals

Human ResourcesHiring, Retention, Attrition, Employee Satisfaction, Skill Development, Workforce Analysis

FinanceRevenue Growth, Margin Improvements, DSOs, Payables, Fraud, GRC

Sales and MarketingLead Gen, Campaigns, ABM, New Launches, Inside Sales, Martech

Run-OperationsProductivity, Efficiency, Automation, Operations

Supply ChainSourcing, Storage, Supply, Planning, Logistics

SecurityPosture, Health, VA, PT, SIEM, Network Security, Access, ISMS

Customer Signals | Types

New Initiative Participation

Response to marketing

NPS Score

Social Media Markers

Attitudinal Indicators

Response

Experience

Behavior

Profiling

Category/Item Propensity

Life Time Value/ Hyper growth Proclivity

Omni-channel Propensity

Demographic Descriptors

Share-of-Wallet

Segmentation

Churn probability

HR Analytics

HR Analytics Value Chain

▪ Talent Mining/Attraction ▪ Plan for sponsoring selected conferences based on

hiring needs. Organize meet ups, analyze speakers, predict topics of interest. Plan hackathons on crowd sourcing platforms or in-house.

▪ Resume Screening using NLP, AI▪ Build NLP model which can best match the Job

Descriptions with the plethora of resumes extracted from various portals.

▪ Reduction of Bias for Promotion▪ Take feedback regularly and generalize achievements

ensuring diversity and mitigating risk.▪ Chatbot for HR Related Queries

▪ Chatbot to reply associates on general queries related to HR policies or announcements.

▪ Auto Response model for applicants/candidates▪ Candidates to get automated emails on their stage of

interview process.▪ Detect Attrition and Predict duration of stay before

attrition▪ Predict if an associate would attrite within a year and if

yes by when he/she would leave.

• Auto scheduling of appointments for interviews

• Create available time slots for interview panels and best match with candidates availability for auto scheduling of interviews.

• Optimum staffing• Identify complex projects, skills, resources

available and over-time spent by associates to optimize staffing.

• Compensation rationalization• Identify industry trends, competitor

benchmarks and rationalize compensation within the organization.

• Automation of operational HR Activities• General operational activities like pay slip,

employment certificates, taxation etc can be automated.

AR Vs VR

Augmented Reality

Search for dinosaurin Mobile

and View in 3D

IoT

How IoT works

RFID

Agriculture

Link

Activity - Problems to Solutions - How IS can revolutionize Farming.

Smart Meters - IoT

Impact of IoT

Business Impact

IoT in Retail

IoT in Manufacturing

IoT Impact in Manufacturing

IoT Impact across Sectors

Security Breaches in IoT

Databricks

DATA PROCESSES

Buzz analysisSocial media Data

Pricing strategyDemand Forecasting

Inventory Data

POS Data /

Shipments

ERP Data

Pricing strategyDemand Forecasting

Supply chain analysisRoute optimization

ShipmentsOrders

Portfolio optimizationProduct development

Third Party Data

Segmentation AnalysisBrand Equity Analysis

BudgetingWorking Capital Mgt

Sales and DistributionPlanogramming

Financial Systems

Sales Planning Systems

Portfolio OptimizationProduct Development

Survey DataSegmentation AnalysisBrand Equity Analysis

BudgetingWorking Capital Mgt

Sales and Distribution

Low

Volu

me

of D

ata

HighLack of Structure

High

Data Audit Profiling Cleansing Modeling Analysis

Syndicated DataIRI /Nielsen POS dataSales Lift AnalysisPromo EffectivenessROI Calculation

Market TrendsCategory Trends

Secondary Research

Validation

AI

AI Landscape

AI Applications

AI in Legal

AI in E-Commerce

Link

Challenges of AI

AI Performance

AI in Personal fitness

AI in Insurance

Link

AI in Banking

Learning SQL

SQL & NoSQL Differences

NoSQL & Apache HBaseRamco Cements LinkMother DairyBurberry’s Social Story

SQL Learning• https://www.tutorialspoint.com/mysql/ • https://sqlzoo.net/ • https://in.udacity.com/course/intro-to-relational-databases--ud197 • https://www.khanacademy.org/computing/computer-programming/sql • https://www.codecademy.com/learn/learn-sql • https://www.w3schools.com/sql/default.asp • https://academy.vertabelo.com/ • http://www.sql-tutorial.ru/ • https://www.essentialsql.com/

ResourcesDating Apps use AnalyticsNetflix leveraging Big Data AnalyticsThe Joy of DataAge of Big DataScience Documentary 2016 | Big DataThe Great HackThe Human Face of Big DataBBC Magic Numbers Mysterious World of MathsIoTIBM Watson in HealthcareThe Industry 4.0 with SiemensIndustries 4.0 - The Fourth Industrial RevolutionIoT in SCMSAP for Retail Use wearable Technology to Create Compelling Customer ExperiencesHow Assistive Tech Makes Aging in Place Possible for SeniorsTed talk on IoT

Augmented Reality op Rotterdam Central - National Geographic The new ŠKODA Fabia Augmented Reality Experience at WaterlooCoca-Cola Magic Augmented Reality App - Campaign ResultsRimmel's Get The Look appBBC Frozen Planet Augmented Reality - created by INDEAugmented Reality Demo for American Museum of Natural History

Virtual travel 'transporter‘

Merrell virtual hike

A Virtual Honeymoon to London

The Topshop Virtual Reality Experience AW14

Boursin – Sensorium

Michelle Obama 360

#VolvoReality XC90 Test Drive Teaser

Happy Goggles - A virtual reality headset made from a Happy Meal Box. Smart House

Databases

WHY SQL FOR MANAGERS?

Increase in data competency, thus

reducing turnaround time

Better presentation of hypotheses to

stakeholders using data

Better communication with technical teams

and evaluation of their work

Data-driven decision-making

Apply technology to

business problems

A skill that can never go out of

date

SQL Vs NoSQL

1. SQL databases are relational, NoSQL

databases are non-relational.

2. SQL databases use structured query

language and have a predefined

schema. NoSQL databases have

dynamic schemas for unstructured

data.

3. SQL databases are vertically scalable,

while NoSQL databases are

horizontally scalable.

4. SQL databases are table-based, while

NoSQL databases are document,

key-value, graph, or wide-column

stores.

5. SQL databases are better for multi-row

transactions, while NoSQL is better for

unstructured data like documents or

JSON.

Pros & Cons

ProsThey scale out horizontally and work with unstructured and semi-structured data. Some support ACID transactional consistency.Schema-free or Schema-on-read options.High availability.While many NoSQL databases are open source and so “free”, there are often considerable training, setup, and developments costs. There are now also numerous commercial products available.

ConsWeaker or eventual consistency (BASE) instead of ACID.Limited support for joins.Data is denormalized, requiring mass updates (i.e. product name change).Does not have built-in data integrity (must do in code).Limited indexing.

ProsRelational databases work with structured data.They support ACID transactional consistency and support “joins.”They come with built-in data integrity and a large eco-system.Relationships in this system have constraints.There is limitless indexing. Strong SQL.

ConsRelational Databases do not scale out horizontally very well (concurrency and data size), only vertically, (unless you use sharding).Data is normalized, meaning lots of joins, which affects speed.They have problems working with semi-structured data.

Relational DatabasesNon-Relational/NoSQL Databases

RDBMS: UTILITIES & COMPONENTS

Schema

Relation2-D table to store data

and relationships between data

CardinalityNumber of tuples in

a table

Overall structure of the database: tables, fields, relationships

Individual records or rows that hold

atomic values

AttributesNamed columns that represent data

Tuples

DegreeNumber of attributes in a table

Data and schema subject to change with requirements

Need for securing sensitive data

Data growing continuously & rapidly

Need to keep track of historical data

Need for higher processing speed

SQL Databases

Few Popular Databases•DB2•MySQL•Oracle•PostgreSQL•SQLite•SQL Server•Sybase•RethinkDB•Berkeley DB•memcached•redis•couchDB•mongoDB•GemStone•Teradata•Hana

SQL Commands

Transaction Control Language

TCL

o For managing transaction in a database

o COMMIT

o ROLLBACK

o SAVEPOINT

o For giving access rights to database users.

o GRANT

o REVOKE

Data Control Language

DCL

o For retrieving data from the database

o SELECT

Data Query Language

DQL

o For inserting new data and manipulating data existing.

o INSERT

o UPDATE

o DELETE

Data Manipulation Language

DML

o For defining and altering the structure of the database.

o CREATE

o DROP

o ALTER

Data Definition Language

DDL

Note: Self-Learning Links at the end and not covered as part of this module

TerminologyEntity – A thing in the real world with an independent existence.

Entity type – Set of entities that have some attributes.

Weak Entity Set – An entity set may not have sufficient attributes to form a primary key and its primary key consists of its partial key and primary key of the parent entity.

Table - In RDBMS data is organized in tables, these tables are called relations.

Row/Tuple – A row in a table represents the relationship among a set of values.

Attributes – Are the properties of the relation and are also known as columns.

Degree of a relation – The number of attributes (columns) in a relation (table).

Cardinality of a relation – The number of tuples (rows) in a relation (table).

View – A view is a (virtual) table that doesn’t exist physically, instead is derived from one or more underlying base tables.

Primary Key – Set of one more attributes that uniquely identifies tuples within a relation.

Candidate Key – All combinations of attributes that can serve as a primary key are the candidates for the primary key position.

Alternate Key – A candidate key that is not a primary key.

Foreign Key – A non-key attribute whose values are derived from the primary key of some other table is a foreign key in its own table.

DDL (DATA DEFINTION LANGUAGE) – The DDL provides a set of definitions to specify to specify the storage structure and access methods of the database system.

DML (DATA MANIPULATIVE LANGUAGE) – The DML enables user to manipulate or access data as organized by the appropriate data model.

There are 2 types of DML -

1. Procedural / Low-Level DML – DML requires a user to specify what data is needed and how to get those data.

2. Non - Procedural / High - Level DML – DML requires a user to specify what data is needed without the need to specify how to get those data.

DML Compiler – It translates DML statements in a query language into low – level instruction that the query evaluation engine can understand.

Query Evaluation Engine – It executes low level instruction generated by the DML compiler.

Metadata – Data about data.

DDL interpreter – It interprets DDL statements and record them in tables containing metadata.

Detailed Database Structure

JoinsThere are various types of joins available in SQL server. They are as follows.

Inner Join

Outer Join

Full Outer Join – also known as Full Join

Left Outer Join – also known as Left Join

Right Outer Join – also known as Right Join

Self JoinCross Join

--Inner JoinSELECT column-names FROM table1 INNER JOIN table2 ON table1.columnname = table2.columnname

--Right/Left Join

SELECT column-names FROM table1 RIGHT OUTER JOIN table2

ON table1.columnname = table2.columnname

--Self Join

SELECT column-names FROM table1 JOIN table1

ON table1.columnname = table1.columnname

--Cross Join

SELECT column-names FROM table1 JOIN table2

ON table1.columnname = table2.columnname

Window Functionswindow_function_name(expression) OVER ( [partition_defintion] [order_definition] [frame_definition] )

--Returns the sales for each employee, along with total sales of the employees by fiscal yearSELECT fiscal_year, sales_employee, sale, SUM(sale) OVER (PARTITION BY fiscal_year) total_salesFROM sales;

Data Architecture for IS