Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

43
javier ramirez @supercoco9 Get more from Analytics with Google BigQuery datawaki

Transcript of Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Page 1: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez@supercoco9

Get more fromAnalytics with

Google BigQuery

datawaki

Page 2: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

about me

19 years working on software: banking, e-commerce, government, CMS, start-ups...

founder of https://datawaki.comhttps://teowaki.com

Google Developer Expert on the Cloud Platform

mail: [email protected] twitter: @supercoco9

datawaki

Page 3: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

BigQueryis awes..

I use GoogleAnalytics

Page 4: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

Isn't Google Analytics good enough?

Page 5: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

Google Analytics is great but...

It lets you access aggregated data and reports, not individual sessions/visits data.

Even premium accounts get sampled reports when there are too many data (and not all the reports can be unsampled).

Page 6: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

Google Analytics is great but...

If you need to manage many different segments, and if you want to combine segments, it can get tricky.

Moreover, you can only segment or create reports using the pre-defined filters, which might or not be enough for you*.

*even if segments have experienced a huge improvement with Universal Analytics

Page 7: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

Google Analytics is great but...

It's not easy to cross data in Analytics with data from other sources (CRM, invoicing system...)

Now you can use Import Data from UniversalAnalytics, but there are many constraints to what you can do

Page 8: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

Google Analytics is great but...

Good for knowing what's happening in your application, but difficult for:

* business intelligence/big data (data mining, find patterns...)

* machine learning (classify information, predict future trends...)

Page 9: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

big data -analysed and organised into information- has big value

Page 10: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

BIG DATA

HIPSTER

javier ramirez @supercoco9 https://datawaki.com

Page 11: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the structures of your database architectures.

Ed Dumbill program chair for the O’Reilly Strata Conference

javier ramirez @supercoco9 https://datawaki.com

Page 12: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

bigdata is cool but...

expensive cluster

hard to set up and monitor

not interactive enough

Page 13: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

What if I could...

..query billions of rows in seconds..

..using a SQL-like web interface..

..on a fully managed cloud..

..paying only when I use it?

javier ramirez @supercoco9 https://datawaki.com

Page 14: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Designed to run analytics over huge volumes of raw data, and to integrate with other data sources

javier ramirez @supercoco9 https://datawaki.com

Google BigQuery

Page 15: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Google BigQuery

Data analysis as a service

https://cloud.google.com/products/bigquery/

javier ramirez @supercoco9 https://datawaki.com

Page 16: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

onemorething

Page 17: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Google Analytics Premium users get free daily exports from GA to BigQuery.

javier ramirez @supercoco9 https://datawaki.com

Google BigQuery + GA Premium

Page 18: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

All your raw data.Unsampled.Use it however you want.

BOOM!javier ramirez @supercoco9 https://datawaki.com

Google BigQuery + GA Premium

Page 19: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI
Page 20: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

o'reilly

Page 21: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

khan academy

Page 22: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

US Cellular Case Study5th largest US telecommunications company over 10.6 million customers

They didn't know how many offline (in-store +telesales) sales were originated by online media.

After combining GA data with other internal data,they can more accurately attribute sales to digital channel (website, social, search and display)

It helps them optimize their campaign and forecastsales

Page 23: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

data schema

javier ramirez @supercoco9

Page 24: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

it's just SQL

javier ramirez @supercoco9 https://datawaki.com

Page 25: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT trafficSource.source, SUM( totals.transactions ) AS total_transactionsFROM playground.ga_sessions_20140621GROUP BY trafficSource.sourceORDER BY total_transactions;

basic queries (metric/dimension)

SELECT device.isMobile, SUM ( totals.pageviews ) AS total_pageviewsFROM playground.ga_sessions_20140621GROUP BY device.isMobileORDER BY total_pageviews;

Page 26: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT IF(DOMAIN(trafficSource.source) is null,

trafficSource.source,DOMAIN(trafficSource.source))

AS normalized_source, SUM ( totals.transactions ) AS total_transactions

FROM playground.ga_sessions_20140621GROUP BY normalized_sourceORDER BY total_transactions;

basic queries with a twist

Page 27: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT ( SUM(total_transactionrevenue_per_user) / SUM(total_visits_per_user) ) AS avg_revenue_by_user_per_visitFROM ( SELECT SUM(totals.visits) AS total_visits_per_user, SUM( totals.transactionRevenue ) AS total_transactionrevenue_per_user, visitorId FROM playground.ga_sessions_20140621 WHERE totals.visits>0 AND totals.transactions>=1 AND totals.transactionRevenue IS NOT NULL GROUP BY visitorId ) ;

Average amount spent per visit

Page 28: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

2 segments, combined

Page 29: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT hits.item.productName AS other_purchased_products, COUNT(hits.item.productName) AS quantityFROM playground.ga_sessions_20140621WHERE fullVisitorId IN ( SELECT fullVisitorId FROM playground.ga_sessions_20140621 WHERE hits.item.productName CONTAINS 'Light Helmet' AND totals.transactions>=1 GROUP BY fullVisitorId ) AND hits.item.productName IS NOT NULL AND hits.item.productName !='Light Helmet'GROUP BY other_purchased_productsORDER BY quantity DESC;

Users who bought product A,also bought product B

Page 30: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT prod_name, count(*) as transactionsFROM(SELECT fullVisitorId, min(date) AS date, visitId, hits.item.productName as prod_nameFROM (SELECT fullVisitorId, date, visitId, totals.transactions, hits.item.productName FROM (TABLE_DATE_RANGE([dataset.ga_sessions_], TIMESTAMP('2014-06-01'), TIMESTAMP('2014-06-14'))))WHERE fullVisitorId IN(SELECT fullVisitorIdFROM (TABLE_DATE_RANGE([dataset.ga_sessions_], TIMESTAMP('2014-06-01'), TIMESTAMP('2014-06-14'))) GROUP BY fullVisitorId HAVING SUM(totals.transactions) > 1)AND hits.item.productName IS NOT NULLGROUP BY fullVisitorId, visitId, prod_name ORDER BY fullVisitorId DESC)GROUP BY prod_name ORDER BY transactions DESC;

* example query from the lunametrics blog. Check them out for more awesomeness

Products that are purchasedand lead to other products being purchased

Page 31: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

SELECT fullvisitorID, visitID, visitNumber, hits.page.pagePathFROM playground.ga_sessions_20140621where hits.type='PAGE'order by fullvisitorID, visitID, hits.hitnumber asc

Identify user path/user actions

Page 32: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

individual users data is awesome

Cross CRM data with individual users actions to seehow your response to incidents affect your users.

Use the “frequently bought together” query and findusers who didn't buy the related products. Send ane-mail campaign with an offer for those products.

Page 33: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

integrating with external data sources

* Connectors/REST API* Export into GCS* Import into BigQuery

javier ramirez @supercoco9 https://datawaki.com

Page 34: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Working with external data

Page 35: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

BigQuery pricing

$20 per stored TB

$5 per processed TB*the 1st TB every month is free of charge

** GA premium get $500 free credit monthly

javier ramirez @supercoco9 https://datawaki.com

Page 36: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

for GA premium users BigQuery is effectively

for free

*unless you upload huge external data or make huge queries

javier ramirez @supercoco9 https://datawaki.com

Page 37: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

What if I don't havea GA Premium

Account?

Page 38: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

just send your own data

javier ramirez @supercoco9 https://datawaki.com

define a data structure that fits your needs (or replicate the one GA provides)

use a JS snippet to send data to your server, thento BigQuery

Page 39: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

or use

javier ramirez @supercoco9 https://datawaki.com

Just add an extra snippet to your GA

datawaki

Page 40: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

send data from any other source (CRM, back-end,sensors, mobile apps, log system, external tools...)

datawaki

Page 41: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

PostgreSQL

Log

NGINX

Log

NGINX

Logstash

Redis

BigQuery

Neo4j

RubyWorker

RubyWorker

RailsApp

Alertsystem

datawaki in a nutshell

Reportsystem

userinteraction

data input

Page 42: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

javier ramirez @supercoco9 https://datawaki.com

* Get full access to your data

* Receive reports by e-mail

* Get individual or group alerts* if there is a purchase over $1000 * if a user has visited a product page over

20 times in one week and didn't buy* if a product is seen over 200 times one hour

* every time a product reaches 5000 views

datawaki

Page 43: Get more from Analytics with Google BigQuery - Javier Ramirez - Datawaki- BBVACI

Want to know more?https://cloud.google.com/products/bigquery/

Need help?https://teowaki.com/services

Thanks!Gracias

Javier Ramírez@supercoco9

datawaki