Post on 13-Jul-2020
Insights from designing a Sitecore-BI
integration
Why are we talking about this?
• Data as competitive advantage;
the “new currency”
• Sitecore as a contributor of
behavioural data
• Accessibility of tools: PowerBI,
Azure, Google Data Studio
The data collected by Sitecore can
inform important business
decisions, especially combined
with CRM and ecommerce
3/28/2018 4
Single View of …. Everything?
ERP
Finance
AI / ML Data Consumers
Etc…
A more present-day example
6
1
Lessons learned
from recent
Sitecore-BI
integration
[Disclaimer: Things get easier in Sitecore 9!]
Our ongoing client story: business context
• Professional membership association
• Seeking to continually increase brand value and offer
deeper value to the professional community
• Online revenue from publications, subscriptions, courses,
educational materials
• Recently revised and re-issued corporate objectives
• Running Sitecore 8.1
8
The asks
• Build a reporting framework for user behaviour to aid in the
measurement and optimization of marketing activities
• Collaborate with Client’s BI team to extract relevant data
into Client’s data warehouse
• = a great opportunity to demonstrate the value of xDB!
9
Insight #1: Consensus on how to
measure the digital business is a
fundamental prerequisite to BI
Initial ingredients
KPI Measurement Framework Maximizing use of Sitecore metrics 20 KPIs and tagging detail
12
Business Objective
Increase membership signups
by 10% in 2018
Marketing Strategy
- Align channel experiences
to customer intents
- Streamline the signup
process
KPI
- Members signups
- Signup conversion rate
- Engagement value
Segment
- Channel
- Persona
- Goal
Measurement Framework
** maximize usage of Sitecore-specific metrics and filters so they could be easily
implemented and measured
Insight #2: Marrying your KPIs to
marketing questions ensures the
dashboard can be designed to
answer them
3/28/2018 14
KPI: Session Value
This key measurement provides an understanding of the business value [Client]’s audience produces
Segments
+ All Data (Sitecore, GA)
+ Pattern Card (Sitecore, GA)*
+ Content Category/Tag (Sitecore, GA)
+ Member vs Non-Member (Sitecore, GA-Long Term)
+ Goal/Event (Sitecore, GA)
+ Country, Region, City (Sitecore, GA)
+ Channel, Campaign, Source (Sitecore*, GA)
+ New vs Returning (Sitecore, GA)
Questions That Can Be Answered
+ How much business value is being generated per visitor?
+ Which profile produces the most/least business value per visitor?
+ How much business value is generated on a particular category per visitor?
+ How much business value are Members producing per visit?
+ How much business value is being produced that included a particular asset/goal per session?
+ What geography is producing the most/least business value per visitor?
+ What marketing activity is driving the most/least business value?
+ How much business value are return/new sessions producing?
Insight #3: Choosing the right data
source requires careful
consideration and future-proofing
3/28/2018 16
Google Analytics vs. Sitecore Analytics?
+ Client had established maturity in using Google Analytics
+ Google Data Studio possible for short-term drill-down reports
+ However, GA came with some limitations:
+ Data queries would begin to apply sampling after a certain threshold to
keep performance in check
3/28/2018 17
3/28/2018 18
Pros of Sitecore that won out
+ Full, unrestricted access to all behavioral data and ability to do a nightly pull
into the data warehouse
+ Consistency between use of engagement value both as a KPI and later for
onsite personalization (as opposed to using only GA goals)
+ Extensibility of contact data model, useful later with future ecommerce
+ Client’s commitment to Sitecore as digital platform and licensing investment
Insight #4: Choosing the right
connection method gives
maximum data flexibility for BI
3/28/2018 20
Option 1: Connecting to the SQL Reporting Database using native connectors in Excel or Tableau
+ The way that Sitecore aggregates the data into those tables (facts and
dimensions) did not support the pivoting and filtering of data that the
dashboards needed
+ For example, the tables aggregated around specific metrics, or they removed
too many other fields that we wanted in the database
3/28/2018 21
Mapping dashboard data points to source options
3/28/2018 22
Option 2: Using Sitecore’s Reporting API to fetch data
+ This API will deprecate once xConnect is released in Sitecore 9
+ I would not recommend investing in a code/ETL layer using this older API
3/28/2018 23
Option 3: Querying Mongo directly
+ We had a need for more freedom in manipulating and pivoting the entire dataset in
multiple ways
+ We could not derive all data from the aggregated tables in the way we needed
+ We settled on making as much of the raw data available in the data warehouse so
the BI team could have complete freedom in building the dashboards and joining
to other business data
+ Caveats: we’ll invest in the 8.1 Mongo spec which will evolve with upgrades, and
we’ll need to do some customizations/API calls to get the full data we need
Insight #5: Pulling extra data at
collection time future-proofs the
architecture, with some exceptions
3/28/2018 25
Example 1: Content Category Tag lookup
+ How to discern which visitors had looked at content tagged with a particular
category?
+ We couldn’t do it reliably by template type, URL or any other piece of data
natively stored in the xDB
3/28/2018 26
Considering options
+ Option 1: Do an API call at transformation time to look up whether a content
tag existed on a viewed page
+ This runs the risk of tagging having changed when doing the lookup retro-
actively
+ Option 2: Capture the tag in a Page Event when the interaction was recorded
at collection time
+ We have a snapshot of the tag string at the moment of capture
+ Caveat: additional storage space used in Mongo
3/28/2018 27
Example 2: Storing Traffic Type lookup locally in data warehouse
Insight #6: Iterations work really
well for dashboard assembly and
testing consumption and value
3/28/2018 29
Educating the BI team and iterating on questions
+ Our project ran across a number of weeks alongside BI sprints
+ The BI team was working with a SQL-based data warehouse, but were
unfamiliar with Mongo
+ This structure allowed for a weekly rhythm of questions to naturally arise as
they mapped the dashboard field requirements to their transformation pipeline
+ I kept a living document of questions and answers
+ Very important: the BI team was provided a full dev data environment with
expected enrichment tagging
3/28/2018 30
Example: Translating Mongo GUIDs to Sitecore-searchable GUIDs
+ Run this Mongo Shell command (in this example, Page ID of first page visited)
+ db.getCollection('Interactions').find({_id:LUUID("f5a7ece7-5073-ad47-a4bd-
7963ad484513")})[0].Pages[0].Item._id
+ Browse to this utility:
+ http://guid-convert.appspot.com/
+ Use the HEX value to search the GUID in the Sitecore interface
+ Alternatively – the Studio 3T tool lets you paste and convert the GUID from Java encoding to .NET
encoding, with same result of being search in Sitecore
+ Must remember to change Preference to .NET notation
3/28/2018 31
In particular:
+ The implementation will challenge the KPI definitions and clarifications will
likely be needed
+ “what decisions will be made with this KPI?”
+ “why did we prioritize it as important?”
3/28/2018 32
Example: Translating dashboard requirements to formulas with Mongo fields
+ The formula for KPI “Loyalty Rate” is defined as:
+ 1. Total Visits (Interactions.Id.Count with at least one page contains “thought leadership” page event)
+ 2. Total Contacts (Interactions.ContactId.Count(Distinct) within the total visits defined as point 1 )
+ 3. Total Repeat Contacts (Total Contacts that have more than 1 Visits within the total visits defined in
point 1 )
+ Rate(%) = Total Repeat Contacts / Total Contacts(%)
Transformation pipeline (2 minutes)
Pull every 24h
Join on primary key (CustomerID)
“primary-eligible”
CRM
3/28/2018 34
v
35
2
The vision of
Sitecore 9
xConnect: making it easy to work with xDB data
+ Build applications to read, write or
search xDB data making it easy to
retrieve and consume analytics
data in an easy and flexible way
+ Sample applications
+ CRM integration
+ Data warehouse
+ Predictive modeling
3/28/2018 37
How organizations can leverage xConnect
Pulling CRM data
into Sitecore for
segmentation and
personalization
Capturing
physical/IoT data
such as in-store
beacon tracking
Simplified export
to data
warehouse
Applications of
data science for
greater insight
Sitecore Cortex: “Your own data scientist inside Sitecore’
+ Embedded machine learning
+ Coming with version 9.1
3/28/2018 39
The promise of machine learning for marketing
… or why is everyone putting machine learning in their software?
+ Machine learning lets us replace human judgement with prediction
+ What do computers do better than humans? Learning from and finding patterns within large datasets to make “good enough” predictions
+ (What do humans do better than computers? Creative applications of ML!)
+ Tedious, data-crunching processes that exist – in marketing, or in any discipline – are candidates for automation with machine learning
+ A classic marketing example:
+ The automation of image tagging via image recognition techniques
+ Instead of a human looking at each image and tagging “flower” or “car”, a computer could do that, and get smarter about it
Native connectors … happy dashboarding!
thank you