Simulation: A new modeling approach
Transcript of Simulation: A new modeling approach
simulate your market.
Simulation: A new modeling approachHow businesses can benefit from simulation
2
Contents
Contents .......................................................................................................................................................................... 2
Introduction: Social Simulation ...................................................................................................................................3
How to Simulate a Consumer Market .....................................................................................................................10
How Data Fits into Simulation..................................................................................................................................12
How to Build Confidence in Attribution ..................................................................................................................13
How to Avoid Over-Specification When Calibrating an Agent-Based Model ..................................................17
How Marketing Mix Modeling and Simulation Account for Uncertainty .........................................................19
Over-Specification: Comparing Agent-based and Marketing Mix Models ......................................................22
Attribution to What? Understanding Cross-Channel Interactions ...................................................................25
From Optimization to Strategy: Deeper into the Attribution Stack ...................................................................29
Marketing Effectiveness Measures: Know Your Options ...................................................................................33
Marketing Mix Modeling and Agent-Based Simulation – What Is the Question? .........................................34
Advantages and Disadvantages of Simulation .................................................................................................... .. 5
© Concentric, Inc. 2016-2017. All rights reserved.
3
Simulation is an analytical approach that gained in prominence with the advent of computers
and greater computing power. It is most useful when the system of interest has the following
properties:
1. The macro-outcomes result from the interactions of many micro-components.
2. The rules of interaction between the micro-parts are known.
3. The macro-outcomes are not obvious, without the calculation of all micro-behaviors.
Examples of these types of systems are the Universe, the Solar System, the Climate, the human
body, chemical substances, and elementary particles. It is not surprising that simulation is an
essential tool in astronomy, meteorology, chemistry, and physics.
Another example of such a system is human behavior. As an example, in the area of consumer
markets, sales, brand equity, word-of-mouth, and consideration all are metrics of interest to a
marketer. These metrics are products of the interaction of many consumers interacting with
certain rules. Simulation approaches that analyze human behavior are called Social Simulation.
Social Simulation is different from data analysis methods like regression or Markov chain models,
which are common social science approaches today. These methods are empirical. They derive
insight from data. As such, they inspire confidence. At the same time, they lack the ability to
make inferences about new or future states of the system of analysis. Simulation, on the other
hand, is a deductive method. It needs data only to be validated, i.e., compared to observation to
test its validity. To produce insight a simulation needs two things:
1. Rules of interaction of the micro-components.
2. Initial conditions.
In a sense, the rules of interactions form a system of logic and the initial conditions are the
premises. The macro-outcomes emerge as the logical conclusion of the rules and the initial
conditions. For this reason, simulation helps us to think about new or future states of the system
of analysis. The common difficulty in simulation is validation, but the latest advances in various
fields have led to breakthroughs in this area.
Introduction: Social Simulation
4
Lately, Social Simulation has been gaining prominence in economics as a way to improve
macroeconomic models and forecasting. Despite this, it is not well established in the business
community.
This white paper discusses many areas of advancement of Social Simulation that we have
achieved through the development of Concentric Market®, our simulator of consumer markets.
The paper offers best practices, clarifies the role of Social Simulation, and compares it to other
approaches.
5
Simulation is an approach that is used most commonly in two situations.
The first situation is when uncertainty is high due to sparse data. One such example is a
simulation of an ancient Native American tribe, the Anasazi, a culture that lived between the 9th
and 14th centuries. It is hard to run typical analytics on the limited available data, so researchers
use simulation to understand what happened to the tribe.
A second common use of simulation is for experimentation in a low-cost, low-risk environment.
Researchers at CERN simulate particles colliding in the Large Hadron Collider, before they
validate their forecasts in the expensive real-world collider in Switzerland. More common
examples include airline pilots practicing on simulated flights and doctors learning on test
patients.
Both of these applications of simulation are helpful to scientists and researchers, but they come
with a set of advantages and disadvantages. We have grouped these advantages and
disadvantages into three broad areas related to technology, process, and socialization. The
following table gives a summary of the advantages and disadvantages, that we elaborate on
below.
Advantage Disadvantage
Technology Forecasting under uncertainty Good theories needed
Able to answer many questions No standardized approach
Process Low data requirements to model Challenging to validate
Easy what-if scenario analysis Potential scope creep in projects
Socialization Low cost High skepticism
Innovative approach Political implications
Advantages and Disadvantages of Simulation
6
Great forecasting power, but a good theory is needed Data analysis methods such as regression are limited to forecasting effects of events that are
similar to what has already happened in the past. For example, if a brand has been investing in
TV ads within a range of $50M to $100M in the past few years, a marketing mix model is excellent
at forecasting what would happen if spend is within those bounds. However, the model is likely to
produce nonsensical results once it extrapolates to forecast what would happen if TV spend is
doubled or if a new marketing channel is deployed.
Simulation has an advantage over these methods in that it allows us to forecast things that have
never happened before and to run scenarios outside of historical bounds. The caveat is that we
need a good theory and causal hypotheses about how the system we are interested in analyzing
works. Theories that have high predictive power, at least in social science, are hard to come by
and may take years to develop.
Flexible, but not standardized Simulations, and agent-based modeling in particular, provide highly flexible techniques for
answering a wide range of research questions. These questions include what happened in the
first moments of the Universe, how wind turbulence around aircraft works, how the World Wide
Web evolves, or how to better design hospitals. Although simulation can be applied in a variety of
contexts, a formalized set of rules and best practices is not always readily available. For this
reason, simulation modeling (especially in social science) is incredibly creative, but may be
daunting for new researchers who have no single reference to consult when starting out.
Building a simulation does not require data, but validation does Simulation is an excellent approach to analyze problems when the available data is limited, since
no data is necessary to construct a simulation. Validating a simulation, however, often requires
multiple data sources to achieve a great degree of confidence in its representation of real-world
dynamics. The process of validation is a disadvantage for simulation when comparing to data
analytics approaches, since validating simulations is often more difficult.
For example, if we wanted to simulate traffic on a road, we would not need any data to start. We
could construct a simulation that incorporates modeled cars, driver behaviors, and road
conditions and voila: we have a traffic simulator.
7
Analysis of this traffic simulation could provide surprising insights – such as the pattern of traffic
jams migrating in the opposite direction that automobiles are traveling. But to test whether such
insights are valid, we would need to use various data. We would seek information about road
conditions in a range of contexts – in cities, on highways, in the U.S. and other countries, and
under different weather conditions. We could then recreate all of those scenarios within the
simulation and see how well they match what actually happened in the real world. If the
simulation data and the actual “validation” data match, then we have confidence that our
simulated model of traffic is useful and valid.
To get the simulations to match real-world outcomes, we need to change the theoretical rules
guiding the simulation or test different assumptions until they do. Simulations have the benefit
of forecasting multiple metrics simultaneously, but this can make it challenging to get all of the
assumptions synchronized. One change may improve the forecast for one metric, but degrade the
fit for another. Fortunately, expanding computing power and improving algorithms continue to
reduce the time and effort to overcome the process barrier of calibrating and validating
simulations.
Run any what-if scenario, but stay within budget We build simulations with the goal of having a “Petrie dish” in which to experiment in a controlled
and low-risk environment, before taking action in the real world. At the outset of a project, a team
can often list off a broad range of hypotheses to test within the simulator. Once a simulation is
built and what-if scenarios can be run, the desire to keep testing more and more scenarios often
grows.
Going back to the traffic simulation example, the initial goal of the simulation might be to
determine whether to replace 4-way-stops with roundabout intersections in a particular section
of town. The questions may compound from there: What will the impact be of traffic lights in
other parts of town? How should the signage be placed? How will this impact traffic during the
farmer’s market? It’s easy to see how the number of questions and scenarios can multiply very
quickly.
Enabling a team to test and answer more questions is a great value-add that simulation provides.
Projects may start by focusing on a single research question, but often grow to incorporate more
8
complex ideas. This dynamic and creative process can build consensus by bringing more
stakeholders to the table and ultimately lead to better decisions.
However, it is important when time and resources are limited to ensure that the scope of a project
does not expand beyond the available budget. Reaching agreement with the working team on the
appropriate balance between focus on a specific deliverable and open-ended exploration is a good
step to take early in the planning process.
Low cost, but high skepticism Compared to the cost of experimenting in the real world, the use of simulation requires very little
time and resources. Think about marketing: if we were to run various experiments in which we
varied the amount we invest in different channels, we would have to go through dozens of
budgets over as many years to gather enough data to answer a question with certainty. In the
meantime, our brand and business may have gone in an undesirable direction. The alternative to
real world experimentation is to run simulations to test different marketing plans. Within
minutes we can test many ideas before acting on a plan and making decisions in the real world.
The disadvantage of this approach is that some audiences today are skeptical of simulation. Most
of today’s analysis, especially in marketing, is based on reporting and building deterministic
statistical models to describe what has happened in the past. Researchers often prefer these
descriptive approaches to methods that test theories about the future. We believe that this
skepticism is a result of the relative novelty of simulation in marketing analytics, and that with
more success stories and validated forecasts, this skepticism will subside.
Innovative, but political Simulation may be one of the most innovative approaches researchers engage in today. We have
seen people advance in their careers for their intra-preneurial spirit in introducing simulation
within their organization. Long-standing and thorny problems get tackled every day with social
simulation, but the socialization of simulation results often presents organizational challenges.
Because it identifies trade-offs between a range of metrics, simulated insights may bring
conflicting interests to the forefront in organizations where stakeholder incentives are not
aligned.
9
Building consensus around a simulation is done best when the process is organized from the
start. All stakeholders should agree on the simulation’s framework, assumptions, and questions
to be answered at the outset. In our experience, this is the best way to ensure that the simulation
findings are impartial.
10
Simulation is flexible and somewhat unconstrained by nature. This open-endedness makes
simulation useful for understanding real-world markets in a broad variety of contexts. But given
this open-endedness, it becomes important to remain focused on a specific objective and to bring
some structure to the simulation.
Accordingly, the first step in building any Concentric Market simulation is to construct a
simulation framework. This framework should be built based on the consumer decision we are
seeking to better understand, in line with some business objective. The framework consists of the
following elements:
• Alternatives: What set of brands or options are in the consumers’ choice set?
• Attributes: What factors do consumers consider when selecting among alternatives?
• Touchpoints: What marketing activities or events influence consumers’ decisions or
behavior?
• Segments: What different types of consumers exist in the market?
Most importantly, these elements should be selected according to the specific consumer decision
you wish to simulate. Here are some additional guidelines when building a simulation
framework, along with the typical number of framework elements in simulations:
Alternatives (3-10):
• Consider the market share of the competitors. It’s usually appropriate to include the
dominant competitors in the market and to exclude brands with very small shares.
• Factor in how directly the alternative competes with your brand. It’s a good practice to
consider not only the size of the competitor, but also whether or not it presents a threat or an
opportunity for your brand’s market share. You may choose to include a brand with low
market share if it is an emerging threat to your brand.
• Consider any substitutes in adjacent categories that would be helpful.
Attributes (2-6):
• Consider the factors that matter most to a consumer when making a decision. Typically, this
list can be summarized into a few key drivers of choice.
How to Simulate a Consumer Market
11
• Observe what consumers are talking about and what messages marketers are broadcasting.
This can provide insight into what factors matter the most.
• Think about both tangible attributes (physically experienced by the consumer) and
intangibles (more ethereal in nature, e.g. “cool,” brand reputation).
Touchpoints (5-150):
• Consider anything that can change consumer perceptions or consideration or stimulate
conversation in the marketplace. Touchpoints may include traditional media channels, and
also other types of interactions with the brand (for example, in-store touchpoints, events, PR,
salespeople, online blogs).
• Remember to consider not only your brand’s marketing activities, but also the activities of the
competitors.
• Think about the level of detail at which marketing strategy decisions will be made when
determining how granular to make the list of touchpoints. For example, if the simulation will
only be used to determine broadly the mix of television and online advertising, then it would
probably not be helpful to expand the touchpoint list to include splits for different TV
programs, 15 vs. 30 second ads, etc.
Segments (1-6):
• Consider how the population can be divided into groups based on differences in behavior,
media habits, drivers of choice, and perceptions/consideration about the alternatives.
• Select a set of segments that are mutually exclusive and encompass the relevant population.
A segment may be demographic, attitudinal, psychographic, or based on other factors so long
as that holds.
• Determine whether there are any key non-purchasing influencer groups (e.g.
doctors/pharmacists, children) that should be included in the simulation. Even if an
influencer group does not purchase any of the alternatives, they may still be active in the
social network and have an impact on outcomes.
12
Building a simulation does not require any data. Validating a simulation, on the other hand, does.
Building a simulation only requires that we make assumptions. We assume initial conditions,
constraints, and behaviors within the system we seek to understand. Qualitative or quantitative
data may be used to inform some of these assumptions, but often times we just apply insights or
judgment.
Once we have made assumptions and built a simulation, we run the simulation and see if it
accurately reflects real world dynamics. This validation requires data.
Calibrating a valid simulation is an iterative process. The test-and-learn nature of the process is
what frees us from being bound by data when initially building the simulation. However, the
validation points by which we judge the accuracy of our simulation are based on data.
Initial Conditions
+
The Right Rules
->
Simulated Outcomes = Real World Outcomes
How Data Fits into Simulation
13
Simulation, like any other modeling approach, is an imperfect representation of the real world. A
question that commonly comes up when performing analysis and forecasting is “how can we be
confident that what this model tells us is right?” Of course, all modeling results have some
uncertainty, but attribution is a particularly difficult finding to validate.
In a nutshell, attribution is the decomposition of a variable into the elements driving changes in
its value. Attribution results in a measurement of the impact of each element. For example,
marketing mix analysis often decomposes sales into base (an unexplained contribution) and
individual marketing channel contributions. The difficulty in building high confidence about
attribution is that the numbers reported are virtually impossible to measure in the real world. If
we say that TV contributes 30% to the increase in sales, we are making an accounting statement.
There is no way to go into the field and truly isolate that 30% figure. Getting to that 30% requires
assumptions about how we measure effects and how we quantify contribution.
Yet, while attribution is in this sense more art than science, there are ways to increase our
confidence in the attribution results that come from a simulation. More often than not, increasing
the confidence requires investing more time into validation of the model against metrics that are
actually observable and measurable in the real world. Only then, by proxy, is our confidence
boosted that our latent results are as valid as the results we can compare with measured
observations.
Below we outline 4 routes that can be taken to increase the confidence in the simulation
attribution:
1. Calibrate the simulation to multiple metrics at an aggregate level.
2. Perform a holdout forecast.
3. Compare individual consumer journeys to simulated agent journeys.
4. Conduct in-market tests to assess impacts.
Calibrating the simulation to multiple objectives Simulations are often built with a relatively large number of parameters and initial conditions.
This often raises a question of how one can be confident that they have not over-fit the
How to Build Confidence in Attribution
14
simulation. One way to avoid over-fitting risk and build confidence is to calibrate to more data
points.
Typically, regression approaches fit to one time series, for example, sales over time for one brand.
With a market simulation, one may calibrate to sales for the primary brand of interest as well as
sales for its competitors. The number of data points that build confidence in the fit is then quickly
scaled up by the number of competing brands.
In addition to calibrating to sales across brands, one may also calibrate to other metrics beyond
sales. Simulations can be calibrated to brand metrics such as consideration, perceptions, and
word-of-mouth volume. The number of calibration points can be scaled even further by the
number of KPIs that are fit.
Concentric Market® forecasts multiple metrics, all of which can be compared to data.
Fitting a simulation to data from multiple brands across KPIs builds confidence that it is not over-
specified and that it recreates the dynamics of the real-world.
15
Performing a holdout forecast Often stakeholders are interested in the predictive power of a model or simulation. In this case,
the evaluation of the approach goes beyond analytics to explain the past and into how well it
projects future outcomes. A holdout forecast is one approach for assessing predictive capabilities.
When conducting a holdout, the historical data is split between a calibration time frame and a
holdout period. The model or simulation parameters are tuned to the calibration time frame only.
This calibrated model is then used to make a forecast during the holdout period.
An example of a calibration and hold-out analysis.
Using a holdout, we see how effectively the simulation was able to forecast past outcomes and
this provides a better sense of what sort of predictive accuracy may be anticipated in forecasting
future outcomes.
Analyzing consumer journeys One of the advantages of simulation is that it recreates the dynamics that take place at an
individual level in the real world. Aggregate sales outcomes emerge from many individual
consumer decisions. One may be interested in seeing not only how well macro outcomes are fit
by a simulation or model, but also how well the approach recreates individual-level data. These
journeys can give us a better sense of how all the marketing activities and earned touchpoints
interact to lead to sales.
16
If data on individual consumer journeys is available, this can serve as another way to assess the
validity of a simulation. Simulated agent journeys can be recorded and compared to the data on
individual consumer journeys. If the simulated journeys align with the actual consumer journeys,
then the simulation is tuned to the micro-level dynamics of the simulation.
Conducting in-market tests Ultimately, an audience may determine that the best way to assess the impact of a marketing
strategy is to actually put it into practice in the real-world. An in-market test allows for data to be
gathered that can help measure impacts. Although running a test in the actual market allows one
to see how an outcome pans out in the complex real-world, it is difficult to run a controlled
experiment when the many variable factors of the real world come into play. Even a real-world
experiment has its limitations in terms of building confidence in a particular finding.
An example in-market forecast that results from calibration and hold-out analysis.
17
Practiced statisticians are very familiar with the problem of over-fitting when building traditional
econometric models. As the number of parameters approaches the number of data points fitted,
two things will happen:
1. The model will fit the history progressively more precisely.
2. The model will become progressively more over-specified, eventually to the point of losing
all statistical and predictive value.
This is why experienced model-builders place such high value on parsimony and generally avoid
complicating a regression model with extra variables unless they contribute significant
explanatory power. The classical “degrees of freedom” problem can eliminate a model’s
credibility.
To summarize, as the number of regression parameters increases with a number of data points,
over-fitting risk also increases.
Similar concerns may arise when building a simulation. While having the ability to adjust a
variety of initial conditions and settings in an agent-based model may provide a lot of flexibility,
may we encounter the problem of over-specification when calibrating an agent-based model?
The answer is yes, absolutely. With all the various model settings in Concentric Market® one
could simulate an annual sales figure that matches with historical values an array of input
assumptions. Many of these assumption sets would not align with reality and result in
simulations that are misleading and lacking in insight.
There are two ways to avoid the problem of over-specification in agent-based modeling.
1. Reduce input uncertainty: Decrease the number of freely floating inputs when calibrating your
model by building confidence in your input settings via research, data analysis, expertise, or
norms. This will cut down the model calibration search space to a more manageable size and
reduce your risk of over-fitting.
How to Avoid Over-Specification When Calibrating an Agent-Based Model
18
2. Increase the number of validation points: Calibrate to more metrics and trends based on what
you know to be true. Go beyond sales to assess consideration, word-of-mouth volume, and
brand equity outputs. Analyze the sales and brand metrics over time to see if the trends are
sensible. Look at brand equity attribution to see if the relative impacts of each touchpoint are
realistic. Carry out sensitivity analyses to see if the simulation mimics marketing responses
you would expect.
By implementing these two approaches, you are attacking the problem of over-fitting risk from
both sides: You are reducing the number of parameters in your search space and increasing the
number of data points in your validation.
Your efforts should always be geared towards the business question you are ultimately trying to
answer. This should help guide where to focus your energy in reducing input uncertainty and
selecting your validation points.
Concentric Market is meant to be used as a “thinking system.” The more thought that goes into
the system inputs and validation efforts, the more confidence you will have in the insights the
system generates.
So, keep this simple rule in mind:
As Thinking ↑, Over-fitting Risk ↓
Always.
19
“I can calculate the motion of heavenly bodies but not the madness of people.” said Sir Isaac
Newton. Almost four centuries later, the sentiment is still strong among practitioners of
predictive analytics that forecast human behavior.
It is true – we lack theories and empirical evidence to come close to predicting human behavior
with the accuracy found in the physical sciences, and making choices based on models of human
behavior is risky. But we have identified ways of quantifying and analyzing that risk.
One of my projects back when I worked in actuarial science was to quantify a range around the
expected paid losses (i.e., the portion of losses that the insurer actually pays). I set to work
analyzing the historical data and began attempting to quantify the range.
Process Risk After fitting a model to the data, my first approach was to estimate how much variability seemed
to be inherent in the process. I quantified the range by estimating the error term of the model –
i.e., the volatility observed in the actual data around the model fit – and assuming a distribution
for the error term. This type of variability is often referred to by actuaries as “Process Risk.”
Parameter Risk I proudly shared my results with the group, but then another issue was raised. In order to
compute an error term, I had to assume a model with specific, fixed parameters. What if those
parameters, which had been calibrated to a sample of data, were not accurate? On top of the
“Process Risk,” there is additional risk due to uncertainty surrounding the model parameters –
“Parameter Risk.” Adding a bit more statistical rigor to my analysis, I came up with an estimate of
this term.
Model Risk With both process and parameter risks taken into consideration, I was confident in the range
around the expected loss estimate. Then I was asked a final question. What about “Model Risk”?
What do we do if the assumed functional form or structure of the model does not reflect the
How Marketing Mix Modeling and Simulation Account for Uncertainty
20
reality of the underlying process? This one had me stumped. After months of analysis and
research, I could not solve this puzzle. Actually, it turns out the solution to the “Model Risk”
predicament had alluded the entire group.
An outline of the three types of risk.
In statistical models, the different types of risk can be quantified in a formulaic manner. In
simulations – different types of risk can be assessed in an empirical manner – based on more
iterative approaches.
So how does simulation deal with the various types of risk?
Monte Carlo analysis assesses Process Risk This involves running multiple iterations of a simulation with the same initial conditions. The
simulation can evolve differently each time due to the probabilistic and path-dependent nature of
the approach. We can then observe the inherent risk in a given strategy.
Sensitivity analysis assesses Parameter Risk This entails building a series of simulations with inputs that are tweaked in each run. Variation
in simulated outcomes may be observed as the inputs are adjusted. We can then assess how
sensitive outcomes are to shifts in initial conditions or assumptions for which we are uncertain.
Testing different rules of the simulation assesses Model Risk Assessing Model Risk requires critical thinking about the fundamental assumptions made in the
simulation framework. Have we considered all the relevant competitors or drivers of choice? Do
21
we understand the consumer segments and all the touchpoints that affect them? We should also
consider the rules of the simulation. Are the underlying algorithms relevant in our real world
market? Are our assumptions about how consumer decision making, paid media effects, and the
social network accurate? These questions raise challenges that cannot be addressed overnight.
Model Risk is best mitigated through collaborative, creative, and strategic thinking.
So, Newton is still right: We cannot account for all types of risk and uncertainty. This is not just
because of the limitations of our analytical methods, but due to the shear unpredictability of a
complex, ever-changing world. The best we can do is to decipher patterns and build insights
through data analysis and statistics, research and observation of the world around us, and our
intuition. Simulation can help in our efforts to better understand the uncertainties and risks
involved in markets and business strategies. And these efforts will be most successful with a
team that is open, adaptive, knowledgeable, persistent, and creative.
22
A common objection that we hear during discussions about agent-based modeling (ABM) is that
agent-based models are over-specified and therefore their findings are invalid. Although it is
possible with ABM — as with any other modeling technique — to encounter the issue of over-
fitting, building the simulation diligently will minimize the risk of over-specification (more details
on how to minimize the risk of over-specifying are given in an earlier essay). In fact, agent-based
models often have a lower risk of over-specification than marketing mix models. In this essay,
we will explore an illustrative example.
A metric that is commonly used to assess whether or not a model is over-specified is the ratio of
calibration data points to model parameters. If there are too few data points per model parameter,
the model is over-specified. In an extreme case where there is one parameter for every
calibration data point, the model can achieve a perfect fit. The parameters, in a sense, would just
be a transformed restatement of what happened historically. This model would not be very useful
for making any forward-looking projections or drawing any conclusions about what happened in
the past.
Let’s compute the ratio of data points per model parameter for example agent-based and
marketing mix models below:
Marketing Mix Model Calibration Data Points:
Marketing mix models are often fit to a time series of a brand’s sales. In this example, let’s
assume the model is fit to two years of historical weekly sales data. That gives us:
2 years x 52 weeks = 104 calibration data points
Calibrating Model Parameters:
Marketing mix models are typically fit by linking changes in sales to different marketing
activities. Let’s assume there are 8 touchpoints that are being analyzed. There will be a
coefficient that defines the impact of each touchpoint on sales. Additionally, each touchpoint‘s
Over-Specification: Comparing Agent-based and Marketing Mix Models
23
marketing activities can also be modulated by parameters that define the point of saturation and
decay rate of the marketing impact. Taken together, there are 3 parameters defining each of the 8
touchpoint effects plus a y-intercept – so a total of:
8 touchpoints x 3 parameters per touchpoint + 1 = 25 calibrating model parameters
Ratio of Data Points to Model Parameters:
104/25 = 4.2 data points per model parameter
Agent-Based Model Calibration Data Points:
Concentric Market® simulations are often fit to multiple time series. These time series
correspond to KPIs across competing brands. Let’s assume the simulation is fit to one year of
historical weekly sales data, but not just for one, but for six competing brands. Let’s also assume
that the simulation is fit to tracking data on consideration and perceptions for each of the six
brands. However, in this case, let’s say the tracking study was only run quarterly. The total
number of calibrating data points is then:
6 brands x 52 weeks of sales + 6 brands x 4 quarters of consideration + 6 brands x 4 quarters of
perception = 360 calibration data points
Calibrating Model Parameters:
Concentric Market simulations are built by integrating data from various sources and calibrating
a set of parameters to available calibration data. Let’s assume that the settings defining
consumer behaviors and initial preferences and consideration levels are provided from a
consumer survey or tracking study and are therefore constrained. The parameters that are used
to fit the simulation are then based on three types of settings:
1. Touchpoint impacts on consideration and perceptions: 4 per touchpoint
2. Consumer consideration and perception decays: 2 in total
3. Decision-making parameters: 2 in total
4. 8 touchpoints x 4 parameters per touchpoint + 2 + 2 = 36 calibrating model parameters
24
Ratio of Data Points to Model Parameters:
360/36 = 10 data points per model parameter
In the example above, the risk of over-specifying the ABM is lower than the risk of over-specifying
the marketing mix model. Despite the fact that the ABM is fitting a time period that is half the
length, the number of calibrating data points per parameter is over twice as high as the marketing
mix model. This is because an ABM fits an entire market as opposed to one particular brand in
isolation.
Ultimately, there are a number of approaches that may be taken to build confidence in the validity
and usefulness of a given model. These approaches include alternative specification formulas
like a log function, calibration against multiple time series, holdout forecasts, and in-market tests.
By diligently taking such approaches, our users consistently build valid and useful market
simulations with agent-based modeling.
25
Cross-channel effects occur in situations where the sum of the parts is not equal to the whole.
These effects can be positive (synergistic) or negative (inefficient).
For instance, imagine that you have two media channels at your disposal: TV and Facebook.
Should you run them separately or together? The answer to that question depends on the nature
of cross-channel effects between these two channels: whether there is a positive or negative
interactive effect from running these two channels together. If the sales lift you get from running
these two channels together is different from what you would get if you ran each channel
separately and added the sales lift you got from each, then there is a cross-channel effect. The
types of cross-channel effects that can occur are defined as follows:
Synergistic Cross-Channel Effect
A case when the cross-channel effect results in a synergy
In this case, the sum of the parts is less than the two considered as a whole. There is some extra
lift in sales that results from a synergistic interplay between both channels.
Attribution to What? Understanding Cross-Channel Interactions
26
Inefficient Cross-Channel Effect
A case when the cross-channel effect results in an inefficiency
In this case, the sum of the parts is greater than the two channels working together, which has
created some inefficiencies.
The cross-channel effect is therefore defined as the difference between the individual lifts and
the total lift:
[Lift in Sales from TV] + [Lift in Sales from Facebook] + [Cross-Channel Effect] =
[Lift in Sales from TV & Facebook].
The effect of marketing activity on sales is often analyzed using regression techniques using
sales as a dependent variable. Cross-channel interactions may be quantified with regression by
including an interaction term in the model specification. This analytical technique is useful for
measuring the effects of media, i.e., allowing for sales attribution and cross-channel interactions
to be computed. But the task of understanding what mechanism actually causes the cross-
channel effects remains a mystery.
To fully understand the mechanism that drives cross-channel interactions – we need to
recognize that the linkage between marketing activity and sales lies in the mind of a consumer.
The consumer’s decision-making behavior is what ultimately drives sales. So how do consumers
choose?
27
Consumer Decision-Making is the Key to Cross-Channel Effects The growth and acceptance of behavioral economics as a science behavior has been instrumental
in allowing us to take some simple rules to create simulated models of consumer decision-
making that take into account heuristics and rationality. This knowledge has allowed us to build
models to measure consumer behavior that go beyond notions of perfect equilibrium and neo-
classical economics that were never originally designed for this purpose.
In simple terms, consumers make decisions based on options they would consider, and select the
option they perceive most highly on attributes that are most important to them (based on the
behavioral principle of utility maximization). This view assumes a path in the consumer-decision
making process as follows:
The components of the consumer decision-making process lead to the different cross-channel effects
This approach means that both brand consideration and brand perceptions are important in
driving sales. Some touchpoints may be more effective at building consideration and others at
building perceptions. It won’t do the marketer much good to focus on building consideration if the
product is perceived very poorly – a perception or satisfaction bottleneck that may not be solved
by throwing more media at the problem. Likewise, it might be unhelpful to attempt to build
deeper engagement for a brand no one recognizes – a consideration bottleneck.
Synergies and inefficiencies may result from the consideration stages that exist in the complex
non-linear path to consumer decisions. Inefficiency may occur when activities focused on
consideration-building or perception-building continue when brand consideration or brand
perceptions are already maximized. We may saturate consideration and observe that any
additional marketing yields no incremental benefit. A synergy may occur when touchpoints that
28
build consideration and others that build perceptions are used in conjunction. We will get more
out of our perception-building media campaign, for instance, if more of the consumers are made
aware of the brand.
In short, inefficiencies may occur when we concentrate too much on one step in the consumer-
decision making process, and synergies may occur when we balance our efforts between different
steps. See below for a mapping of TV and Facebook from our example.
A mapping of two channels on two criteria: how well they affect consideration and perceptions
Building understanding of what drives synergies and inefficiencies within a marketing plan can
bring valuable additional insight to strategic decision-making. Thinking about the path in the
non-linear consumer-decision making process – and the various bottlenecks and saturation
points therein – may allow a team to consider new creative strategies that balance consideration
building and perception building activities. Simulations may then serve as an aid to testing these
hypothetical strategies prior to implementation to include interim metrics such as brand
consideration, brand perceptions, and word-of-mouth activity.
So, as you think about the attribution of all of your brand activities, ask yourself: attribution to
what?
29
Sales attribution is a common diagnostic used in evaluating marketing effectiveness. This
involves decomposing a brand’s sales into “Base” and “Media” components. Media is often further
divided to attribute sales by distinct touchpoints. The result can be visualized as a stacked
attribution chart as follows:
Stacked Attribution Chart
This chart provides an indication of how much touchpoints are lifting sales and therefore can be
used to assess the Return on Investment (ROI). A planner may use these outcomes to change the
mix, putting more weight into the touchpoint with greater ROI.
Simulation makes it possible to dive deeper into attribution analysis to provide more insights and
further guidance on marketing strategy. Here are four more detailed views of the stack that
simulation provides:
1. Understanding how creative and messaging impact the effectiveness of each touchpoint.
2. Identifying the synergies and inefficiencies that exist between channels.
3. Decomposing the base to understand the role of earned media and brand history.
4. Placing the attribution into a consumer-focused competitive context.
From Optimization to Strategy: Deeper into the Attribution Stack
30
Creative Execution and Messaging Emphasis The concepts of creative execution and messaging emphasis allow us to go beyond analysis of
levels of investment and reach to analyze the effectiveness of media. Creative execution defines
how well the ad is created — the production value and how effectively it communicates the
message. Messaging emphasis defines what attributes of the product are the focus of the ad. For
example, is the ad part of a promotional campaign or a branding campaign? Unlike most
marketing mix modeling, simulation allows us to test the effects of different creative executions
and messaging emphasis and decompose each individual touchpoint contribution further.
Creative Execution and Messaging Emphasis
Cross-Channel Effects Touchpoint impacts may interact with one another through duplication or social network effects.
Inefficiencies occur when reaching the same audience via multiple touchpoints leads to no gain.
Synergies occur when messaging on two touchpoints yields better results than using each
touchpoint on its own. Simulation can serve as a guide to better understanding and replicating
these interactions in the consumer’s mind and provide a platform to test different approaches to
fully leverage synergies and avoid inefficiencies.
31
Cross-Channel Effects
Decomposing the Base The “Base” is often the forgotten piece of the attribution stack – an unexplained variance that
does not yield any actionable insights. Running a simulation that incorporates the effects of
earned media – word-of-mouth (new restaurant opens up) and product experiences (the joy of
driving your automobile) - plays a major role in driving a brand’s performance. Sometimes the
base is simply a matter of consumer inertia (think about insurance), seasonality (canned soup), or
a brand’s history (Coca-Cola vs. Pepsi). Better understanding the base may provide strategic ideas
on how to avoid erosion of the base or how to keep it growing strong.
Decomposing the Base
32
Considering the Competition Sales outcomes are the aggregate result of many individual consumer decisions. Consumers
make choices among a set of alternatives – and so in many cases a brand’s performance is just as
much dependent on what its competitors do as what the brand does itself. Building a simulation
of an entire market where simulated consumers choose among competitive alternatives allows
for an assessment of how a brand will fare in different competitive environments. If a brand
positions itself appropriately, it may rapidly grow by cannibalizing competitive share. If not
positioned appropriately, the entire attribution stack may sink into oblivion (think about
Blockbuster Video with the emergence of Netflix and Redbox). Considering competition is key to
evaluating strategies to boost and maintain market share over the long run.
Considering the Competition
Conclusion – Moving beyond Optimization to Strategy The initial view of the attribution stack allows for an assessment of ROI that can guide decisions
around how much to invest in touchpoints. However, assessing the stack from the lens of
creative messaging, cross-channel synergies and inefficiencies, earned media and brand history,
and the competitive environment allows for a richer understanding of how a brand is likely to
perform in its market. Simulation provides flexibility in approach to plan and react strategically
in a variety of ways.
33
There are a variety of approaches used to quantify the impact of marketing activities and attribute
sales to touchpoint activities. Such methods include:
1. Marketing Mix Modeling
2. Test and Control Group Experiments
3. Attribution Modeling
4. Agent-based Modeling
In marketing mix models, sales or some other KPI of interest are regressed against marketing
activities to understand how changes in the KPI are related to changes in marketing activities.
The resulting coefficients define the sensitivity of the KPI to touchpoint activities. From this
model specification, the quantity of sales to attribute to a given marketing activity may be
derived.
In test and control experiments, one group (the test group) is exposed to a marketing treatment
while another group (the control group) is left unexposed. Ideally, these two groups are otherwise
completely the same. The difference in outcomes (sales, response rate, or other KPIs) between the
two groups is then attributed to the marketing treatment.
In attribution modeling, individual level data is used to assess what individual touchpoint
activities (commonly digital ads) led to a sale or some other action. These models typically link a
KPI (such as a sale or website visit) to one or some combination of activity that preceded it. These
accounting approaches include last touch, first touch, or some other weighting across touchpoint
activities.
In Concentric Market® agent-based models, the dynamics of a marketplace are recreated by
simulated consumers who decide between alternatives, interact in a social network, and
experience marketing. In this way, the impacts of marketing activities are considered in the
context of all interactions a consumer may have with brands in a market – including not just paid
marketing for a single brand, but competitive actions and earned media.
Marketing Effectiveness Measures: Know Your Options
34
Often, we are asked about how agent-based simulation compares to marketing mix modeling.
How do the methodologies compare? Although parallels between the two exist in the context of
planning paid investments in marketing, the question is flawed because the two methodologies
serve very different purposes.
Marketing Mix and Simulation for Planning Investments in Marketing Marketing mix modeling is a statistical approach that explains variation around some baseline
value in a dependent variable (often times sales) due to marketing activities. A regression is
constructed between time series of sales data and marketing to attempt to correlate changes in
one to another. The model’s coefficients are calibrated to minimize the error between actual and
modeled sales. The model is then applied to answer questions around marketing investment
level and mix.
Agent-based modeling is an approach based on simulation that recreates a market and its sales
from the bottom-up. Interdependent variables within the simulation recreate the dynamics of the
market. Each sale in the simulation results from the action of an individual simulated agent. The
initial conditions and parameters define how consumers behave, influence one another, make
decisions, and interact with media. The parameters of the simulation are adjusted to minimize
error between actual and simulated sales. The calibration process confirms that the simulation is
an accurate representation of the world. The simulation is then applied to answer a broad set of
questions about marketing and product development resource allocation.
A number of parallels between the two methodologies often create confusion about their
similarity. Each has coefficients or parameters that are adjusted to minimize error in an attempt
to better reflect reality. Each approach may be used to guide decisions on paid marketing
investments. The key distinction is that marketing mix analyzes a single brand’s activity in
aggregate, while agent-based simulation recreates a brand’s marketplace based on individual
consumer actions. The differences are summarized further below:
Marketing Mix Modeling and Agent-Based Simulation – What Is the Question?
35
Differences Marketing Mix Modeling Agent-Based Modeling
Approach Top-down regression Bottom-up simulation
Scope One brand Multiple brands, multiple
segments
Outputs One metric Multiple metrics
Forecasts Deterministic Probabilistic
Inputs Quantitative time series Quantitative or qualitative info
Marketing mix modeling is well-suited to answer questions related to the optimization of paid
marketing investments. Agent-based simulation provides the capability to answer a broader set
of strategic questions about resource allocation.
Marketing Mix and Simulation Work Well Together We have seen a number of cases where a decision maker has both a marketing mix model and an
agent-based simulation in their toolkit for making decisions. In this context, simulation is used to
extend marketing mix to answer more business questions. A typical process is described below.
The marketing mix model is calibrated to specify the impact of the different marketing channels
a brand uses. The coefficients of the marketing mix model are then used to guide the selection of
parameters for the agent-based simulation. The simulation is tuned to replicate a marketing mix
model’s response curves and marketing attribution results, while also calibrating to sales and
other KPIs. At this point the simulation is calibrated to multiple objectives and also incorporates
insights on consumers from research or other data.
The mix model or simulation may now be used to answer questions around marketing
investment and mix. The simulation may be applied to address a range of questions. Because a
simulation incorporates multiple KPIs, segments, attributes, competitors, and probabilistic
outcomes it may extend mix to do the following:
36
• Interdependent KPIs: Consider trade-offs between sales, perceptions, consideration, and
word-of-mouth.
• Consumer segments: Develop a segment targeting strategy.
• Attributes driving choice: Test creative messaging.
• Competing alternatives: Respond to competitive actions and/or optimize a portfolio.
• Probabilistic outcomes: Assess risk based on a distribution of likely outcomes.
In summary, marketing mix modeling is an approach for performing optimization of paid
marketing investments for one brand in a stable environment. Simulation explores a broader
range of strategic options that consider the competitive market and consumer behavior. So, the
question of how marketing mix compares to simulation always comes back to what questions you
need to answer as a decision-maker.