The importance of calibration in Marketing Mix Modelling

What is calibration in Marketing Mix Modelling?

Calibration in Marketing Mix Modelling (“MMM”) is a crucial step that involves fine-tuning the model to ensure its predictions align closely with the incrementality and/or lift estimates derived from controlled experiments. This process is essential to enhance the model’s reliability and applicability to the real-world, making it more effective in allocating the impact of various marketing activities on sales. Calibration adjusts the model parameters based on historical data and experiments, helping to account for vast uncertainty and/or discrepancies that may arise from incomplete or noisy data. 

By calibrating your MMM, you can ensure that the MMM accurately reflects real-world scenarios, thus providing more dependable and casual insights for decision-making.

Using experiments to inform Marketing Mix Modelling

Incrementality and lift tests measure the additional effect that a marketing action has on sales compared to a baseline scenario without that action. These tests help determine the true impact of marketing activities by isolating the effect of individual campaigns or channels. Incrementality tests, often conducted in the form of a geo-experiment, compares the performance of a group exposed to a marketing action with a control group that is not. This comparison helps establish a causal relationship between the marketing activity and observed changes in sales. By incorporating the results of these tests into MMM, you can better understand the causal impact of your marketing efforts and account for the unique complexities associated with your business, leading to more informed budget allocation and strategy decisions.

Using Marketing Mix Modelling to inform experimentation strategy

In addition to using experimentation to inform MMM, MMM can be used to suggest which marketing channels or campaign types should be tested further. These are just two subsections of what is considered triangulation in Marketing. MMM identifies the channels and campaigns that have the most significant impact on sales and (in the case of Bayesian MMM) their associated uncertainty. These insights help pinpoint which areas of your media mix are likely to yield the highest returns and how confident we can be with these insights.

For example, if MMM suggests that your paid social channel drives high sales, and is relatively low down on the response curve, you may be inclined to invest more into that channel as the MMM suggests room to grow. However, if the MMM reports an average ROAS of 8, with lower and upper estimates of 1 and 15, you might decide to conduct a detailed incrementality test in which budgets are only increased in one region and the additional impact on sales is observed. By doing this you’ll gain greater confidence in what to expect if you were to increase budgets across all regions and significantly reduce the chance of wasting any additional budget on an ineffective channel. This targeted approach ensures that marketing resources are invested in the most promising areas, maximising ROI.

How does calibration work?

Calibration through the prior distributions

Bayesian Marketing Mix Modelling introduces a probabilistic framework that allows for the incorporation of prior knowledge and the quantification of uncertainty in model estimates. Calibration in Bayesian MMM involves the fine-tuning of priors, which are essential components that influence the posterior distributions of the model parameters.

Priors represent the initial beliefs about the parameters before observing any historical data. These beliefs can be informed by industry benchmarks, expert opinions, or previous studies. For instance, a non-negative prior distribution might be used to enforce the assumption that advertising cannot have a negative impact on sales. 

The calibration process involves iterative adjustments to the parameters of the prior distribution to improve the model’s generalisability. This can be done by comparing the model’s predictions to actual observed outcomes from a controlled experiment and adjusting the priors to reduce discrepancies. For example, if the initial priors lead to overestimating the impact of a particular channel, the mean of your prior distribution can be adjusted to reflect a lower expected impact.

After adjusting the priors, the calibrated model is evaluated using techniques like cross-validation, posterior predictive checks, and comparing the model’s predictions with holdout samples. This evaluation ensures that the priors are set in a way that enhances the model’s ability to generalise to new data.

Calibration through the likelihood

In addition to incorporating prior knowledge, Bayesian MMM produces insights through a combination of the prior and the likelihood. The likelihood simply describes the probability of observing the data given our priors. 

Calibration through the likelihood function allows for the adjustment of parameter influence over time by adding additional lift test observations into the likelihood. This method ensures that the model remains flexible and responsive to changes in the data. The main benefit of this approach to calibration is that the parameter adjustment isn’t fixed throughout the entire data set and it is updated at the time of the lift test, enabling more realistic and dynamic modelling. However, it is important to note that this approach does not always reduce uncertainty, as the variability inherent in the data can still persist.

In the context of Bayesian hierarchical modelling, in which numerous channels are associated with an “overall” hyperparameter, calibration can be particularly beneficial. The most common hierarchical structure in MMM involves parameter means. For instance, you may choose to model the performance of Meta ads, Pinterest ads and LinkedIn ads, with individual parameters that share an overall parameter called the “paid social mean”, capturing both the individual channel impacts and their collective influence on overall sales. 

When modelling with a hierarchical structure, a single lift test can inform multiple channels, reducing the need for extensive testing across all channels and minimising disruption to business as usual. In this case, one incrementality test for Meta might influence the contribution of both Pinterest and Linkedin. This approach leverages the hierarchical structure to share information across different levels, improving overall model performance and reducing the operational impact of testing.

Why is it important?

Calibration in MMM is crucial for ensuring the reliability and applicability of the model’s predictions. By fine-tuning the model’s parameters to align more closely with observed data from lift and incrementality tests, calibration enables the model to better capture any unique complexities associated with your business. This process ensures that the MMM can better predict the true effects of different marketing activities, making the insights derived from the model more actionable and trustworthy. Without proper calibration, the model’s outputs could be significantly off, leading to poor decision-making and suboptimal allocation of marketing resources.

Introduces causality

Calibration introduces causality into MMM. By incorporating lift tests and incrementality tests into the calibration process, you can more accurately isolate the effects of individual marketing activities and establish a clearer causal relationship between these activities and sales outcomes. This approach allows for the differentiation between correlation and causation, providing more robust insights into which marketing tactics are genuinely driving sales. Consequently, you can make more informed decisions about where to invest their marketing budgets, optimising your media mix for maximum impact.

Reduces uncertainty

Calibration also plays a vital role in reducing the uncertainty associated with marketing effectiveness and enhancing the generalisability of MMM. By continuously adjusting the model to reflect new data, calibration ensures that the model remains relevant and accurate over time. This adaptability helps to account for external factors and fluctuations, reducing the margin of error in the model’s predictions

As previously mentioned, calibration in hierarchical modelling can inform multiple channels. As a result, we might find that the results of one incrementality or lift test not only decrease the uncertainty of one particular but any channels associated with the same hyperparameter. Again, this reduces the need for extensive testing and disruptions to business operations. This shared learning across channels enhances the model’s overall robustness and applicability, making it a more powerful tool for strategic decision-making.

As we wrap up this exploration into the importance of calibration in MMM, it’s clear that incorporating insights derived from incrementality tests in your model is not just a technical necessity, but a strategic advantage. Incorporating causality into your model via incrementality tests and lift tests transforms your MMM into a robust mechanism for understanding true cause and effect. By embracing the principles and practices of calibration in MMM, you position your marketing efforts to achieve greater accuracy, accountability, and effectiveness.