When spending significant amount of money on TV (or, more generally, above the line advertisement) the ROI calculation is the fist task that goes to analytics or data science unit within marketing structure. The question to answer is quite hard though: there is no tracking, no digital tools can be applied. Even using things like vanity URLs (which are often disliked by marketers – as it’s better to advertise purely your brand, not some one-off URL), the data is quite not reliable. The only suiable approach is one based on a what-if scenario: what would have happened if we did not spend at all? Having such a baseline (of course with appropriate distribution around it to cope with error measurement) will enable to calculate the difference between actuals and the ‘what-if’ scenario. This article describes briefly how to use state space models encapsulated in a brand new R package ‘casualImpact’ to achieve this goal.

Bayesian inference in such scenario is particularly convinient due to two reasons: enables inference in a pure probabilistic way, embracing the projection errors and outcoms as distributions. Moreover, sampling is the only practical way to deal with state-space models, due to presence of latent variables. We can write the basic model in two equations:

\(

y_{t} = Z_{t}\alpha_{t} + \varepsilon_{t}

\)

\(

\alpha_{t} = T_{t}\alpha_{t-1} + R_{t}\eta_{t},

\)

where \( \varepsilon_{t} \sim N(0,\sigma^{2}) \) and \( \eta_{t} \sim N(0,Q_{t}) \). The first equation is called the “observation equation”, whereas the second is called “state equation”. The model can be easily collapsed to static linear regression, and also extended to cover for linear trends, exogenous variables or seasonality. For details, see original paper: Brodersen et al., Annals of Applied Statistics (in press).

What we would like to obtain from the model is a predictive distribution of the time series of interest (sales? page views? subscriptions? likes?), and having it to be able to calculate the posterior distribution of the effect size. Fortunately, a brand new package developed at Google called causalImpact does exactly these things.

Let us assume, that we launched a TV campaign a month ago in the UK, while operating also on different markets, like Portugal, Germany and Switzerland. For the past year, we have not differentiated these countries too much. This is an important assumption, as we need to be able to have something similar to a control group in an A/B test (although doing A/B tests on above the line advertisement is of course not possible). Let us assume, that the data represent sales in thousands Euro from 1st of June 2014 till the end of August and look as follows:

1 2 3 4 5 6 7 8 |
> head(data) UK Portugal Germany Swizerland 2014-06-01 102.6280 41.66452 123.3561 140.7721 2014-06-02 103.4969 43.71173 122.0195 140.3073 2014-06-03 102.6721 43.79845 123.0858 139.2176 2014-06-04 104.5191 44.48944 122.6763 139.3611 2014-06-05 103.2314 45.74736 122.6879 140.0325 2014-06-06 102.9227 44.63914 122.2020 141.1033 |

We have launched a big TV campaign at the beginning of August (let’s say during first two weeks) and would like to know the direct impact.

We need to let R know where the event had happened by dividing the time axis into two periods: before the campaign and after. We do it by specifying first and last day of each period:

1 2 |
pre.period <- c(1, 61) # June and July post.period <- c(62, 92) # August |

Now we can perform the Bayesian estimation and prediction in just 2 lines of code:

1 2 |
library(CausalImpact) impact <- CausalImpact(data, pre.period, post.period, model.args=list(niter = 1e4)) |

In the background, we perform Gibbs sampling (as conditionally cojugate priors exist in the class of state space models) and produce a Markov Chain of length 10 000. The default number if MCMC iterations is 1000, however personally from my experience the longer chain the better (as reaching the stationary distribution is asymptotic), within the reasonable time of computation. In this place we can also have some limited control over prior. However if needed, package code is accessible and readible so can be adjusted there.

Now the key part: let us analyze the results! First, let’s plot the default output:

1 2 3 |
pl <- plot(impact) + scale_x_discrete(breaks=c(1, 31, 61, 92), labels=row.names(data)[c 1="31," 2="61," 3="92)" language="(1,"][/c]) plot(pl) |

I have to admit, that this is quite good looking and clear. The forst graph represents the model fit and the what-if forecast (our baseline) with dotted line. The solid one is the actual data, while the blue region represents the 95% credibility interval (NOT confidence – there is a subtle difference between these two).

The second plot shows the difference between the actuals and fitted values. The bold horizontal line represents zero. As we can see, before campaign the difference oscillated around zero (which is good – we want our model describing data well) and after it ‘jumped’ for a while up. We can tell a good story here: at the beginning of our campaign we could see it in action, however when the time was passing we can clearly see sales in UK going back to ‘ususal’ patterns. Analysis of this plot, together with the calendar of media events can give valuable insight into media buying strategy for the future.

The next graph, most bottom, gives the final uplift answer. It represents the cumulative of effect additions from the start of the campaign. We can clearly say, that the posterior expectation of the campaign generated profit is about 40 [kEUR]

It is worth to have a deeper look at the results:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
> summary(impact) Posterior inference {CausalImpact} Average Cumulative Actual 108 3352 Prediction (s.d.) 107 (0.46) 3313 (14.23) 95% CI [106, 108] [3285, 3341] Absolute effect (s.d.) 1.3 (0.46) 39.4 (14.23) 95% CI [0.36, 2.2] [11.12, 67.4] Relative effect (s.d.) 1.2% (0.43%) 1.2% (0.43%) 95% CI [0.34%, 2%] [0.34%, 2%] Posterior tail-area probability p: 0.00378 Posterior prob. of a causal effect: 99.62226% |

We can read some key information here, like:

- there is 99.62226% probability that there was an effect at all
- the final effect estimate is 39.4, with 95% credibility interval of [11.12, 67.4]
- relatively to daily levels, the effect was 1.2% on average (however we know that this average has little sense, as the effect occured only at the beginning of trial)

Moreover, we can even generate a verbal report!

1 2 3 4 5 6 7 8 9 10 11 12 13 |
> summary(impact, "report") Analysis report {CausalImpact} During the post-intervention period, the response variable had an average value of approx. 108. By contrast, in the absence of an intervention, we would have expected an average response of 107. The 95% interval of this counterfactual prediction is [106, 108]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is 1 with a 95% interval of [0, 2]. For a discussion of the significance of this effect, see below. Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 3.35K. By contrast, had the intervention not taken place, we would have expected a sum of 3.31K. The 95% interval of this prediction is [3.28K, 3.34K]. The above results are given in terms of absolute numbers. In relative terms, the response variable showed an increase of +1%. The 95% interval of this percentage is [+0%, +2%]. This means that the positive effect observed during the intervention period is statistically significant and unlikely to be due to random fluctuations. It should be noted, however, that the question of whether this increase also bears substantive significance can only be answered by comparing the absolute effect (1) to the original goal of the underlying intervention. The probability of obtaining this effect by chance is very small (Bayesian tail-area probability p = 0.004). This means the causal effect can be considered statistically significant. |

Having such information, is essential to future budget planning, media optimization and understanding the responses to creatives.