MarketingDistillery.com is about web analytics, data science and marketing strategy

This article takes only 4 minutes to read

Measure Your Performance

It all comes down to EBIDTA/PMC. No matter what, you will always be challenged on that. If you are positive it’s already good. However, sooner or later, there will always appear the question: how are we doing versus last year? And here we are in the hell for a statistician: you’re not asked for incremental value, trend breaking, spikes. But for all those nasty things: YoY, YTD etc. And what then? This article aims at giving you a handful of excuses why & how you should convince your CxO to make you do your job in the way you believe it should like, not in the path followed by accounting.

But lest start from answering the basic question: where does the YoY come from? This metric, in its assumptions is quite a good one. When analyzing time series, i.e. data where the order of observations matters (e.g. data associated with measurements in time) the basic thing you can do is to explain the current state by it’s past via so called autoregressive models. We can write one like this:

$$y_{t} = \sum_{i=1}^{p}\alpha_{i}y_{t-i} + \varepsilon_{t}$$

For p=1 it means, that the best forecast for current time period is the value of last one (e.g. for today yesterday, for this year the last year – provided you have figured out your seasonality right!). In simple language: if we make daily about 200 sales, the best forecast for the next day is.. 200 sales. Does it work? Yes it is, in forecasting growth rates on exchange rates no model without exogenous variables has beaten this simplistic view!

Going back to marketing: If you run a marketing budget which generated $10m PMC last year, $9m 2 years ago you have it at the back of your head that it is reasonable to think that this year’s PMC should oscillate around $10m – provided no internal/external shocks appeared (e.g. internal: you’ve introduced a new super cool product; external: a new competitor appeared and sells your stuff for half the price). And you’re right! Your gut feeling is so true – logically thinking: if nothing changed, why my PMC should change?

To sum up: USE YoY/WoW/DoD metrics if:

  • Your data are not exposed to seasonality or you don’t have much observations, e.g. how many years back can you have your annual PMC? 10? 15?
  • You made sure your data are of a constant frequency and you explain latest time period with one period earlier and no major changes occur in your inside or outside environment.

Wait – this article should convince me NOT to use YoY metrics! Well, now you know when to use it. If your case fills the rules above – go for it. If not.. Let’s analyze at first some most common mistakes which are violations of assumptions above.

  1. Let’s violate the first assumption. Imagine a simple scenario: you are looking at weekly sales and you work in travel industry. You come to work, calculate your report and see the picture below [fig01]. THIS IS IT!! We have a skyrocket, you jump to your boss and ask for a raise? No! You double check whether this is not a part of the global trend, i.e. the seasonal waving. Like in travel industry, it might raise in the summer, fall on winter (or the opposite, if you’re in ski business or in southern hemisphere). In this example, figure 1 does not make any sense if we zoom out and look at figure 2 [fig02]. What are we showing? We did good or bad? You can’t see answer to this.

    fig01

  2. Daily/weekly YoY. Sounds a bit contradictory, right? How can you make DAILY/WEEKLY data YEAR on YEAR?! Yes you can and unfortunately many big companies make this mistake without single thought. So the temptation is: look as this week’s data, have a look at the week 52 weeks ago, take a difference in percentage and display the number. And here’s the biggest question: do you exactly remember how your business looked like a year ago during this week? Do you control you environment & products in a manner you can be certain about your silent assumption that “nothing has changed”? Have you taken into account e.g. floating holidays and don’t compare this weeks versus last year’s Easter?

Both of them together. Take your dashboard and answer yourself: isn’t it by chance showing yesterday vs. the day 364 days ago in a seasonal industry? If so, what quality will have the decision you’ll make basing on this?

Additional assumption – the logic – and it’s violation. Before calculating a metric, maybe we should first have this thought whether it makes sense at all? Let me give you a real life example from my career. When reporting marketing activity, you are usually keen to see its efficiency, i.e. from a unitary spend, how much we made profit? No matter how sophisticated is your incrementality measure, at the highest level of management you’ll end up in ROI calculation: 100*([your profit]/[your cost]-1). Great. And now the most evil thing. Make a week on week (or year on year) comparison. So last week we had 20% ROI, this week we have 30%. 50% efficiency growth? Not exactly. We are not allowed to do so. Never. For a simple reason. What if we have broken even the week before? Infinite efficiency growth? No. Always make sure your metrics make the very basic sense: do not divide by something that can be both positive or negative, don’t add apples to oranges etc. So now you know that there might be some challenges in measuring your performance. What to do about it?

THE SOLUTION

Modeling! Even a simple model can save the day. Even the simplest is better than what we’ve seen before. Linear regression? Why not! Take you sales, add a trend, add a seasonal variable (e.g. use Google trends result for a basket of 1-5 keywords closest to your business) and you can answer in a blink of an eye whether we are good or not. Is the trend up or down? Are there any weeks where we significantly outperform or go down? If so, why & when? Ask yourself this question and dig deeper. Find the solution, point the cause. This brings a real value to your business. Want to find out more how to do it in R? Hold on to your hat, next article is coming!

Leave a Reply

You must be logged in to post a comment.