ON CORRELATION AND CAUSALITY

Data scientists are always on the hunt for patterns in data, particularly where there is evidence of causality, for example showing that factor A causes outcome B. Sometimes this is simple: we all know that higher temperatures in the summer cause higher ice cream sales, and lower temperatures in winter cause more colds. However, correlation does not infer causality. While there may be correlation where causality exists, often there is correlation but no causality. Common sense is the first filter: is causality likely (or just co-incidental correlation) and, if the answer is “yes”, what is the likely direction (possibly bi-direction) of the causal effect? The next thing is to consider why there is causality. It is worth thinking about the following:

In 2011 Google released a very useful free tool called Google Correlate. This tool allowed users to enter a search term to discover highest correlating search terms over time — terms with a search pattern that most closely matched the search pattern of the entered term. Often this threw up fascinating correlations, many co-incidental and unlikely to be causal, but also others that were not immediately intuitive but, on consideration, could be causal. These were gold dust for further exploration. The tool also allowed users to draw their own pattern to discover search terms that most closely matched that pattern. This could be used to find out rising and falling trends. Extremely useful. Unfortunately Google decided to close Google Correlate in December 2019 due to “low usage”. Given its high value to data scientists, and the many potential beneficiaries around the world, there is good argument to ask Google to bring it back.

There are various ways to determine causality. One way is to use statistical modelling, for example econometrics or Bayesian modelling, to mathematically determine likely causality. Another way is to use A/B Testing, a controlled experiment that is well suited to the digital environment to determine, through iterative testing, which variants of an ad or web page have the greatest effect. A/B testing can be automated in a closed loop, using machine learned algorithms, and can steadily improve outcomes (often on a diminishing return curve).
— — — — — — — — — — — — — — — — — — — — — — —

Related posts:

Handling time series data can be a bit tricky. When I first had to deal with time-series data in Python and put them into charts, I was really frustrated. I probably spent a whole day just trying to…