Analytics, Damn Liars & Don Quixote: Telling the Real Story with the Right Data04/28/2014
It's ridiculously easy to be misled by analytics reports whose metrics don't normalize for the right set of variables. Poorly-considered metrics that don't tell the whole story followed by analysis which is hamstrung by such myopic scope, summarized into neat little narratives that are self-consistent, convincing, and wrong. This little crime against reason doesn't exist in a bubble; real people will make real, consequential decisions based on it, sometimes threatening to impact our daily lives.
Take the debate in the U.S. on Net Neutrality for example; ISPs want capacity build-outs subsidized by additional costs to both the supply side (e.g., Netflix and similar high-bandwidth services) and the demand side (e.g., broadband service subscribers), claiming that the growing popularity of services like Netflix are causing the majority of QoS issues (network congestion and all-around slower service).
This study -- merely by observing the distinction between "data consumption" and "bandwidth usage" (no, they're not the same thing!) -- shows that assumption to be false. Whether the FCC realizes this yet or not remains to be seen, but for now it looks like they're throwing in the towel on the subject.
As marketing consultants, this underscores for us the importance of carefully considering the real-world accuracy of the picture painted by the metrics on which we choose to report, less we send our company, our clients, their entire industry and its regulatory bodies on an expensive, dog-wagging goose-chase to give up more of our rights as consumers in the name of a mythological byproduct of short-sighted, lazy or disingenuous business analysis.
Remember always when crafting a metrics report to take a moment to stop, zoom way out, and check your base assumptions. If you know the data display could be lying as-is, find a way to cancel out that possibility and check again until you get it right. If you just don't have time, then make sure your analysis includes the proper caveats by acknowledging known-unknowns that may affect the accuracy of the narrative.
In this increasingly-complex marketing ecosphere, the fewer windmills we can send people tilted off towards, the better for everyone.