Pretty Little Lies
- AUTHOR Dan Appleman
- December 18, 2012
- No Comments
One of my first tasks as a junior programmer was to write some software to test hard drives – our company was choosing between two companies and wanted to know which one made the better drive. We’ll call them drive A and drive B.
I wrote some diagnostic software that would run tests and measure the performance of each drive. I was quite proud of the job I’d done, and was confident that the tests would provide good results.
My manager, however, was not happy with the results. My diagnostic displays were very simple and straightforward. So he spent a lot of money to hire a consultant who rewrote my diagnostics and created a very fancy user interface that constantly updated itself as it ran.
Upon running his diagnostics, he discovered that drive A was about twice as fast as drive B, and that is the one the company chose.
Later, when I looked at the results, I noticed something interesting. In those days the display console was connected to the computer through a slow serial port (we don’t use those any more). Those could run at different speeds. The console connection on the computer that was used to test drive A was about twice as fast as the one on drive B. In other words, as it turned out, the diagnostic software was not measuring the speed of the drives at all – because the display was constantly being updated, it was actually measuring the speed of the connection from the console display to the computer – the time it took to update the display.
When I ran my original diagnostic software (that did not suffer from this problem), I found that in fact, both drives performed equally well.
What does this have to do with marketing analytics?
The moral of this story is obvious. Pretty graphs and charts are just that – pretty. What really matters are the numbers used to generate the charts. Are they accurate? Are they measuring what you think they are measuring? Are they true? Are they complete?
Analytics, whether they are marketing analytics or evaluating hard drives, are only as good as the underlying data.