Pretty Little Lies

The following is a true story. At first glance, it may seem to have nothing to do with marketing analytics, but bear with me.

One of my first tasks as a junior programmer was to write some software to test hard drives – our company was choosing between two companies and wanted to know which one made the better drive. We’ll call them drive A and drive B.

I wrote some diagnostic software that would run tests and measure the performance of each drive. I was quite proud of the job I’d done, and was confident that the tests would provide good results.

My manager, however, was not happy with the results. My diagnostic displays were very simple and straightforward. So he spent a lot of money to hire a consultant who rewrote my diagnostics and created a very fancy user interface that constantly updated itself as it ran.

Upon running his diagnostics, he discovered that drive A was about twice as fast as drive B, and that is the one the company chose.

Later, when I looked at the results, I noticed something interesting. In those days the display console was connected to the computer through a slow serial port (we don’t use those any more). Those could run at different speeds. The console connection on the computer that was used to test drive A was about twice as fast as the one on drive B. In other words, as it turned out, the diagnostic software was not measuring the speed of the drives at all – because the display was constantly being updated, it was actually measuring the speed of the connection from the console display to the computer – the time it took to update the display.

When I ran my original diagnostic software (that did not suffer from this problem), I found that in fact, both drives performed equally well.

What does this have to do with marketing analytics?

The moral of this story is obvious. Pretty graphs and charts are just that – pretty. What really matters are the numbers used to generate the charts. Are they accurate? Are they measuring what you think they are measuring? Are they true? Are they complete?

Analytics, whether they are marketing analytics or evaluating hard drives, are only as good as the underlying data.

 

Dan Appleman

About Dan Appleman

As Full Circle Insights’ CTO, Dan Appleman brings a broad technology experience to our customers. In addition to having supported over 30 Salesforce.com implementations with technology solutions, Dan is also the author of the book "Advanced Apex Programming for Salesforce.com and Force.com" and has been a speaker at Dreamforce since 2012. Previously, he was the founder of Desaware, Inc. a developer of add-on products for Microsoft Visual Studio, a co-founder of Apress publishing, and the author of numerous books and articles.