We have the data. We have the pretty reports. But we do not have an educated team who can take the resulting statistics and turn them into meaningful plans. The process of creating Web strategy and executing on Web tactics has gone missing.
Besides being fun to say and a legitimate excuse for me to wear my "Mind The Gap" hat purchased the last time I was caught in the rain in Covent Garden, the “Feedback Loop Gap” is the most pernicious problem faced by those in large companies tasked with running large, sophisticated Web sites.
I don't for a minute discount the troubles and tribulations of those plagued by shrinking budgets, incredulous senior executives, or IT departments facing mergers and acquisitions.
I interviewed more than 50 Web executives for my book Web Metrics: Proven Methods for Measuring Web Site Success (John Wiley & Sons, 2002). From those interviews, I mapped out the process of how a large firm figures out whether their Web site is doing well, which changes are for the better, and which should be terminated with prejudice.
It all begins the day a business person stumbles upon a Web server log.
Your Web server generates a great deal of information. It creates logs. Mountains of data are collected in your server logs every day. The assumption is that this information is generated to help you determine how well you're doing.
Nice thought, but not the case.
Log files are merely the result of good, solid engineering. If you create a system that does something, it's a good idea to record what happened. That way you've got something to look at when things go wrong.
Log files are, therefore, the results of a Web server doing its job, and not a formal effort to capture valuable business intelligence.
The business management side of large companies went to the Web-heads and asked for reports. The log files were available and delivered. Ever since, mankind has been trying to interpret log files like astrologers, phrenologists, and readers of the I-Ching.
Log files do, indeed, contain more data than, say, the configuration of the bumps on your head. But it's a matter of torturing usable information out of them.
Here then, is the log file torture process practiced by multitudinous multinational corporations:
1. Capture Log Data
Everybody's happy to lasso the data that Web servers spit out on a daily basis. It's just a matter of file storage--rather mundane, actually. And it gives everybody hope.
2. Cleanse the Data
Some data are useful, some are not. Some are informative, some are confusing.
It usually requires a couple of rounds of attempted comprehension before the dirty data become obvious and can be expunged.
3. Report
Nice, crisp, colorful charts and graphs adorn the walls of technical and marketing offices. "Look! We have an active, interactive Web site and people are doing things there!"
These charts are pretty to look at. They are proof that people are paying attention.
They are useless.
4. Analyze
At some point, one of the technical committees squints at a series of reports over a several months period and begins to wonder why some numbers are going up, some are going down, while others do not change in the slightest.
They wonder what external forces are at work. They scratch their heads and come up with seventeen technical explanations for the change over time of the appearance of the pie charts, bar charts and diagrams.
5. Deliver
Convinced that they have discovered something important, the technical team calls a meeting with their business compatriots and presents the findings. They expound on the statistics and the resulting graphical depictions in terms of how many people showed up when, how often, for how long, and to what end.
The marketing staffers are thrilled to receive actual data--hard evidence that the work they've been doing has had an effect.
The end.
Notice something missing?
Several things, in fact?
Here's what I looked for and found only in the rarest organizations:
6. Interpret
Technical people and business people gather enough knowledge about Web statistics to actually understand what the charts, graphs and diagrams really mean.
7. Plan
Based upon a thorough understanding of the reported results, a team made up of information system and business people map out the changes necessary to improve usability, increase customer satisfaction, and boost revenue.
Armed with an understanding of site visitors' desires and frustrations, new designs, new content, and new applications are devised and developed.
8. Execute
Select the best options which will result in the greatest impact with the lowest investment in time or money. Secure upper management support. Create a timeline. Instill ownership. Charge ahead.
9. Repeat
Capture, measure, and analyze the results of the changes and incorporate those results in the next planning cycle. Ad infinitum.
The problem facing most large companies today falls between items 5 and 6: The dreaded Feedback Loop Gap.
We have the data, we have the pretty reports.
But we do not have an educated team made up of technical and business people who can take the resulting statistics and turn them into meaningful plans. Feeding Web server statistics, customer satisfaction surveys, sales data, etc. back into the process of creating Web strategy and executing on Web tactics has gone missing.
The technical side doesn't have a view into of the company's business drivers and the business people don't know a) what the data mean, nor b) what valuable information might additionally be available.
We need to sufficiently educate those controlling the marketing and customer service budgets so that they can interpret the data they get.
At best, they will understand the metrics reports well enough to make well-informed plans for the next step *and* instrument the execution of those plans so that the right information is gathered with each iteration: The Feedback Loop.
We can only hope. In the meantime: Mind the Gap.
