From TheFutureOf (18 Sep 08): Responding to Debbie Pascoe’s 16 Aug 08 11:22am comment

Debbie Pascoe wrote:

Regarding “calibration”, let me back up and briefly describe the implementation. As you point out, it is a javascript on the page. The javascript has all the instructions regarding what bits of data to collect and pass to the vendor’s server. When a visitor comes, a call goes out to the vendor’s server and this is how the data is transmitted.

Now, if the owner of the site doesn’t tag all the pages it intended to tag (this happens all the time), they will get no data from those pages, and will have an incorrect view of what happened during the visit. If the javascript is present but does not function properly (this happens a lot), then again they get no data from the page. If the javascript is present, and functioning but the information in the variables is incorrect (this happens – human error can and does creep in), then the data will be incorrect and the assumptions people make when studying the data will be wrong.

What I mean by “calibration” is automatically scanning the site and uncovering these conditions so they can be corrected. This is not a core competency of the WA vendors, and not something, IMO they should undertake. (Ref. link to my most current post below). It is a complex problem, and getting more complex as websites get larger, more dynamic, using ever more complex technologies – ex. creating all Flash modules and deploying tags inside the module.

On a related note, Michael Wexler just posted a significant article titled “What Web Analytics is Missing” – – to which I responded.

My Response:

Yes, owners not tagging all the pages desired, intended or necessary. We share that evil, I guess.

I didn’t know so many possible errors could creep in as far as javascript goes. It makes me happy that one of our original design decisions to was make things as simple as possible on the client’s side and that all the heavy lifting would be done on our side. Our side we can control pretty well. The client? Not so much so.

Thank you for describing what is meant by calibration (”automatically scanning the site and uncovering these conditions so they can be corrected”). This is something our technology does during the running of reports (uncovering invalidating conditions).

Calibration should not be a core competency of WA vendors? Yet WA vendors should partner with people who have that as a core competency. Hmm…

So if they should partner with such individuals, they must in some part be assuming responsibility for proper calibration, correct?

I read your Response to Michael Wexler’s Post re:What Web Analytics is Missing and Michael Wexler’s What Web Analytics is Missing… (with any luck I left a comment there). I asked some NextStagers about this. The discussion was very interesting. The core issue is something we’ve seen repeatedly in labs; people making first order estimates based on inadequate data because they failed to figure out the basic parameters of their experimental systems.

After a lot of conversation it pretty much came down to “Yes, calibration should be something offered by vendors” and there was a caveat that plays to your partnering theory, that such a service should be included in the fee structure.

The fact that this topic has risen to the “conversation topic” level is an indication that it happens enough to be recognized in the web analytics community. Of course, one of the joys of our society is that we think contracts assign and absolve responsibilities (except probably with very large clients).

Lucky us, huh? Sometime when we’re together, remind me to tell you what one of our early employees, a fellow from Australia, thinks about contracts in the US.

The amusing piece of all this is (to me) that (I thought) web analytics was all about accountability.

Glad you enjoyed the thumb comment. I only wish it weren’t true so often.