The Unfulfilled Promise of Online Analytics, Part 1

by Joseph Carrabis on November 12th, 2009

Man is the symbol using animal,
Inventor of the negative, separated from his natural condition
By instruments of his own making
Goaded by the spirit of Hierarchy,
With knowledge of his own mortality
And rotten with perfection.
- Kenneth Burke

I've had this theory that thermodynamic principles could be used to predict user attitudes and behaviors in finite populations for a while. A population threshold has to be reached before accuracy could be achieved. One prediction of the theory is that once that population threshold has been reached, the largest segment of that user population will be unsatisfied users for any given product or service. You don't need to sample the entire population or even the threshold. Another fallout from the theory is that you can create an exemplar group, study that, and make extremely accurate predictions about the entire population (including segments of the population not represented by the exemplar). I've been studying population dynamics for different industries for a while and had an opportunity to study the online analytics community, some results of that are shared here.

I initiated the study by sending the following (or at least a similar) request to people in the online analytics community.

would you be willing to write me your thoughts on “the unfulfilled promise of web analytics/search”? I'm preparing a column/blog post. Your response will be kept confidential (I'm keeping everybody's responses confidential).


My request was intentionally open ended (surprise!). I wanted to know their responses, not what might be predicated by any guidance on my part.

One respondent wrote, “In a survey, this question would be tagged with a 'leader bias'.” I pointed out that as this particular respondent opened their response with “The question was 'why is it that web analytics isn't delivering on its promise'?” they demonstrated that they were quite willing to follow any leader bias that may have existed. The fact that they rephrased my original request is a demonstration that the bias — hence prejudices, acceptances and beliefs — existed long before my request was made.

The question isn't whether or not leader bias exists, the question is “where were respondents willing to be lead?” and is typical of my (and NextStage's) use of the Chinese General Solicitation. Knowing what someone responds isn't as actionable as knowing how they respond (you're shocked I'd offer that, yes?).

I also didn't ask for responses from people who had previously demonstrated (via other writings, etc) they had a “company line” or brand to protect.

The title of this post was originally “The Unfulfilled Promise of Web Analytics” and came from a conversation I had in early June '09. I was talking with some folks, one of whom was an SVP of web analytics and marketing for an international marketing company. Unprompted, this individual shared their disillusion with the web analytics field and provided the title phrase.

In all I received some 60 responses, some from people “just doing their job” to others with national and in some cases international reputations to protect. The responses came from everywhere except South America, Africa and East Asia (I hope to cast a better net in the future). I kept the original spellings and grammar of respondents when I quote them (AmerEnglish is not the native language of several of them) because doing so keeps their intent clear over my own.

I've studied the “largest user group will be 'unsatisfied' users” phenomenon across industries and (so far) it holds true. No doubt I'll write a formal research paper (and include an extensive bibliography) about this phenomenon in my copious free time someday.

In the meantime, allow me to share the results specific to the online analytics industry with you.

See this tool? I must know what I'm doing because I use this tool.The quote on the right was made to me during a discussion. It was offered jokingly and I accept it as such. I also know a little about how the mind works and where such statements — even as jokes — come from.

The person making that comment went on to tell me about a recent conference they attended. At some point a bunch of attendees got into a cab to go out to dinner. One of them offered that companies wanted more and more accountability in their analytics.

The universal response was “What? Accountability? It's time to get into another business.”

Such responses are understandable and they can only be made by people at or near the top of their industry. Nobody wants to work and everybody wants to play. The more fun (play) they can have in their job the more they'll enjoy it. Being accountable isn't fun, though.

When Web Analytics came around, my first thought was 'cool, plenty of data'And such responses must be put next to “When Web Analytics came around, my first thought was 'cool, plenty of data'. Little did I know data would replace the actual business reflection that spun all of this.”

I recognized true schisms in the responses. I'll mention one here because it relates to the first quote above (I'll get to the other schisms further on). It deals with people experiencing non-conscious incongruities between their identity-core and their identity-personality (more colloquially, The Impostor Syndrome, feeling they're frauds. Personality, Identity and Core make up an individual's psychological self-concept. Different disciplines have different terms for these elements). I would have thought that such sentiment would be prevalent at the lower end of the disciplinary spectrum and it wasn't. More than one well recognized individual shared that they feel like the emperor without any clothes when challenged about their conclusions. They often want to respond as is indicated above; “See this tool? I must know what I'm doing because I use this tool.”

The schism here was more psychological and psycho-social than analytics wise. Did these individuals have confidence in their analysis? Most often, yes. Did they have confidence their analysis would be accepted/have meaning/provide value?


Sensing or believing that one's work is not honored or respected is damning. Such attitudes are psychological death to the vulnerable and emotionally uncomfortable to the strong.

“…the question of accuracy did not shake off easily. To be totally honest, I kept this to myself.”

Maybe new fields need to emerge -- web psycho-analytics perhaps?Online Analytics is a numerical discipline. That's its whole point; here are numbers that prove something. It is not a psycho-social discipline or, as one respondent wrote, “Maybe new fields need to emerge — web psycho-analytics perhaps?”

Such fields may exist and may emerge if they don't exist already. What is true about them — if they're primarily left to the current online analytics paradigm — is that they will require large numbers to demonstrate accuracy. The accurate metricizing of any social system (the internet as an information-exchange is such a system) requires threshold numbers for accuracy to be demonstrated when traditional methods are used. For example, a data space of 50,000 people within 2 days is reasonable for traditional analytics methods to prevail (use of conditional change models can shrink these numbers considerably). Typical numerical methods involving smaller populations require either longer timeframes or smaller environments to demonstrate reliable, repeatable business value.

“Web psycho-analytics” requires different numerical methods and mathematical paradigms from traditional analytics to demonstrate reliable, repeatable business value.

The above is especially true when individualization — the ability to recognize a visitor as neuro-socio-psychologically unique from all other visitors — is to occur.

...your analytics will not match the vendor's numbers. If you add two or three analytics systems, the numbers will not match each other. This creates situations where it is impossible to reconcile any data sets.However, until such methods are widely adopted clients and consultants are left with

  • conflicting numbers from different tools (“…your analytics will not match the vendor's numbers. If you add two or three analytics systems, the numbers will not match each other. This creates situations where it is impossible to reconcile any data sets.”),
  • conflicting numbers from the same tool (“Even using the same tool depending of how it is set-up it can lead to very different numbers.”),
  • tools that are difficult to use (“And those vendors said it was really, really easy! Pfff! Liars!”),
  • conflicting vendor definitions (“Vendors have different standards, meaning that what one vendor considers a visit is not the same as another vendor, thus making comparisons is often misleading.”) and
  • unachievable expectations (“Web Analytics is often sold as the thing that will improve your website results by 100-200%, well that's not true.”)

“…talk to other people about what you were trying to accomplish and beg for them to play along.”

Jim Sterne asked me what I'd learned about web analysts a while back at an eM SF. I was onstage at the time. My statements have (I believe) proved cassandric. I offered that there was discontent bordering on malcontent. There was little to no job satisfaction and advised the eM staff to start shifting their conference focus from pure WA to cross disciplinary offerings. I've also openly stated that I left the WAA because it had all the hallmarks of a society in decline (think Rome, Persia, the USSR, …, all overthrown by invaders from without or within).

This comment was made a few years back and I have no knowledge of the WAA as it exists now.

Where does this discontent among online analysts come from?

One place is unaccepted accuracy (as mentioned above). And if the accuracy is accepted, it isn't acted upon. But there are a lot of hindrances to accuracy that are beyond the analyst's control.

Tagging seems to be a major issue in this area. Tagging was originally considered a solution to the accuracy problem. But the world works in balance — especially when unnatural processes are assembled together. Tagging solved accuracy issues but required more sophistication in the collection and analysis of the data. This sophistication required the involvement of other organizational players, some of whom couldn't or wouldn't play along. The end result is that tagging — a relatively simple concept and method — still does not have an industry standard.

So long as I show that I'm doing something I'm not responsible if nothing useful gets done.The tagging problem (I believe) would go away if clients — not vendors and not consultants — were invited to find common ground in what they're looking for (more on this later). At present clients have no fixed, pervasive idea of what advantage online analytics provides. It doesn't reduce costs, reducing costs is done by rethinking processes. Instead of rethinking their internal processes companies “…have become very lazy.” When the website isn't producing what they believe it should produce the solution is to get another tool.

But there is no magic bullet. Companies who go from one tool to the next are like psychotherapy patients who stay in therapy with no desire to get well. One respondent confided “So long as I show that I'm doing something I'm not responsible if nothing useful gets done.”

The unfulfilled promise of web analytics, at the root, is because of people.Business politics can not be ignored when considering the unfulfilled promise of online analytics. “Where one person has all the authority and all the ability to change a site as they see fit: optimization actually really works. A new headline here, a picture of somebody looking into the camera there. Demand increases, everybody's happy. But, for most corporations, this is not the case” was a sentiment stated often if not as eloquently by many respondents.

“If only I had this report, as shown into the vendors slick presentations…”

All we get in the tools are simple averages and little in terms of correlation. In my opinion this leads me as an analyst to have a high degree of scepticism in the underlying data and hence difficult to really delve into hard core analysis of the data.One of the problems that came through the responses was that online analysts often demonstrate a victim mentality. This was greatly the case in web analytics and often in search. When asked directly, “If you know these problems exist in your industry why don't you take steps to solve them?” responses tended to manifest feelings and verbiage of powerlessness. The phrases “We can't do anything to solve this.”, “It's out of our hands.”, “It's beyond our control.” and “We don't have access to those people.” were repeated on both sides of the Atlantic. One respondent offered “…is it in the realm of a little web analyst within a large multinational to actually do that?”

Consulting online analysts are caught between the vendors and the clients they serve. If not a victim mentality, this “serving two masters” creates a psychology that's very close. It also ties back to Why hasn't Marketing caught on as a “Science”? and Matching Marketing and IT Mythologies about analysts and marketers finding common ground.

Consensus points abounded on these research elements. Vendors were viewed

  • as only being interested in selling licenses,
  • as promising more than could be delivered (“the space has been high jacked by vendors who promise a mountain of diamonds without much effort. This is not true.”),
  • as not offering proper or worthwhile optimization tools and methods (“…the important thing is the optimization that is done afterwards.”) or
  • as offering wolves in sheep's clothing — tools that actually produced simplistic results, could not do deep analysis and therefore produced skepticism about the underlying data.

Analytics and the web are suppose to be transparent and easy to track. However, once you start marketing you find out that is not really the case.It is likely that as more and more accountability is demanded from different organizational groups measurement efforts will merge and (perhaps) result in easier corporate buy-in. What may not go down well is that these efforts are more likely to come from marketing than from analytics. Multi-channel marketing will need to learn from online analytics if it is to have value to any business.

“There are simply not enough employees in the companies focusing on adopting web analytics in the organization.”

A challenge that falls out of the above section and the above quote is truer for web analysts than their search-based compatriots in any given organization. Web analysts have fewer champions at the top of the corporate ladders than do marketers and search (which is often not considered an analytics discipline even though the science of search has been documented elsewhere). Marketers have traditionally been closer to the top of the corporate recognition ladder than analysts could ever be. This goes back to the opening statements about work versus play; marketers play, analysts work. This is demonstrated in language if not in physical reality, and one needs to recognize that perception is reality.

Online analytics grew out of (and could very well still be mired in) IT departments. Worse, any kind of analytics smells of accountants (hence accountability), and everybody knows the accountants only come in when the business has failed or is recognizably close to. One respondent wrote that they knew their company was in trouble when the bank sent accountants in (evidently the waves of layoffs and learning they were US$20M in the hole weren't warnings enough).

Online analytics is a discipline of numbers. Whenever there's a discipline of numbers it means there's an evidentiary trail for decisions. Consider the political and psycho-economic meaning of this for a moment.

If I have the option of taking advice from someone who goes with their gut then I really can't be held accountable because there are no numbers, therefore from any evidentiary standpoint I'm pretty safe. Should things go sour it's a political issue because there was no real evidence that we should have gone pro or con, we went with our guts, flipped a coin and took what came.

Even better, it was (point finger in some general direction) their gut feeling, we went with it, it flopped, it was their gut not mine, they're out and I'm still good.

But if I go with hard numbers and my decision is in error? Now it's psycho-economic and I'm the idiot and fool because I didn't understand what I was doing. Both I and the group that helped me make the decision are forfeit.

So which is politically safer to place higher on the corporate ladder, to listen to and feel good about? But even at the top of the corporate ladder guts and numbers are in conflict; the average CMO corporate lifespan is about two years, often less.

“The tools promise a lot, and can live up to most of it.”

Most psychotherapists would look at the responses and recognize a love-hate relationship in the making if not already extant.

However, the love-hate relationship doesn't take the form most psychotherapists are familiar with. Most love-hate relationships exist between an individual and some one thing external to that individual (another person, another thing). The love-hate continuum usually takes the form of “I can't live with (it) and I can't live without (it)”.

...many larger companies buy (what used to be) expensive, full-featured web analytics packages, only to use the tip of the iceberg: the core metrics that should be obvious.This isn't the case with analysts. Most of those surveyed liked what they do and believe they add value for their efforts. They love what they do, just not who they do it for or how it is done (“It's not so much the unrealized promise of web analytics, as organizational politics leading to weak and vaguely defined goals in larger organizations.”). This creates a triangulism and triangulums are always psychologically deadly.

An example of psychological triangulism is the parent who loves their partner and recognizes their partner has an unhealthy relationship with their common child. Parent 1 is caught between protecting the child from the partner and protecting the partner from the eventual wrath of the child (think Oedipus and Electra). Their loyalties are constantly divided (as mentioned above and especially if no psychological reward manifests itself). The psychological challenge escalates until Parent 1 finds themselves developing their own animosity towards the child. They mistakenly believe if the child were not present the parent-partner relationship would be better.

The end result is that both child and parent-partner relationship suffer. Here the analyst-client relationship and the online analytics industry is suffering.

I think the promise is fulfilled for some and not for others!  The difference is the level of sophistication of the user.This tension is manifesting in the industry in the same way it manifests in the therapist's office — fingerpointing. Consider the following responses, some obviously from consultants, others from vendors, and those where the lines blur greatly:

  • “What's sure is that when it comes to Web Analytics and vendors, there's just one number that counts: quarterly sales. When it comes to clients, well they don't really know which number they're looking for but know it's damn hard & expensive to get it. Those in between, the expensive consultants, they're just trying to make a living and fight for peace on earth and accountable decision making.”
  • “Log file data and Web analytics are both sources of information. They are tools, like a hammer. A hammer in the hands of an unskilled, ignorant but self-righteous and overly confident carpenter? That is a scary thought. Well, it is equally as scary to me about Web analytics and log file data. There are plenty of unskilled, ignorant, self-righteous, and overly confident search engine optimization (SEO) professionals, Web analysts, and other marketing people. Even many search engine software engineers are not competent carpenters or architects, but they honestly believe they are. And we are buying what they have to sell.”
  • “I honestly don't think there are unfulfilled promises of web analytics. The companies are doing great and the software is progressing all the time. I love analytics!”
  • “Why is it so hard for people in the web to take actions and optimize based on what the tool reports? One of the reasons of this is that often they don't have a clue of what can be changed or they have an idea which is incorrect.”
  • “If web analytics is under-delivering in any way, it is largely because of most organizations inability to address web analytics at the strategic level rather than a tactical tool to optimize the online marketing channel.”
  • “I think the promise is fulfilled for some and not for others! The difference is the level of sophistication of the user. For some companies, even if they deploy it properly there is more volume and nuance to the data than they can properly grok.”

Is more training the answer? And if so, who do we train?

I'll restate here what I wrote in Learning to Use New Tools; the use of any tool is going to require training across the usage spectrum. The use of new tools definitely so. This training can be self-training and the user should be prepared for scraped knuckles, smashed thumbs and lots of cursing. Self-training is great when the user has lots of time and patience. Otherwise, take a class or let the experts (“consultants”) in.

Do remember Buckminster Fuller's definition — An expert is someone who can spit over a boxcar. I often tell people that the front of my shirt is soaked based on my failed efforts.

It reminds me of the development of web sites themselves ten years ago - everybody had to have one, still not being absolutely sure what to use them for.More training is the answer only if the training results in well-reasoned and understandable business actions. Tools and trainings are worthless without knowing what one wants to build (“It reminds me of the development of web sites themselves ten years ago – everybody had to have one, still not being absolutely sure what to use them for. Of course the free tools have done their part in this evolution.”).

“The unfulfilled promise of web analytics and search is measuring outcomes instead of outputs.”

Our culture (western, not analytics) has been “objective and evidence driven” for about 400 years. There has been the unstated Field of Dreams-like belief that “If you have the numbers, the truth will come”.

I believe most of the analysts surveyed would consider this a desirable yet inaccurate depiction of the real world. Their tools produce “…beautiful charts that don't tell me what to do to make things different – not better, just different. For that I have to go somewhere else.” None of the analysts surveyed wrote or talked about growth curves, forward discounting, debts, rates of depreciation, technological obsolescence, energy consumption (the company that can correctly respond to market needs faster wins because a) it responded correctly and b) it required less energy to do so).

This greatly surprised me. For all the analysts “in the room”, none talked of analysis. Several responses demonstrated a level of contempt regarding available tools (vendor agnostic) so it's possible analysis per se isn't a subject of high regard in its own community.

The above presents a discomfiting scenario. It demonstrates a severe disconnect between “what should be” and “what is”, something in keeping with C.P. Snow's two cultures yet far more pervasive (in this industry) therefore far more damaging.

If this paper focuses more on psychologies than on analytics it's because the responses dictated it so. need a team of people who know what all this is about to digest it for the more common mortals.“…analysis is a story based upon data put into context.”

The quote starting this section was very telling but not unique. No respondents believed the numbers alone proved anything, nor even when presented as part of a strategy. And few respondents seemed to be equally comfortable in boardrooms as in spreadsheets.

Yet the need for online analytics to be part of a larger picture, a grander story, was everywhere. Analysts uniformly perceive themselves as

  • not part of a unified business reporting structure,
  • not contributing to the big picture, and
  • lacking the political power or psycho-social maturity (within the organization) to sit at the grownups' table.

“And then there's this vague notion from Mr. Kaushik…”

Let me emphasize that I did not choose the exemplars noted in this paper. Respondents demonstrated exemplar recognition in conversation and written material.

Any pervasive duality will present itself in exemplars (not to be confused with my previous mention of exemplars as part of this research). Here the exemplars (or probably more accurately, “doyas”) are Avinash Kaushik and Eric Peterson with Avinash Kaushik leading the pack in references by almost three to one.

Equally interesting was that the anti-Kaushik camp's complaint wasn't necessarily against Avinash Kaushik himself, it was against his “You, too, can do this” mantra (perceived if not actual). Yet another schism appears; those who need (for whatever reason) analytics to be hard and those who need it to be easy.

Web Analytics is hardEric Peterson is well known for his “Web Analytics is hard” statement (interesting Reading Virtual Minds Vol. 1: Science and History tie in; the majority of respondents wrote web analytics or search. Very few capitalized online analytic disciplines. Most people capitalize their own discipline. It demonstrates a non-conscious recognition of the value of what they do).

This belief begs the question of whether or not something can be “hard” (meaning “difficult”) if it is properly understood. Educational Psychology, Cognitive Psychology, Sports Medicine, Kinesiology and related disciplines all demonstrate that anything done improperly is hard. Many people give up on mathematics due to poor teachers, poor curriculum, lack of discipline, … To them, math is hard. Aikido is dangerous without proper instructors present.

But is something in and of itself difficult? Only if there's a social or political reason for it to be so. Perhaps the priests wish to keep the mysteries of the divine for themselves. This provides them the opportunity to select who'll enter their ranks, who'll excel, and to whom the teachings will be “difficult”. Only one respondent offered a centralizing attitude (“I'd rather be of the school of thought that web analytics can be easier… if given time and approached in the right way.”).

Id rather be of the school of thought that web analytics can be easier... if given time and approached in the right way.Here again politics more than psycho-economics rears its head. “I will protect my (place in the) industry by making it difficult for others to succeed in that industry” hence controlling the industry itself. The problem with this ethos is that eventually a large enough (ne' “threshold”) group will occur that takes the industry in some other direction completely.

There are psychologic ramifications to both “hard” and “easy” statements. “Hard” statements set up the majority of participants to fail, or if not to fail then to prepare for failure rather than success. Likewise, the “easy” statement can cause false expectations of success to develop. What is obvious from the responses is that Avinash Kaushik owns the “actionable outcomes” space and neuro- and psycho-linguistic Towards space when it comes to online analytics as a discipline (his was the only work directly quoted in the responses; “Actionable insights and metrics are the uber-goal simply because they drive strategic differentiation and a sustainable competitive advantage.”) and Eric Peterson owns the neuro- and psycho-linguistic AwayFrom space when it comes to online analytics as a discipline.

AwayFrom and Towards are used in their neuro- and psycho-linguistic sense here to describe how people hence the industry is thinking, not necessarily how the industry is moving. See's Chris Bjorklund interviews viral marketing expert Joseph Carrabis, founder of NextStage Evolution, Part 4a) and Using Sound and Music on Websites for more on these concepts.

The exemplar messaging is polarizing an industry already divided by a great many other factors. I can say playing guitar is easy and I know I'm never going to be a Segovia or Kottke. Likewise, I recognize I could play better if I practiced more. This “centering of duality” needs to take place in the online analytics world if it is to survive, yet most respondents demonstrated extremum statements (statements with language demonstrating polarity behavior and belief) rather than centering statements (statements with language demonstrating unifying or centering behavior and belief) in their responses.

All things require some degree of practice before facility in their use is obvious. There's also the intersection of lack of correct practice and lack of understanding. This can be mixed into The Impostor Syndrome mentioned earlier (see Reading Virtual Minds Vol. 1: Science and History or I'm the Intersection of Four Statements for more on The Impostor Syndrome). Anything can be difficult if the practitioner doesn't really understand what they're doing, is acting by rote but from neither repetitive action nor repetitive practice of the correct action.

Disciplines may be represented by exemplars and responses to the exemplars are sometimes not the responses to the discipline. Respondents tended to present AwayFrom behaviors regarding Avinash Kaushik and Towards behaviors regarding Eric Peterson in their responses (noting as offered earlier than Avinash Kaushik is more in their consciousness than is Eric Peterson and with basic normalization applied).

And then there's this vague notion from Mr. Kaushik: give more insights, knowing more about what's going on within your visitors minds & hearts so that you can better service them.These presentations are understandable. Correct or not, the perception is that Avinash Kaushik wants to move the industry away from a “numbers are evidence” basis (one respondent offered “And then there's this vague notion from Mr.Kaushik: give more insights, knowing more about what's going on within your visitors minds & hearts so that you can better service them. Sure, cool, sounds great. Still scratching my head. With surveys you say? Asking them a question? Just 4 questions? Ok so when I get the answers, is this representative? Should it influence my copywriting, my product offering, my pricing scheme?”)

The concept of “knowing more about what's going on within your visitors hearts and minds” is one I and NextStage strongly encourage.

You're shocked, I know. Simply shocked.

I also encourage evidentiary — hence numbers based — decision making practices.

A curiosity of this research is that no exemplars arose on the search side of online analytics. Search respondents noted Avinash Kaushik and none of their own. This could be due to the different lifespans of search and web analytics, the different mentalities and ego structures that arise in these two disciplines or simply that no one in search demonstrates a strong enough personality for a cult-of-personality to develop around them.

“How you measure success depends on how you define success”

There are many ways to interpret the above and all of them point to a lack of standardization. I remember conversations where the definition of success was moving away from online sales to “I got their name” or “they downloaded a paper”. These conversations always intrigued me because they were examples of defining success in terms of the visitor's action, not the desired outcome of the site owner.

This is another example of non-standard definitions plaguing an industry and no one stepping up to lead the way (equally interesting, no respondents mentioned any professional organizations in their communications. This indicates online analytics professional organizations are not serving their membership enough to warrant conscious recognition). Online analytics is quite capable of comparing the numbers between “sales” and “newsletter signups” and the comparison truly is one of apples and oranges; business development versus transactional business, strategic vision versus “I went to the bank today” tactics.

And if the success definition the consultant is comfortable with, knows how to demonstrate and can defend is one the business client can no longer accepts?

“The consensus among industry leaders is that web analytics will be a different entity in five years.”

The consensus among industry leaders is that web analytics will be a different entity in five years. Its ultimate purpose is to facilitate action in support of any initiative on the web, so it also is much like plastic.Clients are asking for more … something … from their vendors. One respondent stated “Procter&Gamble is moving from 'eyeballs' to 'engagement' but leaving 'engagement' for others to define.”

This is the intersection of Jim Sterne's “how you measure success” mantra with the “gut vs numbers” statements above. The only sure winner of letting others define your success is that politics will prevail.

A recent Forrester paper indicates a move towards free analytics over for-pay analytics. The report is interesting and perhaps more interesting when viewed outside the online analytics silo.

I point out in Reading Virtual Minds Volume 1: Science and History that growth numbers can seem impressive until you recognize population dynamics, population ecology and evolutionary rescue at work. I used these and similar concepts in From TheFutureOf (13 Mar 09): The Analytics Ecology and From TheFutureOf (5 Jan 09): Omniture and Google Considered Environmentally to indicate that populations would shift, go near death then bounce back dependent entirely on the existence of (again) threshold populations (I hope readers appreciate how important the threshold population concept is in any socio-environmental dynamic).

Conclusions for Part 1

In the end, it seems the online analytics world is setting itself up to fail. It's as if an architect were to create a negative space then attempt to fill it. Analytics doesn't matter be it search or web; all business — B2C, B2B, B2whatever and whatever platform you're using — is going to come down to personal relationships, establishing them, maintaining them, personal interaction and commitment (readers who've heard or seen my “10 Must Messages” presentation will recognize those communications here).

Nothing communicated by any respondents indicated that analytics is in and of itself a worthless discipline, only that it is a misunderstood hence misguided discipline in the online world. Yes, all forms of analytics will get you to the door (and in some cases may even open the door) and in the final conclusion it will be the establishment and demonstration of trust that powers commerce, not numbers. Or at least not numbers alone. This indicates a shift

  • in what the numbers are about,
  • how they are demonstrated,
  • how to derive actionable meaning from them and
  • how accountability is framed

are in the offing.

Problems are (in my experience) pretty easy to discover. Solutions, though…

(more on possible solutions in next month's post)

RVMsmallfrontcover.jpgHave you read Reading Virtual Minds Volume I: Science and History? It's a whoppin' good read.

From Analytics, Industry, Psychologies, Tool Use, User

  1. I applaud Joseph Carrabis for writing “The Unfulfilled Promise of Online Analytics, Part 1″.

    There's been a fundamental schism in analytics since the 1930's – between 'advertising' and 'marketing', really, as far as I can tell, since Hopkins died and was forgotten.

    So when Joseph holds up a mirror to the web analytics industry of course it's going to be ugly.

    Of course you're going to see a massive, gaping, puss-filled wound running diagonally across the face.

    I'm not going to shoot the person holding the mirror. Neither should you.

    And I'm not going to personalize the debate, either. I think we might be tempted to boil this down to a difference between two wildly successive authors.

    It's more than two authors. They're just latest incarnations of that schism.

    How do we stitch the face back together?

  2. Thanks, Chris. I appreciate the nod. Part 2 — solutions — is going to require thought and help (from everyone, please). – Joseph

  3. Joseph. Thank you for your lucidity. I think, however, its safe to assume there are, in fact, now 3 camps…;)…

  4. Howdy and thanks, Daniel.
    You've got me, though. What other camp? (never assume Joseph is smart or can catch on to subtlety. I'd prefer people to appreciate my dolthood on most things. Ask Susan, she can corroborate easily and with many examples). – Joseph

    • If the primary sentiment of demarcation is somewhere along the lines of whether applied web analytics practices should be 'hard' or 'easy', by the mere suggestion of alternative thought…a suitable foundation can be found to justify any one of new directions…those which might fall somewhere closer to:

      Web Analytics is _________________ .

      Joseph's Response: Excellent use of logic. Thank you!

      Be it: applied statistics, practical mathematics, decision support, behavioral research, patterns, socio-dynamics, socio-anthropological (;) psycho-etc.*….

      Depending on where the primary process and support model is deriving its justification for governing managing protocol, neither 'camp' can appropriately employ its scope and still fit what you allude to as an existing camp of those who, for lack of a better term, DO understand and to whom will not or should not find any particular arrival at valuable information 'Hard' or 'Easy'.

      As a beckoning to the industry for the minds and individuals willing to challenge themselves or, carve out their living by doing what came naturally, the previous 'bicameral' state of thought and concept was enough to whet the appetites of the capable. Now, with the numbers feasting at the tables…your reference to the historical collapses appears inevitable (Rome/Persia…etc). My most challenging history professor left a mark on me when he said: 'With People, come politics…and ultimately, decisions made with a backward sense of consideration for the many to the benefit of a few…'

      Joseph's Response: ROFL, thanks.

      The problem, moreover, as I see it, is that the focus is on consulting and assisting as an external entity to provide a SERVICE to a company. On a scale of abstraction…this is, imo, an 8 (where 10 is planet x10). Its tough to sell analysis as a business because its tough to bottle it. Tougher still by the fact that there is an external relationship to some party which must, if willing to engage, be based on less than trust, tainted by opinion, and, skewed by fear. (This, I think, probably speaks to the 'victim' which you mentioned early). Since this is at least the perception as we've seen it (as I personally have seen it). It means that there is a solution…and its finding methods to apply a particular class and function of analysis to a technology, software, application, or system which can more readily be measured and structured around the goals of the client/user to meet their scalability needs and feed their understanding of specific subsets of the 'analytics' fields, with relation to online 'ecosystems'.

      Joseph's Response: I agree with this as I understand it. As pointed out earlier, my understanding can never be guaranteed to be accurate.

      Thusly, the 'NextCamp' might be a school of thought in applied web analytics with its place in solutions, from a general sense but with the flexibility to meet the challenges of businesses from diverse objectives, and with a structured system of foundational principles derived from scientific and mathematical fields, and less the liberal arts of communication or schmoozing…this addition would then provide a more constructive, productive, evolving group of both of the previous while co-existing mindsets, while offering a more contributing and proactive community to apply to each…

      Joseph's Response: I agree with this and present a challenge at the same time; NeuroScience, NeuroEconomics, Educational Psychology and related disciplines are constantly closing in on adding “science and mathematics” to “communication or schmoozing”. Even the majority of attempts at creating human-response machines (Kismet, etc) are based on demonstrating emotional and not binary response.

      I know my response is probably laced with Non sequiturs and some tangents…but I think the general sense is pretty palatable…I'm definitely not knocking anyone/any company for being able to make a living just 'doing analytics', easy or hard…there just seems to be, as you very clearly point out…a gravitation to one of two poles, with a loosely defined standard, layered abstraction in practicum, and certainly a highly politicized acquisition scale.

      Always a pleasure. I'd love to hear what Susan's top 3 might be…surely 1 must have something to do with hang-gliding….

      Joseph's Response: I'll ask her to add a comment or two…in her copious free time…
      And thanks again for reading and commenting. – Joseph

  5. Analytical culture, indeed. It matters.

    Here's what I find really interesting. If you look at business cultures / companies that were analytically driven *before* the web, you find very little of the angst expressed above among their web analysts or any of their analysts, for that matter. I'm talking about catalog cultures and a lot of database marketing shops.

    So, it seems to me that the conditions above are not so much tied to web analytics but web analytics is a good example, a business microcosm, of what happens when you bring accountability into a business unit. If this happens in a unit as young and flexible as the web imagine trying to bring accountability into other “soft” or “gut” areas like offline Marketing or Service…

    The fact web analytics is relatively cheap and (multi-channel companies might say) not capable of doing much “damage” if done poorly paved the way for widespread adoption. This has created a huge pool of test cases where we can see what happens when analytical discipline is brought into a company. All of the above does not bode well for “Competing on Analytics”.

    There is a right way and a wrong way to bring analytics into these areas, and while there are some brilliant successes (often showcased at eMetrics) there apparently is evidence of much more cultural failure than success.

    Accountabilty breeds fear, fear drives avoidance and avoidance creates Least Common Denominator behavior.

    This is exactly the reverse of the outcome desired.

    Who wants to fix this? How will a company Compete on Analytics if we can't change the culture?

    • From now on, I'm considering letting Jim have absolute governance over my babbling…his composition is, in essence, almost exactly what my point was in a much less convoluted delivery. —Thanks Jim — Daniel

  6. Wait a second…Jim Novo? You're reading my blog?
    Seriously, thanks for your comment. Very lucid, very enlightening. I did not know about what you shared regarding catalog, etc., businesses. I did get a chance to observe a NH based group move from pure catalog to mixed catalog-online and noted a “poor country cousin” mentality shaping although I didn't know why.
    I also agree that it's a cultural failure, not an analytics failure. None of the interviewing I did led me to think those participating felt analytics should be discarded (hence a “failure”), only that they weren't happy with the current “analytics culture”.
    And my deep thanks for reading and commenting. – Joseph

  7. Hello again, Daniel.
    I added my comments as Joseph's Response: … in your comment. It was easier for me that way. I'll extract my responses to a separate post if that's easier for folks. – Joseph

  8. Maybe the best Web Analytics (heck, Analytics) article I've ever read. 21 years in Business Intelligence and Analytics have taught me that everything is about people and relationships. Everything you mention here is symbolic of an audience that has yet to figure out how to take information and turn it into something that a Chief Financial Officer or a Chief Executive Officer embraces.

    That transition, from obtaining information that is self-evident to an Analyst, and converting it into something that is self-evident to a Chief “X” Officer is extremely hard, and it an art form that must be learned individually.

    Failure, almost guaranteed, leads to the comments that people sent to you.

    • Thanks, Mr. Hillstrom, for reading and commenting.
      You know, it wasn't my goal to … well, to achieve this kind of notoriety (I don't throw hand grenades, Jim. Kisses maybe but never hand grenades. Not intentionally. Way too violent for my likes). I basically like to study things and I tend to write about what I study. I'm glad people are finding it a worthy read. – Joseph

  9. Agree with Hillstrom's first statement. This is a pivotal article, and Joseph, I'm sure you know too much about the History of Science not to know the grenade in your hand is not a fruit (French pun)!

    I'd be curious to examine this question from a sociological point of view, and at the risk of sounding marxian, reconsider web analytics as a praxis, maybe too loaded with internal politics where numbers are often used to support someone/team's agenda. In short, the desired/supposed objectivity of numbers (and the obvious clarity of knowledge they are supposed to bring) is impossible to attain as soon as those numbers are expressed within a social context.

    • Howdy, Jacques,
      First and as always, thanks for reading and commenting. Yes, I know the French pun and chuckled when I read it.
      Examining from a sociological point of view…hmm…I'd like that very much. The reconsideration integrating praxis concepts is a good one and worth the time involved and I'm not up for doing it alone. Is there room in the official charters for an outsider to offer comment?
      Ah, but then we get to your “…impossible to attain as soon as those numbers are expressed within a social context.” and I think “'Nuff said.”
      Although for you, my friend, I'd make the effort.
      Again my thanks. – Joseph

  10. Daniel, you are too kind.

    Joseph, the “hand grenade” bit on Twitter was simply a way to get people's attention, er, “Advertising” I think they sometimes call it – not a commentary on the content itself. In other words, I wanted people to read your post!

    At a deeper level, if you look for root cause in this area – keep asking why? – you find people are not connecting the analysis to long-term value in the corp. While it's OK to improve conversion rate or some other metric in the end you have to translate that into real value in a way that's meaningful.

    I suspect people know this and it makes them uneasy that they're not doing it, or their attempts have been pushed aside. This creates the feeling that somehow “it's all a game” and results in the kind of comments you received for people in companies that *have* analytics but don't really *use* them to make important decisions.

    Meaning, when the analytics support their guts, analytics = good. But, as is more often the case, when the analytics debunk their guts, they push them aside and use their gut anyway, analytics = bad.

    This is a culture thing, a dynamic you described well above; at some level it's simply personal risk management. These antagonistic analytical environments can be mitigated by developing the power of persuasion and linking more tightly to the financials, as Kevin pointed out.

    But even deeper than that, when you ask the “5th Why?”, I believe you come up with something like this, a CEO that thinks:

    I want to Compete on Analytics. I want to rely more on facts and less on gut. Now, how exactly do I make that happen without destroying my company, what do I do first, second, etc?

    When we can answer that question, we'll start to move forward with a lot of these analytical disciplines in companies outside the small group of those who use the analytics to temper their guts, as opposed to reinforcing them when it's convenient to do so.

    • Hello again, Jim,
      Advertising? Wish I knew how to do it and thanks to you for filling in my gap.
      Before responding to this comment I wanted to get back to your previous one. I was in meetings all afternoon and driving back I got to thinking your concepts of culture change and how large a delta-T would be necessary (we did something similar for an company — changed their online and offline internal cultures — years ago. There's a whitepaper about it somewhere and if people are interested I'll dig it out). Anyway, thanks for helping me to think.

      “…impossible to attain as soon as those numbers are expressed within a social context.” Quite correct, yes.

      “Meaning, when the analytics support their guts, analytics = good. But, as is more often the case, when the analytics debunk their guts, they push them aside and use their gut anyway, analytics = bad.” Yes, again, quite so.

      “I want to rely more on facts and less on gut. Now, how exactly do I make that happen without destroying my company, what do I do first, second, etc?” Shebangsheboomie, yes. There were no indications that strategic (or even tactical) plans were in place. In my own work I've asked clients for their strategic plans, strategic marketing plans, strategic business plans so that we could integrate into their whole. Most often such things don't exist (and I'm talking even in big organizations).

      Thanks for your comments. – Joseph

      • Mike Mitchell permalink

        The analytics = good or =bad is partly cultural, but appears to have two other issues mixed in: 1) trust–we want to trust analytics, but when the decision has to be made, experience and knowledge win over intelligence–sad to say. That opens up the box to emotion, ego, expectation, habit, and faulty intellect. 2) Modern maladies which may include: time compression, data glut, complexity and it's cousin chaos, and friction (tension between local insight and data, creating conflict culture, processes and the demand for change).

        Analyitics hold out the hope to convert complexity into collaboration, chaos into opportunity and friction into traction. So far we have been disappointed, deceived, tricked, and have become jaded–thus we go back to our gut. Joseph's words here and his outstanding book, Reading Virtual Minds, holds out the first real step toward trusted analytics and the deciphering of complexity. To quote Heinz Pagel: “Science has explored the microcosos and the macrocosmos… The great unexplored frontier is complexity. I am convinced that nations and people that master the new science of Complexity will become the economic, cultural, and political superpowers of the next century.” That century has arrived and so has Joseph. My challenge is–my guts get it, but I am still trying to get my head around it.

        • Mike, thanks for reading and commenting.
          Your thoughts echo much of what is written in The Unfulfilled Promise of Online Analytics, Part 2, so in a way you presaged my publication of part 2.

          And thank you for mentioning my book (now also on Amazon).
          To your statement, “I am still trying to get my head around it”…perhaps we'll meet at a conference sometime and can talk. Otherwise I'm on Skype fairly often (nseJDC).
          Again, thanks for reading and commenting.

  11. Ned Kumar permalink

    First off, I thought this was an excellent discourse on the state of online 'analytics'.
    Joseph Response: Thank you, Mr. Kumar. I do my best and on the rare occasion succeed.

    In terms of the “Unfulfilled promise …” and discontent with web analytics, I agree with Jim that there seems to be a greater dissonance among web analysts than analysts in database marketing [or pre-web era]. Part of this is definitely the whole 'lack of accountability' or in many cases the missing piece in terms of how the analysis affects the bottom line.
    Joseph Response: Yes, agreed. This seems to be the case from my research.

    However, I feel that at a level deeper, this discontent is also partly from the dichotomy of perception on what analytics should mean.
    Joseph Response: Excellent Point!

    Having been in analytics for a long time I know that many of my peers who did analytics [before there was web analytics] think of analytics as a process rooted in robust methodology (statistical, machine learning, experimental design, etc.) and one which led to a solid result because of the supporting evidence.
    Joseph Response: Your statement (to me) indicates a scientific method. Emails, comments, Skype and phone calls from readers of this post are causing me to think that the “web time frame” has somehow negated scientific methodology from online analytics. I'm wondering if a case could be made that the web time frame — judged in seconds, minutes and hours instead of days, weeks or months — has caused people to ignore true scientific method because proper science requires recognition of finite time periods to perform. What are analysts to do if the data gathering cycle is five hours (for a social campaign, for example) while actionable analysis of that campaign data requires five days (hyperbolizing, I know, and I still believe it's valid)? Certainly this time dependency didn't exist pre-web (and I haven't had time to research that statement at all, it is a guess. Please, those with knowledge in this area, respond).
    So I think it might be an interesting study to determine if “things moving at internet speed” + “the internet economy” + “business opportunities” = a root challenge to “a thriving online analytics society”. (and I truly hope you (and others, if they wish) respond to this. I do believe this could be a kernel of something far more greater than it first appears).

    Further, 'analytics' for this group went way beyond the normal trending and reporting — and included data mining, forecasting, and predictive analytics. As these folks have moved on to either Management and/or other web analytics roles, they have [I think] come to expect similar 'capabilities' with the web data and tools.
    Joseph Response: Yes, yes! I believe your statement here concurs with mine above.

    But unfortunately, most WA tools today fall short in terms of analytical capabilities as compared with for example analytic tools and capabilities in database marketing.
    Joseph Response: ditto!

    As to “gut” vs “facts” (or discarding facts when things didn't look good), I think that existed even before there was web. This to me has more to do with the personalities of the people (analyst, Management and Leaders) involved than being a web analytics only issue. It is the nature of humans to want to look good — how things end up would depend on the outcome from the id vs ego battle within an individual.
    Joseph Response: Well stated! Thank you.

    I am optimistic however that with technological advances that will make multi-channel view of customers a norm (rather than an exception as it is now) and some education on the need for good planning (strategic/product/Marketing), analytics using web data (I just don't want to call it web analytics) will play an increasingly important role in the future.
    Joseph Response: I agree with you. I would appreciate your thoughts and insights during my preparation of Part 2 of this series (next month's post).
    Thank you for a very thought-inciting and well reasoned comment. – Joseph

    • Mr. Kumar,
      Thanks for reading and commenting. I responded to your comment via Joseph Response: … directly in your comment. It was easier for me (and what you offered was very stimulating. Thank you!). Please let me know if you or others would rather I pull my comments into a separate comment in this thread. Happy to do so.
      Again, thank you for reading and commenting. – Joseph

  12. Joseph: just one word “merci!”
    Joseph Response: Mon ami, trois mots, “vous tes bienvenu.” (vous rire, je connais)
    (Stephane, was I close? At all?)

    Your unique style and approach will become a landmark in the short history of web analytics! :)
    Joseph Response: Well…better landmark than landmine, yes?

    The Web Analytics Association, probably as we all do, is wondering where the industry is heading, if we're reaching out to the right audience, how to develop best practices, if there will even be such a distinct discipline as “web analytics” in the future, or if it will simply fold into “analytics” and “business analysis”… (me think the latter!)
    Joseph Response: Like you, I believe the latter and only because this is the pattern systems follow over time.

    Your research, and the discussion it sparked, is invaluable to help us shape our collective (and personal career) future.
    Joseph Response: Glad I could help.

    John Kotters revealed that at most 30% of change initiatives succeed… that leaves a pretty hefty margin for failure! Sad to say… but I can't help think back to the early days of the web, when it was more a craft then a science (some would say it was “hard”), and relatively few people mastered the art of creating websites. Then tons and tons of less experienced people flocked to be part of the web industry. The initial chaos, experimentation, failure (and learning) gradually led way to a more disciplined, systematic approach to web development.
    Joseph Response: Although I agree with the concept I'm not sure I agree with the entirety as you've expressed it. I would offer that ” The initial chaos, experimentation, failure (and learning) gradually led way to a more disciplined, systematic approach to web development” once somebody figured out how to make money from the web. The fact that someone figured out how to make money caused a market to form that was commerce driven rather than information driven (here's a problem no one has addressed yet or well (that I know of, anyway) — the 'net was originally information driven. It became commerce driven and is now struggling (net neutrality is an example, something else I'm researching) because the largest population (“visitors”) wants information (that might lead to commerce but more often doesn't) but the only way to deliver information (currently) is by using commerce to power that information-exchange. Anybody read my work on fair-exchange?).
    That information-commerce cycle is probably going to play into any solutions of the greater problem the industry (seems to be) facing.

    Online analytics is out of its infancy, but is still an adolescent wondering what he will do in life, with the usual arrogance, thinking that nobody else can understand, nobody is as good as he is…
    Joseph Response: I actually had an email exchange on this very point. I invited/encouraged/asked the individual to contribute their thoughts here. Only time will tell…

    And thanks for a stimulating discussion! – Joseph


  13. Joseph,

    What a great collection of thoughts here. I enjoyed reading this, and it is very relevant to things I am working on.


    • Thank you, Mr. Shumpert.
      Please let me know if there's more or other I can offer to further your own work. – Joseph

  14. Joseph, I think your style of commenting inline with the post is quite effective…
    Joseph Response: Glad you like it because I'm using it here.

    Hi Ned! Great to meet you (finally) at eMetrics….and a few comments on your comments heading into Part 2.

    From a Marketing perspective, I see the hangup you described as the frontend / backend problem, which in fact leads to a lack of strong ties to the financials. This may be a “role” problem more than a tool problem, as I have talked with web analysts who don't think they have permission or access to the backend data.
    Joseph Response: This “lack of access to the backend” showed up in my research.

    Certainly, at least for commerce, you don't need a web analytics tool to find out if a customer has purchased at least 3 times or the last purchase was a year ago or any of the data required to prove out the true financial performance of a campaign. And I have to believe many web analysts have the skill (or could easily acquire the skill) to work with non-web data sets that sit in the billing, accounting, or fulfillment systems. Which makes me think this problem is structural rather than tool-centric. Clearly though, having more advanced tools would help bridge this gap.
    Joseph Response: (standing off to the side while two great men converse, nodding in agreement with what I'm reading)

    Joseph, I think most web analysts are trying to follow the scientific method but simply don't have the support to create an end-to-end closed loop, for some of the reasons described above.
    Joseph Response: I would also conjecture that a lot (not all, merely a lot) of online analysts haven't been trained in a scientific approach. For that matter, I would offer than even those with (some) scientific training can get lured away into less rigorous protocols via tool use. I've seen this happen in labs. Good procedure goes out the window because everybody wants time on the new whatever and being able to use the tool becomes the proof rather than the results of the experiment. Whenever I read a journal and see something like “results were gathered using a OhMyGod 212 Pro Imaging System 3000 with the ByGolly Gasmatazz AintShePretty Modifiers in place” I groan. You can't argue with their results…look at the tool they used! (this is something I touched on in Counting Wristwatches at the SNCR Conference).
    I also recognize that few people are as adept as I (and I hope you're all laughing) at bridging the scientific and business worlds.

    As to whether the web itself is problematic – the cycle time issue – I'm not sure that's the case. For example, see this review of a full-blown scientific method test on social media, published in the journal Marketing Science:

    The authors are folks from academia and so of course, follow the scientifc method and build their case on the published works of previous academic efforts. Wonder what the web analytics world would look like using that model?
    Joseph Response: I'm unfamiliar with that work (so thanks for bringing it to my attention). NextStage (not meant as a plug) did something similar and I presented the results at a SNCR NewComm Forum a few years back as part of my “Whispering to Be Heard: The Art and Science of Buzz Marketing” presentation and then again in Montreal at the Communicating for Social Impact, International Communications Association Conference 2008 in a shared presentation, “Transmission of Influence and the Demise of Blogs” I did with Paul Gillin (a very knowledgeable man and someone worth watching). We were able to control a socially networked message, starting, stopping and changing it's course (in geographic and temporal spaces) for highly targeted marketing purposes.
    So while I'm unfamiliar with the work you mentioned I do know such scientific methodologies exist in certain online environments. I'll offer (purely from experience and largely from a paper I'm currently working on, “Social Network Mechanics: A Preliminary Toolkit for Creating and Co-Opting Social Networks for Marketing Purposes” (yep, more light reading, that. Bet you all can't wait for that one) that the scientific method can apply that social models and methods may not be a “web-wide” demonstration of their use (and I hope Paul Gillin offers some insight into this as he's very good at it).

    Finally, it think it's worth mentioning that the journal Marketing Science is published by INFORMS, the Institute for Operations Research and the Management Sciences. My impression is the INFORMS membership, taken as a whole, represents what the business world looks like when every silo practices the scientific method, inclusive of Marketing. INFORMS publishes research & testing in 12 different but interlinked operational functions!
    Joseph Response: Hmm…I wonder if they're looking for content…

    So, it seems to me we're not just dreaming about business optimization using the scientific method, it's happening all around us.
    Joseph Response: If not already happening then soon will be, yes.

    Then, what causes one company to be “analytically friendly” and another to be “analytically antagonistic”?
    Joseph Response: That is an excellent question to include in Part 2, me thinks.

    And thanks again for some stimulating thoughts. – Joseph

    • Thanks for commenting again, Jim, and also for accepting my use of intercommental responses as I used them in your comment above. – Joseph

  15. Intercommental responses, I like it!
    Joseph Response: Good because I'm at it again.

    Stirred by your mention above of analysts being “lured away into less rigorous protocols via tool use”…

    Apparently, quite a few are also “driven away” by pressure from the people they doing reporting for.
    Joseph Response: I would allow that there was animosity in many cases between consultants/analysts and the clients/people they report to. If not animosity then contempt.

    This is not a new problem in analytics, but I was quite surprised to see 50% or so of the people at a web analytics conference acknowlege this had happened to them…

    Certainly this kind of pressure to “torture the data” might result in analysts being cynical about the value they provide, no?
    Joseph Response: Definitely.

    A love-hate relationship, indeed.

  16. Joseph:

    I am immensely grateful to you for the thought you have put into this absolutely engrossing article.

    If had to pick favorites, it would be these these two sections:

    The tools promise a lot, and can live up to most of it.
    The unfulfilled promise of web analytics and search is measuring outcomes instead of outputs.

    I have not seen such a cogent set of analysis and I learned a lot from both sections. I have to admit the second section is dear to me because 150 pages in WA 2.0 are dedicated to measuring outcomes, and yet I learned something new from your article.

    One very minor disagreement.

    I disagree with the strength of the conclusion at the end of part one. I have only had a short time to think about why but…

    My first blush thought is that by the end of part one you are judging a teenager and I am judging a toddler (which is were I think we are with Online Analytics). The difference stems from the expectations and the yardsticks use to measure: 1. Current status. 2. Progress thus far. 3. Future promise.

    I am far more optimistic, than many, and a lot more hopeful.

    I am profoundly grateful to you for the time you have put into this article (can't wait for part two!), you have put us all in your debt.

    Thank you.


    • Greetings Good Sir,
      Thank you for commenting and adding your voice to these thoughts.
      I'm flattered you and others have found it useful. I'm encouraging everyone to offer their thoughts privately or publicly as I'm preparing Part 2.
      A nod to you regarding toddlers and teenagers. A similar comment was made earlier in this thread. I also welcome your optimism.
      Thanks again,

  17. One of Hillstrom's recent points was that many analysts fundamentally are not executives. He goes onto argue, I'm interpreting, that analysts don't understand how decisions are made at the C level, and don't know how to talk to them. He also makes a call for action for leadership (
    Joseph Response: Noted, documented and agreed to.

    Directly to his point, and in a nod to Novo:

    The intersection of Scientific-Quantitative Avenue and Business Strategy Boulevard is a shady one. The city really doesn't take care of it. The lights are burned out. It's dark. There's trash strewn about, and there isn't much traffic calming going on, so a lot of people just of breeze through it on their way somewhere.

    There's gaggle of Web Analysts at the South West corner. We have your Dataminers that are across the street, smoking and huddling for warthm, by and large keeping to themselves over there. There's a handful of MBA's who didn't go into finance at the other corner: they're the kids playing with their skateboards and doing some thrashing before they trade their board for a Vespa and go work for a bank.

    And we have a few frustrated market researchers – brand people really – lingering about outside the Denny's, kind of debating if they want to go become a planner inside and order the Moon's Over Myhammy. Of course, they'll go back into the kitchen and tell the cook how to do their job. But it's a hell of a lot warmer inside the Denny's than it is outside the street.

    Then a fight breaks out amongst the web analysts. They're all busy cutting each other with knifes (hence the opening graphic description of the self-inflicted scaring we got going on) when we really should be figuring out how to beautify the neighborhood. Maybe get some traffic calming put in to frustrate the hell out of the suburbanites, and while we're at it, dousing the Denny's with gasoline and letting the pure brand people collect some insurance money.

    It's going to take a neighborhood association to fix the 'hood.

    And the good news is that I think there are dollars kicking around for some beautification.

    I'm really convinced that the first step is to realize that web analysts, dataminers, marketing scientists, and market researchers live in the same neighborhood, and a large degree what we're fighting about at our corner just doesn't matter to anybody else except those at the narrow, South West Corner.

    I echo Hillstrom and Novo's sentiment, and I raise them:

    It's not just Web Analysts who got a problem.

    The neighborhood has really gone down hill since Claude Hopkins died.

    Joseph Response: I love the imagery you've used and I'm on board if it's helpful and others deem my involvement worthy. – Joseph

  18. Ben Robison permalink

    I feel a little like the servant speaking up at the King's table as I wade in on the discussion.
    Joseph Response: You and me both, my friend.

    I think an additional piece of this particular puzzle (war?) is the way that we define “Web Analyst.” We use that word like it describes a homogenous population, but I believe it does nothing of the sort.
    Joseph Response: Excellent Point, Good Sir! Thank you for making it. I believe I also inferred if not directly stated as much in my original post — I surveyed people from all parts of the online analytics career spectrum, high and low, well versed and not knowing the song (An Ghaidhlig, “Chan eil na faclan agam. Chan urrainn dhomh fonn a thogail. Fhalbh's cac!” (does everybody feel better now that I've gotten my Gaelic out of the way?)).

    No doubt it sometimes describes a person who is steeped in statistics, marketing, business analysis, or business intelligence. These “web analysts” know their data, they are intimately familiar with its strengths and weaknesses, they know where it came from, who its parents are, which side of the tracks it grew up on, and what it likes for breakfast.
    Joseph Response: I love the expressions you've used above. I hope we get to meet sometime. I'll bring cigars if you'll bring some Scotch.

    Web Analyst also describes (more often in my experience) someone who knows nothing at all about any of these things, can't swim, and just got thrown into the deep end of the pool without floaties, swimming lessons, or a lifeguard. These people got moved onto the web analytics team from somewhere else in the organization because their company recognized a need for “web metrics” and “restructured” them there. Or they're fresh out of college with a degree in basket-making and got hired because the perceived need for web analytics is so great that anyone with a pulse can get a job (think Accounting-firms right after SOX hit). These analysts are willing to perform, but lack orientation and direction.
    Joseph Response: Again, quite accurate according to my research. Responses along the lines you've written above came from both sides of the Atlantic.

    These are, of course, two extremes and reality is a spectrum between these two end points, but heavily weighted towards the bottom end (again, my opinion).

    Those on the low end of the spectrum are particularly susceptible to the rallying cries of “Web Analytics is easy!” or “Web Analytics is hard!” perhaps based on their predisposition to think of water glasses as half-empty or half-full (though I believe those who champion these two divergent opinions are actually talking past each other and addressing different aspects of the discipline).

    As a web analyst myself (see servant comment above), I fall somewhere in the middle, knowing systems inside and out but working to catch up in the marketing and statistical arenas in my spare time (when not spending hours reading this post).
    Joseph Response: I promise to do my best in the future to fulfill Mr. Hillstrom's 140 character wish (see Analysts and Executives). Please don't expect it any time soon, though.

    My belief is that regardless of whether the industry is a toddler or a teenager, there will be a difficult culling on the path to maturity as the lower-end of the spectrum is forced out, self-selects out, or moves upstream, because its at the upper end of the spectrum that web analytics can earn its seat at the King's table (or the CEO's, CMO's, etc).
    Joseph Response: Your paragraph above is a very powerful one that is making me think (always a scary thing for those who've seen it close up). Hmm…something for me to ponder. Thank you!

    Only when analysts can gain confidence in their data, their abilities, and their analyses will they be able to learn to speak the King's language. Joseph, feel free to comment inline since if it suits you =)
    Joseph Response: Thanks for letting me respond intercommentally. I truly am hesitant to offer the following suggestion (even though I equally truly believe it will be a helpful one) because I don't want this to be a “NextStage plug”. That stated (and if others are willing to offer a similar service to what I'm about to propose, please feel free to do so here); NextStage offers a training, “Know How Someone Is Thinking in 10 Seconds or Less”.
    This is a training that started out as a college offering in the early 1990s, became a seminar we delivered throughout the US and Canada (still primarily in college environments and having nothing to do with analytics) and has been incorporated into the NextStage trainings as both a seminar and a class.
    Most times these trainings are used by people who need to communicate in “hostile” or “unfamiliar” situations. When Susan and I visited Rene and Aurelie in Madrid, Rene had the opportunity to observe this training first hand; we sat in a market for a drink and I told him how the people in the market perceived themselves (as Spaniards, as Madridians (??), as people of a given socio-economic class within a greater cultural whole), how they perceived non-Spanish Hispanics, how they made decisions, how to influence them in their decision making, etc., etc., and all without knowing Spanish thus without knowing what they were saying, only understanding what they were thinking based on how they were communicating.
    In any case, Susan and I will offer that training gratis to conference participants at some conference provided whoever runs the conference covers expenses, etc., etc.
    And I hope people understand that's not a plug, only the offering of a service (as in “something we can do”, not “product or service”). For those who want to know why we'd offer such please read NextStage's Principles as I hope those explain things better than I.

    Again, thanks for bringing your voice and thoughts into the discussion. It's muchly appreciated. – Joseph

    • Mr. Robison,
      Thanks for reading and commenting. I'm responding intercommentally (it seems to be working. Please let me know if you believe otherwise). Your comment and my response will be up in a few moments.
      Thanks again. – Joseph

  19. Ned Kumar permalink

    Joseph, this has been a great 'socratic' discussion and thanks for providing the platform (There is no drug like a good stimulating intellectual conversation). And I love your intercommental responses – that provides an immediate context for the mind to respond. I only wish I could put in intercommental responses to your interconmentals :-). And Jim (Novo) — ditto here, it was great to meet you at EM after our many virtual conversations.
    Joseph Response: Remind me to tell you about teaching the Socratic Method to a bunch of undergrads sometime. It's wonderfully hilarious (to us).
    Agreed re stimulating intellectual conversation and glad you like the intercommentals. If someone can demonstrate how to let folks intercomment my intercomments, I'm willing to learn.
    And happy to bring you and Jim Novo together.

    Now to a few comments (Btw, you can drop the Mr.Kumar part – not big on formalities).
    Joseph Response: Mr. Kumar, I never assume the familiar until granted permission to do so. Thanks for the permission, Ned.

    Joseph Response: Im wondering if a case could be made that the web time frame judged in seconds, minutes and hours instead of days, weeks or months has caused people to ignore true scientific method because proper science requires recognition of finite time periods to perform. What are analysts to do if the data gathering cycle is five hours (for a social campaign, for example) while actionable analysis of that campaign data requires five days. Certainly this time dependency didnt exist pre-web. So I think it might be an interesting study to determine if things moving at internet speed + the internet economy + business opportunities = a root challenge to a thriving online analytics society.

    Ned: I definitely agree with you that you that there is a 'Warp factor' when it comes to data generation, collection, and usage of web data relative to say data from non-web channels (e.g Sales or Marketing). And many head honchos (the Hippos, C-levels or whatever you want to call it) has exacerbated this situation by not taking time to develop a strategy or an infrastruture but instead forcing their minions to crank out reports at the speed at which the data is generated. While this is good for certain types of situations — in most cases, what it means is that there is not even the thought of using any 'scientific' method or actually pondering over customer patterns holistically (one of the reasons I am averse to folks who say they wish they had 'real-time' data — my question to them, “What are you doing with 'non real-time' data you currently have?”).
    Joseph Response: Agreed. I've only found “real-time” data to be expressly useful in social tracking to determine memetic dissemination. We'll use it in other situations and only with great care, lots of explanations and hand-holding.

    I hear the point you make with the five-hour data gathering cycle/five-day actionable analysis example — but my thoughts are that the concept of “time” is not the root cause for the lack of rigor in the online analytics community (possibly a mediating variable??).
    Joseph Response: Acknowledged.

    In the past, I have advocated a layered approach to analtyics wherein each layer address a certain aspect of the business strategy and uses a certain type of methodology to provide actionable insights and bottom-line impacts for that specific strategy space. Or to use another analogy, we may have continuous stream of information collected every second from visits to the website but the key is to learn how to use this stream as a tributary to your analytic framework (which will also include non-web data) — where increasing amounts of information flows to different areas of the framework using different methodologies to create multiple facets of what is happening (this has the added benefit of triangulation of whether your initiatives and strategies really succeeded. My point here being that rigor or 'scientific approach' can be applied to web data if one really wants it. I think the issue is more about the people (another debate, another time).
    Joseph Respnse: I hope people read the above carefully and paid attention.

    Joseph Response: The fact that someone figured out how to make money caused a market to form that was commerce driven rather than information driven.

    Ned: Well, I hope our community realizes that a pure commerce driven mindset is a recipe for a hard fall. When there is an innovation, you can go commerce driven as folks are curious and excited about your product — however, if you don't have an information/analytics framework in place then that commerce driven success is going to be very short lived indeed (absolutely not sustainable).
    Joseph Response: Hear, hear! I say “Ye, Mr. Kumar (or 'Ned', if you prefer)!”

    Jim Novo – This may be a role problem more than a tool problem, as I have talked with web analysts who dont think they have permission or access to the backend data…….Certainly, at least for commerce, you dont need a web analytics tool to find out if a customer has purchased at least 3 times or the last purchase was a year ago or any of the data required to prove out the true financial performance of a campaign. And I have to believe many web analysts have the skill (or could easily acquire the skill) to work with non-web data sets that sit in the billing, accounting, or fulfillment systems. Which makes me think this problem is structural rather than tool-centric. Clearly though, having more advanced tools would help bridge this gap.

    Ned: Jim, I have to agree with you there that access to 'backend' data is an issue (in fact I stressed on that at my Emetrics talk in DC and why that is critical). It also goes to my earlier point about Management not thinking clearly about the necessary infrastructure, processes, and strategy early enough. On your other point about using web and non-web data, I believe that it goes beyond just skill. I could be an expert with all the web analytic tools, SAS, SQL etc. etc. and still not know how to chart my way through the mountain of data available. I think one needs to have the cognitive mindset and critical thinking capabilities to understand what is important and what data sources need to be tapped to get an answer (the tool is irrelevant as there is a redundancy of tools that can be used for a task).
    Joseph Response: Acknowledged, a “theory v application kind of thing.

    Joseph: being able to use the tool becomes the proof rather than the results of the experiment.

    Ned: Agree and my point above is along similar lines. We are always looking for experts in various tools — instead of testing their mental models and skills on information usage. To Jim's point above, solid analysis using web data can be done without using the web analytics tool at all — because if you know what you want (business need), where the source is (backend), how to get there (methodologies) then there are multiple tools out there that will get you the result. In fact, with the push towards looking at customers in a channel agnostic manner I am predicting that the current tools will go away unless they start putting rigor into their systems and allowing easy access to data integration across channels and platforms.

    Jim: Then, what causes one company to be analytically friendly and another to be analytically antagonistic?

    Ned: My short answer to this is — people and culture. To quote Schein, If you have been trained to think in a certain way and are a member of a group that thinks the same way, how can you imagine changing to a new way of thinking? I don't want to get into a discourse on this topic here but if we want our community to thrive, organizations have to make analytics (with all its rigor) an ethereal part of their culture. This definitely will involve a cognitive shift based on where we are today but at least one can start by hiring “seed” analysts/leaders who has a broader view of the subject and the chutzpah to stand up and influence the thinking of their peers.
    Joseph Response: First, a repeat of my above “Hear, Hear” statement. Also and personally, I'd love to open this discussion to Scheinisms and their import on cultural change as I believe it would be both relevant and useful.
    And people who know me are shocked I'd suggest that, yes?

    As before, thanks for reading and commenting. – Joseph

  20. Hi Joseph,
    Again, your thinking, speaking through written words and commenting cause me to pause and consider what I am doing as a web analyst, and what I want to be as a web analyst.
    Joseph Response: Howdy, Dylan. How's the reading going?

    I am in the centrist camp where web analytics isn't easy and it isn't hard. Web analytics is a craft that is filled with artisans of many different levels. As artisans, it is our job to help our customers and partners understand the beauty of our work in their words, thoughts, and behaviors. However, we web business analyst people aren't as overt as we should be when talking, doing, and discussing our craft with each other, but we have regular monthly meetings (on wednesday's) and even semi-annual events around the world.

    Since you know history better than I, and sometimes past behavior can predict future results, is there hope for fulfillment?
    Joseph Response: As Avinash commented earlier in this thread, there is hope.
    Answering “is there hope for fulfillment?”… Yes. Fulfillment for all those around the table would require change from all those around the table; not something easy to achieve because trust, respect and understanding are required from all sides of the table towards all other sides of the table.
    I also recognize that fulfillment can be achieved by individual players at the table by their simply leaving the table should it become obvious that others aren't willing to put in the work required for mutually beneficial change to occur. (This is one of the toughest things for people in couple's counseling to understand. They always want to know if the relationship is going to survive. Good therapists respond with something like “The relationship will always survive and it may not be in the form you currently understand or can accept. If you two go your separate ways, the relationship has survived, it's simply taken a different form.”).
    So is there hope for fulfillment? Yes, there's always hope for fulfillment. Is there hope for fulfillment for all parties? That requires all parties to take part. This goes back to the 10 messages stuff I talk about. The first to demonstrate trust (“The Handshake” mentioned in RVM) is demonstrating vulnerability and who among those in the room wishes to be the first to offer their hand?

    Another way to ask this is: As our craft becomes “better” for businesses that want to “understand” behaviors, outcomes, sentiments, and the colors of their websites, how do we unfulfilled web business analysts become more fulfilled, and help fulfill our promise?
    Joseph Response: You've presciented two events that I alluded to above; the craft (and I note that you're not referencing it as a science but as a craft, thus putting analytics into the “art” camp) becomes better and businesses want to understand. You've presented two change factors required for fulfillment to take place. What of the vendors? What do they offer as part of the fulfillment model? Better tools? How do we define “better”? Do they reveal more? Reveal in more detail? Reveal something different? Again, define “more”, “detail”, “different”.
    There's also the recognition that the model implied above has economic power. Analysts have perfected their craft therefore they present greater value to their clients. The clients have developed greater understanding therefore they offer greater task satisfaction to the analysts.
    Again, what can the vendors add? (and I hope all readers understand I'm not writing as a vendor here. Rene and NSA are vendors (if I'm even using the term correctly). NSE offers tools, yes, and only for research purposes. Anybody doubt that NSE is still heavily an R&D house, read some of our whitepapers).

    I look forward to the next post.
    Joseph Response: And I'm encouraging all who wish to to contact me with their thoughts. As with the research that went into this post, anonymity will be applied.

    Intercommenting is encouraged (and thanked), a (Google) wave would be great, and while cigars aren't my favorite I am learning about scotch…for next time we meet in person.
    Joseph Response: I'm really happy that so many folks like the intercommenting. It's much easier for me than reading two comments at once and scrolling back and forth to understand what goes with what, so thanks for the nod.
    I got an invitation to Google Wave and haven't figured out how to get it started. Anybody who doubts my Ludditehood should spend an afternoon with me.
    You're learning about Scotch? We'll need to travel the Whiskey Trail (Uisge Bre Lorge ans a' Ghaidhlig (I got to use Gaelic twice in a post Nyah Nyah)) in Scotland together some time.
    Looking forward to the next time we meet.
    Thanks for reading and commenting. – Joseph

  21. Wow! Thank you, Joseph, for broadening what is, I believe, the single most important conversation facing our industry today. I am not only the CEO of one of the vendor companies, but I also am fortunate enough to sit on the board of the WAA. These topics are core to our future direction and success as an industry and the general theme is one that I have, myself, attempted to propagate for the last year — literally, the focus of my analyst tour last fall (Forrester, Gartner, etc) was very simply, “the unmet promise of the web analytics industry”.
    Joseph Response:First, my pleasure and thanks for joining in the dicussion. I note that you're the first vendor to openly do so and I applaud you for it.

    Second, I was unaware of your efforts in this area. Where are they published? (My reading is admittedly very limited).

    And a tour? You got to go on a tour? I'm still not allowed out of the house except to fly kites…

    In any case, I've never had any luck broadening a path unless that path already exists. There is no need to broaden a path that few walk and (writing as an avid forest walker) often the few that know of such paths prefer they not be broadened, it makes them obvious to others (the “elephant in the living room” scenario, in this case).

    For too long, the resounding cry has been that “the web is the most measurable medium”, yet general practice has resulted not really measuring anything meaningful.
    Joseph Response: I don't mean to be disrespectful here, and you do know how many logical contradictions are implied in the statement above, yes? “general practice” can not exist in any form unless tacitly agreed to by all parties involved. Therefore if “general practice” has resulted in “not really measuring anything meaningful” then all parties should take responsibility for this.

    Additionally, the web was not rapidly embraced by traditional media agencies, diminishing the comparative financial scale. I don't pretend to have the depth of understanding of the psycho-social behavior that you possess; …
    Joseph Response: I'm available for weddings, bar mitzvahs, …

    …however, I agree with and have regularly seen many of the behaviors and frustrations discussed in your post. Vendors HAVE promised the moon and in most cases have only been able to deliver a short trip around the block.

    Joseph Response: Very brave of you to say so. I would encourage other vendors to join in this discussion. I also note Benjamin Franklin's statement regarding getting people to join equally around a table, “Democracy is two wolves and a lamb voting on what to have for lunch. Liberty is a well-armed lamb contesting the vote.”

    The concept of “easy” HAS been readily promoted by many of those vendors and has, as a result, undermined the knowledge, experience and depth required to successfully derive business value from the practice. Eric Peterson is well known for his stance on the subject that, “it is not easy” and I stand beside Eric in this opinion.
    Joseph Response: Whether or not web analytics is “easy” or “hard” I can only answer for myself and within the caveats of my post (practice, understanding, etc). I would offer (based on conversations and experience) that the analysis part is easy, the politics part is hard and because the politics part tends to take more time (my experience, that) the whole practice may be considered “difficult”. The understanding from this is that analysts are good at analysis and less adept at politics, as noted in my post.

    I've noticed that people keep on using “hard” and “easy” in these responses. I think I'm the only one who's offered a definition of “hard” as in “difficult to perform or do” and even that by implication. I would appreciate having someone define these terms in the online analytics context for me because the evidence so far is that people tacitly agree with my definition and that must be placed beside my statements that things are difficult to perform or do because they're being done improperly.

    Help, anyone?

    As with any human behavior, change typically comes when it is forced by some catastrophic event. I think that it's fair to say that the economic activity of a year ago was catastrophic and we have seen a dramatic shift from traditional media spending and to greater attention on the web. Along with that shift and the economic strangulation that most organizations experienced, came a rationalization of the fact that many companies were “spending” on the web, but not many were “investing” in the web — having a web site and web analytics just doesn't cut it, but it costs money. Ultimately, as the CEO of a company and as any CEO should ask, “Will this expense make me money, or save me money”? If it doesn't, why would we spend on it? Thus, the complaints that you cite below generate a great deal of frustration and anxiety for those responsible for the answer:
    “Yet the need for online analytics to be part of a larger picture, a grander story, was everywhere. Analysts uniformly perceive themselves as
    -not part of a unified business reporting structure,
    -not contributing to the big picture, and
    -lacking the political power or psycho-social maturity (within the organization) to sit at the grownups' table.”
    I would propose that web analytics has, by its very nature, prevented itself from gaining a seat at “the grownups' table” — data living in silos, speaking in terms of “pageviews and visitors” as being the currency by which success is measured, “keeping it easy”, staying at arms-length from the rest of the company, generally a lack of integration with traditional business measurement practices. It's a problem and, honestly, a vast majority of the technology developed in this space does not easily support any efforts to counteract this direction; therefore, “easy” was much simpler to sell to unknowing marketers who didn't understand the nuances of the underlying technology required to promote greater accuracy, integration, measurement and true business value. It's been frustrating and confounding for me to observe this industry, that I've spent nearly a decade in, literally almost eat itself alive in the euphoria for something quick, easy and dare I say, “cool”.

    Joseph Response: I keep on hearing the term “sexy”. I'm really growing to hate that word…

    This frustration applies not only to the technology, but also the, governance, systems, practices, processes and expertise invested in — ask any company how often they update their web site in a year, then ask them how often they update their measurement practice — it is incongruent.
    I do, however, firmly believe that the economic travesty of last year, along with a space that is now almost 15 years old, has promoted significant and very rapid expansion in the maturity of the requirements, expectations and knowledge in the space. I know of a number of organizations within which there is now a seat at the table and the questions being asked are real, financially driven questions. The answers are finally getting answered. It is not a proposition as simple as tagging a piece of content and measuring the clicks — sorry. It is not as simple as a bank reporting on credit card application completions — zero value. It is as important as saying, “of those completed applications, what is similar about customers who are using the credit cards and going into debt (because, after all, a bank is in the business of loaning money). NOW, let's go back to everyone with those same characteristics and try to get as many of them as we can to apply!”
    Joseph Response: Wow and thanks for this. Understanding psycho-emotive characteristics (The {C,B/e,M}'s, folks) across populations is something NSE has been promoting for quite some time. Glad to see you're on board with that.

    In many respects, we have come full circle by allowing organizations to essentially accomplish the level of accuracy and conversion that was present back in the heyday of database marketing, before the internet. There is immense value in this customer intelligence and the ability to act on it, for any type of organization. As an industry, there is some phenomenal progress being made, but it must be taken seriously, understood that it is not simple and “cool lollipop pie charts”. Success requires being able to put the web on the same financial plane as the rest of the business, an onus which has much greater significance in the current economy.

    Joseph Response: As I'm busily preparing part 2 of this post, I'm open to discussion about what can be done.

    Thank you again for the discussion!


    Joseph Response: My pleasure.

    • Hello and Thank You for adding your voice to this discussion. I've responded intercommentally to your comment above. – Joseph

  22. Hello, everybody,
    This comment was meant to go on Chris Berry's The Schism in Analytics, A response to Carrabis, Part II and everybody knows I'm not techno-savvy, hence (and with Chris' permission. Thanks, Chris!), I'm publishing it here.

    Howdy, Chris,
    First and with your permission, I'll be cross-posting this comment on The Analytics Ecology.
    Second, it's an amusing piece of serendipity that your post came to me this morning; I've been working on a six point methodology for healing the scars.

    Chris wrote: There are those who look to the past only to find evidence to confirm what they remember having thought. These are proof-seekers or justifiers. No further analysis over and above the baseline amount of proof is required. And, if the proof is unsatisfactory – then the data must be inaccurate. Frequently, all that is required is a simple, static report listing a few numbers.
    Joseph Response: This paragraph deals with what's becoming point 1 of the six point methodology. Good to know I'm being presaged. Gives me confidence I might be onto something.

    Chris wrote: There are those who look to the future. It is perfectly possible to do very thorough analysis about what could happen in the future and optimize against those scenarios. There's a large amount of valuable competitive advantage to be had that way.
    Joseph Response: This points to a couple of blog posts I've written, Minimizing Mistakecule Probabilities and Addendum to “Minimizing Mistakecule Probabilities”, as both deal with predicting best optimization paths.

    Chris wrote: I think there's a huge market for justification. In fact, I think this is why so many vendors go where the market is. They just respond to the market, don't they? And for most people, it's purely about justification. The tools aren't set up to explain anything in the past because that's not where the market is at. The result is redictable. Tens of thousands of reports generated daily, going unread and ignored:
    all a function of market demand.
    Joseph Response: This makes me think I should talk to Rene about developing another tool; The Justification meter. It determines (via analyzing some published report) just how much the authors are attempting to justify what they did or plan to do. Probably have great use in politics.

    Chris wrote: To pin the blame solely on vendors is like blaming obesity on fast food companies. They're only giving the market what it wants, negative externalities be damned.

    (Who is anybody to resist market forces?)

    Joseph Response: …oh, I don't know. NextStage has been doing a pretty good job of resisting market forces for about 10 years now…probably why so few people have heard of us. (I know you're laughing, Chris. I just hope others are, as well).

    Chris wrote: I think there's a growing market for actual learning and competitive advantage – driven by science. This isn't justification seeking or validation seeking behavior though.

    This is driven by upper management – people like Alan Wurtzel of NBC Universal (September 2009) who literally tired of drowning in data. If anything, they're looking for a consolidation of data by way of scientific methods.
    Joseph Response: Agreed. I think is why more and more NSE clients asked us to come up with simple tools/gauges that gave specific suggestions for making things better.

    Chris wrote: an analyst only needs to have the political skill set an alien ambassador on Babylon 5…
    Joseph Response: Love the B5 reference. Everybody thinks I'm a cross between Kosh and G'kar, anyway.

    Chris wrote: Sundry reportage – the generation of justification and validation – will probably take at least twenty years to be destroyed because of the a stubbornly long S curve. And you know what: good enough is good enough for huge swaths of the economy. I understand that there are companies out there that turn trees into toilet paper. I salute them and believe that there are analytical products that are perfect for
    them. I won't dare call those products “science”. (And when they're ready for real science someday, I'll be there too).
    Joseph Response: Also agreed. I've publicly stated that there will be a market for what is now recognized as traditional web analytics for a good many years to come (From TheFutureOf (5 Jan 09): Omniture and Google Considered Environmentally and From TheFutureOf (13 Mar 09): The Analytics Ecology). My reasoning may come from a different place than yours and still I agree with the prediction. I base my thoughts on evolutionary and economic forces. I thank you for your insights and comments on this as I always prefer to bring as many disciplines into my outcome predictions as possible. I'm a strong believer that the more lenses one has to look at something, the more likely you are to see the truth of it.

    Chris wrote: I think there's a solution in the schism: honesty and retitling.
    Joseph Response: Retitling comes easy and is already a well-known methodology, me thinks. I've seen it in action too often when I deliver research results. Honesty, though… I've promised to share with some others some of the sentiments directed towards me, NextStage and the horse we rode in on that came to me in emails and phone calls when there's Scotch and cigars available (and more than just with the Good Dylan Lewis, by the way). There was good old, honest emotion in them, very little honest reasoning. Sigh.

    Chris wrote: If a web analyst has the drive and desire to actually be a real scientist-practitioner, and their company isn't going to go there, then they have the duty to get out or STFU.
    Joseph Response: Ihad to look up “STFU”. You're beginning to sound like me when I'm behind closed doors.

    Chris wrote: If a web analyst doesn't have the drive and desire, then I'd argue that we should retitle that segment of the industry as 'web reporting'. It's not worthy of the term 'analytics' at this point.

    I think that vendors who are clearly in the business of web reporting need to come out and say, “we do web reporting”, and that vendors who do analytics need to come out and say, “we do analytics, scientifically”. That said, we need people who are honest enough and loud enough to call bullshit when a vendor is just that. If it gets nasty, so be it.
    Joseph Response: Yes to the web reporting statements. Regarding the bullpoop… A mid-size conference group asked me to evaluate their keynote speaker once. I turned away for a moment, wondering how to respond. To be honest, after listening to this speaker, I couldn't figure out how anything they talked about applied in any way to what people had come to the conference for. Further, much of their research was decades old, therefore not based on web paradigms, therefore actionably invalid. So I turned back to my hosts and replied, “They do things differently than I do and they've had different training than I, therefore I'm not qualified to answer your question.” I think my hosts are still laughing at my diplomacy.

    Chris wrote: Whether or not we take the same people who are in the industry now will be with me in five years, is the next cause for debate.
    Joseph Response: The comments to The Unfulfilled Promise of Online Analytics, Part 1not the post itself but the comments — demonstrate the answer to your implied question above.

    As always, thanks for making me think. And on a Monday morning…with only 1.5 cups of coffee in me…I hope readers can follow…


    • Thanks for the detailed reply Joseph.

      I laughed exactly where you predicted I would, before reading the prediction. And then I laughed about the prediction. Well played sir.

      To your points: exactly.

      In terms of solutions: do the scientific-practioners nail their 96 theses to the door and leave it at that.

  23. Anonymous permalink

    The topic being discussed in these postings ties in with other research that Ive recently completed. The research involved many topics and one was on Web Analytics. The general feeling from the companies contacted was that Web Analytics generates an enormous amount of information that the companies have no idea what to do with. Time and again the response received was Were getting all of this great information, but no one knows what to do with it. They didnt know if the information they received was useful, or a waste of time because they didnt know how to use the information.

    Many of the Marketing departments were overwhelmed by the amount of information that was provided. Others had the opposite problem, not knowing all of the information, i.e. reports, that were available. The old I dont know what I dont know situation emerged. Marketing had no idea what reports to ask for because they didnt know that those reports could or were being generated.

    The IT departments we talked with felt as though they were caught in the middle; sitting on lots of information that the other groups were not aware existed, but not able to do anything about the information generated. They were told to collect the analytics, but then not told what to do with the resultant data. Analytics seemed to span many departments that werent talking to each other. IT collected, maintained the analytics and reports, while other groups utilized and made decisions based on these reports.

    Another problem occurred with companies who actually utilized the data. Those groups who took the time to analyze the data and formulate plans based on the information collected were often shut down by the higher ups who did not understand the importance of what they were looking at or didnt have the budget to implement the suggestions. Priorities were often in conflict as budgets were tightened and if there was no immediate ROI indicated by the changes suggested then these projects were often put on hold until later.

    Often the data just continued to pile up as more stats were generated, but the time, manpower and budget werent there to change things. People never suggested that they should get rid of the analytics; after all there might be something useful in all that information.

    • The above comment came to me via email and I'm being allowed to share it as a comment provided I do so anonymously. – Joseph

  24. Chris Grant permalink

    I tend to view just about everything said here through the lens of my organizational development training. And what I see is a newish profession (or discipline) going through the usual phases as it matures … or fades, perhaps. Or is absorbed by something else. A lot of what is being brought up here is more or less classic phenomena to an org dev analyst; the existence of this very discussion, too, fits the typical bumpy, rock-strewn growth path. All along the way, the profession has to prove that it's worthy of continuing to be on the path and the definition of “worthy” always morphs in interesting ways, as it is doing here.

    The really interesting part will be the solutions discussion.

    • Hello,
      Thanks for reading and commenting.
      “The really interesting part will be the solutions discussion.” Thanks. I wasn't feeling any pressure so far at all regarding part 2, you know…

  25. Mike Mitchell permalink

    I am not a big science fiction fan, but I do remember enjoying Azimovs Foundation trilogy. Reading this blog and the comments, I half expect to see a comment by Hari Seldon referencing his work with psycho-history. Josephs work may come from some of the same roots Azimov suggested decades ago and he seems the first to be following in the footsteps of Seldon himself.
    Joseph Response: Yes, I'm familiar with the original Foundation Trilogy…I think I read them in high school. And folks who've read Reading Virtual Minds V1 know about the link to AJ Budrys' work. Never read the Foundation prequels, though…

    In later prequels to the “Foundation Trilogy,” written long after the original stories, Asimov described how Seldon gathered the data needed to perfect psychohistory by visiting different cultures spread across the planet Trantor. By observing a variety of societies, Seldon discovered the common features of human social behavior needed to make sound predictions. They are pretty much listed in fact in Reading Virtual Minds, Josephs new book.
    Joseph Response: Hmm…do you suppose I have a case against the Asimov estate? When were the prequels written? I don't know if it would matter. Pretty much what I did and what Asimov had Seldon do is part of standard cultural- and social-anthropology. If anything, it was my basis in other fields of study that brought things together for me. Although now I understand why people always ask me if I've read the Trilogy.

    Like Seldon traveling Trantor, anthrosemiotic anthropologist Carrabis seems to have traveled the disciplines that define or explain human nature in order to forecast social future.
    To quote UCLA anthropologist, Ropbert Boyd: “We have this weird, I think untenable, situation in the social sciences. A student learns one story about human behavior in an economics class, and then something quite different in sociology. Psychology class offers yet another version. And they come up here and we anthropologists tell them all kinds of different things because we don't agree about hardly anything. This is not OK. It's not acceptable that the economists are happy with their world and the sociologists are happy with their world, and that this persists in an institution which is supposed to be about getting at the truth.”
    One of the things that I like about Josephs work is, he is not afraid to build bridges, or blow them up where appropriate.
    Joseph Response: Blow up bridges? Moi? Jim Novo had me throwing grenades. The opinions people have of me…
    And because we're talking about cross-disciplinary studies, two of my favorite jokes:

    1) There's a small fire burning on the floor in a room. There's a bucket of sand beside the fire. An engineer walks in, sees the fire, sees the bucket, pours the sand on the fire and walks out of the room. A physicist walks in, sees the fire, sees the bucket, pours a ring of sand around the fire, sits down and studies the fire. A mathematician walks in, sees the fire, sees the bucket, realizes the problem can be solved and walks out of the room.
    2) A man wants to know what 2 + 2 is so he goes to a mathematician and asks, “Professor, what is 2+2?” to which the mathematician replies, “How may decimal points do you want it out to?” He then finds a psychiatrist and asks, “Doctor, what is 2+2″ to which the psychiatrist replies, “Interesting. How did your mother treat you when you were two?” He then goes to an economist and asks, “Sir, what is 2+2?” and the economist answers, “What do you want it to be, my boy?”

    Thanks for reading and commenting.

    • Hello again, Mike.
      I'm going to respond to this post intercommentally as Joseph Response:. I hope that's acceptable to you. Let me know if you'd like me to extract my responses to a separate comment and I'll do so.
      I'll be posting my intercommental responses soon.
      Thanks for reading and commenting.

  26. fxposter permalink

    I want to quote your post in my blog. It's super, dude.

  27. Hello Joseph,

    I'm entirely new to Web Analytics, as I've only read Mr. Kaushik's “Web Analytics: An Hour a Day” and applied it to my own websites using Google Analytics.

    As I've taken serious thought in pursuing a career in Web Analytics (I signed up for the Intro course at UBC) as well, there's one thing that I derived from this epic article – it was the insight the culture and psychology in the online analytics industry.

    Thank you for putting so much time and effort into this article. I'll probably have to revisit this article sometime again in the near future as I learn more and gain a broader perspective.


    Will Lam

    • Hello and greetings,
      Also thanks for your kind words. You probably know more about web analytics than I do (nobody at NSE claims the title and NS doesn't do what is thought of as web analytics).
      Hopefully The Good Mr. Kaushik is reading and will appreciate the nod towards his work. I know we have some UBC instructors reading and perhaps they'll recognize the nod in their direction, as well.

      “epic”… I've had lots of positive and negative feedback on the length of my posts. Something for me to investigate further, perhaps.
      The time and effort thing…maybe that accounts for the length? A kind of “I agonized through this research and now so will you”? Seriously, my pleasure. I enjoy doing research, learning, finding things out.

      And a welcome back to the future when you return.

  28. ridwanzero permalink

    We couldn't get affiliate programs to work either. Everybody said affiliate marketing was easy and effective and we trashed and burned badly. Tell people to stay away from it.

  29. LorettaM permalink

    Hi There
    We're in the planning stages regarding web analytics and interviewing several candidates and vendors. Your post came just in time to save us considerable money. What are next steps for companies needing more capabilities than free tools provide?

    • Hi LorettaM,
      Others may have different suggestions than I. My suggestion is to determine your exact reasons for getting any analytics solution. A basic question would be “What information do you believe an analytics solution will provide such that your company will suffer without that information being available?” I would answer that question in exacting detail before I do any hiring or product selection. Once answered, find an analyst who has demonstrated the ability to repeatedly and satisfactorily provide that information then ask them what if any tool they used to provide that information.
      After that it may be easier to figure out who and what to go with.
      And thanks for reading and commenting. – Joseph

  30. Coel Mekly permalink

    I found these posts scary, like I was standing naked or something. Very right on, from my point of view.

    • Sorry if the information contained herein made you uncomfortable, Coel Mekly, although I'm flattered I was able to document something accurate to your experience.

  31. Fishkin permalink

    Wow this was a great post.. I' m enjoying it.. good resource

  32. Hello, PDejrd. I think subscribing to the RSS feed should do it.

  33. Uto Mat permalink

    Very good article, well written and very thought out.

    • Thank you, Uto,
      I keep meaning to post here again as we've completed a great deal more research. Fortunately, work sometimes gets in the way.
      More to follow, thanks for paying attention.

  34. I responded to the above on Daniel Markus' Kan Web Analytics Haar Beloften Wel Waarmaken? | post with the following comment:
    Hello and thank you for reading my posts. I appreciate much of what youve written and very much like your graphic (although (and I apologize for my poor language skills) Im not sure if youre suggesting Im in the Trough of Disillusionment or I believe the current state of the industry is in the Trough of Disillusionment. Help?)
    People are asking me to write a Part 3 and that was never my plan. Im open to suggestions from you and your readers, however, so please let me know.
    Thanks again,

  35. Thanks for the nod on your blog, Mr. Dlugosch, very enjoyable read. I left a comment, no idea if it's there or lost in the ether.

Trackbacks & Pingbacks

  1. Tweets that mention The Unfulfilled Promise of Online Analytics, Part 1 | The Analytics Ecology --
  2. uberVU - social comments
  3. TwittLink - Your headlines on Twitter
  4. Mediavorous » Blog Archive » Links for November 12th through November 15th
  5. It is the End of the Year as We Know It « Jim Sterne on eMetrics
  6. Trending Upward | Web analytics limitations … and bright future
  7.   Kan Web Analytics Haar Beloften Wel Waarmaken? |
  8. Twitter chatter link for January 16, 2010 MPThree Consulting Inc.
  9. Last Exit Westend » Blog Archive » Limitations in Web Analytics
  10. Congressman Ron Paul’s Official Presidential Campaign Site — — Analyzed | Politics
  11. Determining Social ROI (and Some Proven Techniques for Getting It) - Stating The Obvious
  12. The Unfulfilled Promise of Online Analytics, Part 3 – Determining the Human Cost | The Analytics Ecology
  13. Defining “Definition” and People as “Programmable Entities” | The Analytics Ecology
  14. The Unfulfilled Promise of Online Analytics, Part 2 | The Analytics Ecology
  15. “As Budgets Get Tighter, Probably the First Thing Cut Will Be Web Analytics Because There’s No ROI” - Stating the Obvious
  16. I Can Crack My Knuckles Therefore I Must Be a Chiropractor! (Musings on Expertise) - Stating the Obvious
  17. “It’s too accurate” (more undocumented uses of NextStage’s Evolution Technology) | Triquatrotritecale
  18. The Schism in Analytics, A response to Carrabis, Part I | Christopher Berry
  19. The Schism in Analytics, A response to Carrabis, Part II | Christopher Berry