The Unfulfilled Promise of Online Analytics, Part 3 – Determining the Human Cost

Knowledge will forever govern ignorance, and a people who mean to be their own governors, must arm themselves with the power knowledge gives. A popular government without popular information or the means of acquiring it, is but a prologue to a farce or a tragedy or perhaps both. – James Madison

There was never suppose to be a part 3 to this arc (Ben Robison was correct in that). Part 1 established the challenge (and I note here that the extent of the response and the voices responding indicates that the defined challenge does exist and is recognized to exist) and Part 2 proposed some solution paths. That was suppose to be the end of it. I had fulfilled my promise to myself1 and nothing more (from my point of view) was required.

But many people contacted me asking for a Part 3. There were probably as many people asking for a Part 3 as I normally get total blog traffic. Obviously people felt or intuited that something was missing, something I was unaware of was left out.

But I never intended there to be a Part 3. What to cover? What would be its thematic center?

It was during one of these conversations that I remembered some of the First Principles (be prepared. “First Principles” will be echoed quite a bit in this post) in semiotics.2

According to semiotics, you must ask yourself three questions in a specific order to fully understand any situation3:

  1. What happened?
  2. What do I think happened?
  3. What happened to me?

More verbosely:

  1. Remove all emotionality, all belief, all you and detail what happened (think of quis, quid, quando, ubi, cur, quomodo – the six evidentiary questions applied to life).
  2. What do your personal beliefs, education, training, cultural origins, etc., add to what actually and unbiasedly happened?
  3. Finally, how did you respond — willingly or unwillingly, knowingly or unknowingly, with all of your history and experience — to what happened.

The power of this semioticism is that it forms an equation that is the basis of logical calculus, the calculus of consciousness4, modality engineering5 and a bunch of other fields. I use a simplified form of it in many of my presentations, A + B = C.6

Talking with one first reader, I realized that Part 1 was “What happened?” (the presentation of the research) and Part 2 was “What do I think happened?” (my interpretation of the research). What was left for part 37 was “What happened to me?”

And if you know anything about me, you know I intend to have fun finding out!

All Manner of People Tell Me All Manner of Things

Oliver's TravelsThe above is a line from Oliver's Travels (highly recommended viewing), something said by the Mr. Baxter character. Mr. Baxter is himself a mystery and — although his true nature is hinted at several times — it is not revealed until the last episode. There we are told about The Legend of Hakon and Magnus. In short, Mr. Baxter could be a good guy, a bad guy or the individual directing the good or bad guy's actions. His role entirely depends on what side you are on yourself, a true Rashomon scenario. I found myself in something similar to Mr. Baxter's situation as how people responded to my research, its publication and myself also depended greatly on what side people were on when they contacted me.

I was both dumbfounded and honored by the conversations Parts 1 and 2 generated. The number of people who picked up on or continued the thread on their own blogs (here (and alphabetically) Christopher Berry (and a note that Chris continues the conversation in A Response (The Unfulfilled Promise of Analytics 3) ), Alec Cochrane, Stephane Hamel, Kevin Hillstrom, Daniel Markus, Jim Sterne, Shelby Thayer and if I've forgotten someone, my apologies), twittered it onward, skyped and called me was…I could say unprecedented and remind me to tell you about a psychology convention in the early 1990s (nothing to do with NextStage, just me being me, stating what is now recognized as common knowledge yet way before others decided it was common. Talk about unprecedented results. I had to be escorted out under guard. For those of you who know Dr. Geertz, his comment upon learning this was “I'm not surprised you'd have to be escorted out by guards. You have that subtle way about you…”8).

But to note the joy means to recognize the sorrow (as was done in Reading Virtual Minds Vol. 1: Science and History Chapter VI, “The Long Road Home”). While the majority of people honored me and a good number of people appreciated that I had done some useful research and donated something worth pondering, there were a few (just a few, honestly) who damned me.

The damning per se I don't mind. It's part of the territory. It was the manner and the persons involved that truly surprised me.

I was accused of possibly destroying a marriage (Susanism: If you think this is about you, it's not. We know a lot more people than just you), maligning certain individuals (usually by people who maligned other individuals during the research. I guess I wasn't maligning the correct individuals in their view), not demonstrating the proper respect to industry notables (same parenthetical comment as previous and you guessed it, another NextStage Principle), that I better post an apology to these same industry notables (two people wrote apologies in my name and strongly suggested that I publish them), …


Who gave me such power and authority to make or break people's lives? Certainly I didn't give it to myself, nor did I ask others to give it to me. And if anybody did give it to me without my knowing I gladly give it back. As I've said and written many times, I do research. When new data makes itself available and as required, I update my research. But until such new data comes in, the research stands.

What I really want to know is if, when the results of research are discomforting, the industry's standard and usual procedure is

  • to change either the research or results so that people feel warm and fuzzy — hence have no impetus to act (according to one person at yesterday's NH WAW, “Don't measure what you can't change”. An interesting statement that I disagree with. Doing so means to throw out meteorology, astronomy, … much of what has been historically measured without any change-ability allowed us to create the technologies that would produce change in previously unchangeable systems)
  • or let the discomfiting research stand — so that the challenge can be recognized and either action can be either taken or the challenge go ignored.

Seems to be the “change either the research or results” is the standard (or at least done when required) because while few asked that I rewrite research or results so that certain individuals appeared more favorably, the ones who did ask sure were some high-ranking industry folks.

Heaven forbid these folks wanting different results published or do complimentary research that either validated or invalidated my results.

Wait a second. What am I thinking? Obviously it would be impossible for them to do research that validates mine.9

Of course, publishing research would also mean publishing their methodologies, models, analytic methods, … and the reasons that ain't gonna happen will be covered later in this post.

And if that is the standard and usual procedure — at least among those in the high ranks — then

  • congratulations to all the companies hiring high ranking consultants to make them feel good rather than solve real problems and
  • be prepared for those coming up through the ranks to learn this lesson when it is taught them.

I'm mad as hell and I'm not going to take it anymore!For the record, not much upsets me (ask Susan for a more honest opinion of that). The sheer stupidity of arguments that resort to emotionalism or are nothing more than attempts to protect personalities and positions, though… Them they do offend me (can't wait to learn how our Sentiment Analysis tool reports this). And more about stupidity later in this post (Let me know if you recognize Joseph's “I'm mad as hell and I'm not going to take it anymore” persona).

When the Stories Meet the Numbers (Statistics, Probability and Logic)

I originally surveyed about sixty people for Part 1. That number grew to about one hundred in Part 2 due to responses to Part 1. Currently I've had conversations (I'm counting phone calls, Skype chats and calls, email exchanges and face-to-face discussions at meetings I've attended as “conversations”) with a few hundred people about those posts.

I noticed something interesting (to me) about the conversations I was having. Lots of people made statements about statistics, probability and logic but were using these terms and their kin in ways that were unfamiliar to me. Especially when I started asking people what their confidence levels were regarding their reporting results.

I'll offer that search analysts (I'm including SEO and SEM in “search analysts”) seem to have things much easier than web analysts do. “We were getting ten visits a day, changed our search terms/buy/imaging/engines/… and now we're getting twenty visits per day.” Granted, that's a simplification and it's the heart of search analytics — improving first the volume and second the quality of traffic to a site. Assuming {conversions::traffic-count} has standard variance, search analytics produces or it doesn't and it's obvious either way.

Web analytics, though… “The Official WAA Definition of Web Analytics” is

Web Analytics is the measurement, collection, analysis and reporting of Internet data for the purposes of understanding and optimizing Web usage.

The analytics organization I see most often cited, SEMPO, doesn't even attempt to define (“SEMPO is not a standards body…”) or police (“…or a policing organization.“) itself. It does offer search courses but the goals of the SEMPO courses and the WAA recognized courses are greatly different (an opinion, that, based on reading their syllabi as someone having taught a variety of courses in a variety of disciplines at various educational levels in various educational settings).

There are twenty-one words in the official WAA definition and a philologist will tell you that at least ten require further definition.

Definitions that require definitions worry me. Semiotics and communication theory dictate that the first communication must be instructions on how to build a receiver. Therefore any stated definition that requires further definition is not providing instructions on how to be understood (no receiver can be built because there is no common signal, sign or symbol upon which to construct a receiver. If you've ever read my attempts at French, you know exactly what I mean10).

One of the statements made during the research for this arc was “[online] Analysts need to share the error margins, not the final analysis, of their tools.” It expressed a sentiment shared if not directly stated by a majority of respondents and it truly surprised me. It states as a working model that any final analysis is going to be flawed regardless of tools used therefore standardize on the error margins of the tools rather than the outputs of the tools.

So…decisions should be made based on the least amount of error in a calculation, not what is being calculated (does the math we're using make sense in this situation?), the inputs (basic fact checking; can we validate and verify the inputs?) or the outcome (does the result seem reasonable considering the inputs we gave it and the math we used?)?

A kind of “That calculation says we're going to be screwed 100% but the error margin is only 3% while that other calculation says we're only going to be screwed 22% but the error margin is 10%.

Let's go with the first calculation. Lots less chances of getting it wrong there!”, ain't it?

More seriously, this is a fairly sophisticated mathematical view. Similar tools have similar mathematical signatures when used in similar ways. When a tool has an output of y with fixed input x in one run and y+n with that same fixed input x in another run but a consistent error margin in both runs, standardizing on the error margin e is a fairly good idea. It indicates there's more going on in the noise than you might think.11

Of course, this means you better start investigating that noise darn quick.

My understanding of “statistics, probability and logic” was often at odds with what people were saying when they used those words. The differences were so profound (in some cases) that I asked follow up questions to determine where my misunderstandings were placed.

Serendipity doing it's usual job in my life, over this fall-winter cycle I took on the task of relearning statistics12, partly so I could understand how online analysts were using statistics-based terms. As noted above, the differences between what I understood and how terms were being used and applied was so different that I questioned my understanding of the field and its applications.

And to whither I wander, I offer a philologic-linguistic evidentiary trail for all who will follow. For those who just want to get where I'm going, click here.

Web Analytics is Hard

Of course it is. Anything that has no standards, no base lines, no consistent and accurate methods for comparisons is going to be hard because all milestones, targets and such will have to be arbitrarily set, will have no real meaning in an ongoing, “a = b” kind of way, and therefore Person A's results are actually just as valid as Person B's results because both are really only opinion and the HiPPOs rule the riverbank…

…until a common standard can be decided upon.

Web Analytics is easy

Of course it is. Anything that applies principled logic, consistent definitions, repeatable methodologies that provide consistent results, … is going to be.

Online Analytics Is Whatever Someone Needs It to Be

Ah…of course it is.

And this is the truest statement of the three for several reasons. Consider the statement “(something) is Hard“.

It doesn't matter what that “(something)” is, it can be driving a car, riding a bike, watching TV, playing the oboe, composing poetry, doing online analytics, … . What that “(something)” is is immaterial because the human psyche, when colloquial AmerEnglish is used, assigns greater cognitive resources to understanding “Hard” than it assigns to “Web Analytics”, and this resource allocation has nothing to do with whether or not “Web Analytics” is easier to understand than “Hard”, it has to do with what are called Preparation Sets13. The non-conscious essentially goes into overdrive determining how hard “Hard” is. It immediately throws out things like “iron”, “stone” and “rock” because the sensory systems don't match (iron, stone and rock involve touch-based sensory systems, transitive expressions such as “(something) is hard” don't) and starts evaluating the most difficult {C,B/e,M}14 tasks in memory — most recent to most distant past — to determine if the individual using the term “Hard” is qualified to use the term as a surrogate for the person being told “(something) is Hard” (ie, our non-conscious starts asking “Do they mean what I think they mean when they say 'Hard'?”, “Do they know what 'Hard' is?”, “What do they think 'Hard' means, anyway?”, “Do they mean what I mean when I say 'Hard'?” and so on).15

What I will offer is what I've offered before; any discipline that defines success “on the fly” isn't a discipline at all (at least it's not a discipline as as I understand “discipline”). Lacking evidentiary trails, definitions and numeric discipline, comparisons of outputs and outcomes degenerates to “I like this one better” regardless of reporting frame.

Teach Your Children Well

Where statements like “(something) is Hard” and “(something) is Easy” really make themselves known is when teaching occurs.

Let me give you an example. You have a fear of (pick something. Let's go with spiders because I love them and most people don't (Only click on this link if you love spiders)). Phobias are learned behaviors. This means someone taught you to be afraid of spiders. It's doubtful someone set out some kind of educational curriculum with the goal of teaching you to fear spiders (barring Manchurian Candidate scenarios). It's much more likely that when you were a child, someone demonstrated their fear of spiders to you. Probably either repeatedly or very dynamically, so you learned either osmotically or via imprinting. Children demonstrate their parents' behaviors in hysteresis patterns. This means that if you measured a parent's level of arachniphobia and assigned it a value of 10, chances are the child would demonstrate their arachniphobia at a level of 100 or so in a few years' time. Children who learn their parents' fears and anxieties do so without understanding any logical basis for those fears, only the demonstration of them. When there is no logic to temper the emotional content, hysteria results.

However, if a parent demonstrates a fear response and the ability to control it, to explain to the child that fear response's origin, etc., most often the child learns caution and not fear (not to mention that the parent usually learns to control their fear). The difference can be thought of as the difference between teaching a child to “Be careful” versus hysterically screaming “EEEEK!”

What's so fascinating about this is that it's also how we pass on our core, personality and identity beliefs whether we mean to or not (I cover this in detail in Reading Virtual Minds Volume I: Science and History). We can be teaching physics, soccer, piano, bread-baking, … It doesn't matter because all these activities will be vectors for our core, identity and personal beliefs and behaviors. If we are joyful people then we will teach others to be joyful and the vector for that lesson will be physics, soccer, piano, bread-baking, … And if we are miserable people? Then we will teach others to be miserable and to be so especially when they do physics, play soccer, the piano, bake bread, …

Thus if any teaching/training occurs intentionally or otherwise, the individual doing the training/teaching is going to de facto teach their internal philosophies and beliefs — both business and personal — as well as their methods and practices to their students. This can't be helped. It's how humans function. If the philosophy and belief is that things are hard, then that philosophy and belief will be taught de facto to the students. Likewise for the philosophy and belief that something is easy. There will be no choice.16

The point is we protect others from what we fear. Humans are born with precious few fears hard-wired into us (heights and loud noises are the two most cited. Heights because we're no longer well adapted to an arboreal existence and loud noises because predators tend to make them when they attack).

So the statement “(something) is hard” either means we fear “(something)” or we wish to protect others from having the difficulties we have when we do “(something)”, and if difficulties existed then the non-conscious mind is going to place a fear response around whatever “(something)” is to make sure we don't put ourselves into unnecessary difficulties yet again.

The statement “(something) is easy” generates the polarity of the above and I, dear reader, I am the neuro- and philo-linguist's nightmare because my training is simply that “(something) is”. My training is that both whatever exists and whatever state it exists in are mind of the observer17 dependent. Thus things simply are and our perceptions, experience and decisions make them hard, soft, easy, whatever, to us individually.

It's always all about you, isn't it?More colloquially, whatever your perceptions of the world are, it's all you and precious little of anything else (a favorite quote along these lines is “What if life is fair and we get exactly what we deserve?” Ouch!).

The Trail Leads Here

There are lots of errors I can understand. A lack of knowledge, of mathematical rigor, of logic training, of problem solving skills, … These and a host of others I can appreciate. Especially in those junior to any given discipline.

But unprovable math, a lack of basic fact checking, outputs that have no meaning based on what's come before and (let's not forget) emotionalism? This really blew me away. Math can be taught, junior people who don't fact check can be trained, making sure units match can be taught and comes with experience, … but emotionalism?

I'll accept any of the above in junior players with the caveat that the first to go has got to be emotionalism.

But senior people failing any of these before offering something for publication? Then defending this lack of rigor with an emotional outburst? And when it happens more than once?

Talk about abandoning First Principles!

We don't need no stinking badgesFirst Principles? We don't need no stinking First Principles!

Challenge logic, challenge research, challenge findings, sure. Challenge a person if they challenge you, sometimes maybe. I'll tolerate a lot, folks (ask Susan for confirmation), and I have a real challenge with such as these — Arguing emotionally and telling me it's logic, arguments based on no facts at all… I'll accept, entertain and work with ignorance, arrogance, discomfiture, anxiety, joy, love, appreciation, anger, … quite a wide thrall of human response.

But arguments such as these are, in my opinion, stupid.

There, I typed it.

Yet because such arguments were presented as such I must recognize that in some camps doing web analytics means to heck with fact-checking, logic, … That it's acceptable to ignore truth and common practice to base outcomes on what one needs them to be. I mean, when someone with title and prestige does it, the overt statement is that others should, will or do do it, as well. Definitely people in the same company should or will do it. Whatever's lacking in the master's portfolio won't be found in the student's (in most cases).

Want to know why I stopped attending conferences? See the above.

Joseph, the Abominable Outsider

Joseph, The Abominable Outsider

Stephane Hamel applauded me (I think) when he referenced me as an industry “outsider” in his A nod to Joseph Carrabis: The unfulfilled promise of online analytics. Others used the term to applesauce me. (I was flattered by both, actually.)

I had been wondering if it was worth my writing a little bit on elementary logic, probability theory, problem solving or some such. A previous draft of this post contained an explanation of elementary statistics and problem solving as it might be applied to online analytics. Now I really had to question such an effort. If the notables don't know how to apply these things…

Where the stories meet the numbers, there Understanding dwells

The power of logic, knowing problem solving methods, basic statistics, probability and so on is that they provide basic disciplines that prevent or at least inhibit mistakes such as listed above. You have the tools and training to basically “…draw an XY axes on the paper, chart those numbers and the picture that results points you in the direction you need to go.” You can be emotional about your research and your findings and you can't defend your research emotionally. The research and findings are either valid or they ain't.18

As for drawing an XY axes, charting numbers and getting some direction…what can you do with such evidentiary information? There are lots of things you can do. Determine the relationships between the numbers and you can exploit their meanings.

But if the basics are beyond the industry greats

  • then explaining the differences between cross-sectional studies and longitudinal studies (cross-sectional studies involve measuring a single (x,y) pair, meaning x is fixed for all y. Longitudinal studies involve countably infinite (x,y) pairs. Longitudinal studies are greatly more expensive than their cross-sectional cousins and is why cross-sectional regression models are often used when longitudinal regression models are needed) won't do much good19,
  • nor will explaining the need for creating a “standard” site for calibration purposes,
  • models can only be standardized once methods themselves are analyzed and an accuracy “weighting” is determined (allowing all models to be compared to a “gold standard”, meaning comparing my results to your results actually has analytic meaning),
  • Figuring out where your normals are on your curveexplaining the meaning of and how to “normalize” samples is out (doing so allows you to see where the normals fall on your standard curve. You put your normals in the middle to lower part of the curve because a) this is where population densities are greatest and b) no naturally occuring line is going to be straight so you shoot for placing your normals on the straightest part of the curve to get some kind of linearity (that y = mx + b thing). Every naturally occuring phenomenon follows mathematical rules that produce curves. Between the two blue lines is where standards occur. Below the bottom blue is “below standard”, above the top blue is “out of standard”. Between the bottom blue and green line is the normal range. You calibrate your methods against the gold-standard normals and anything above is where the money lies),
  • 20

It takes more effort to reorder a partially ordered system than it does to create order in an unordered system (bonds, even when incorrect, have existing binding energy).

I completely understand why so many of NextStage's clients couldn't document the accuracy of the online analytics tools they were using at the time they contacted us for help. This lack of documentation was something I was very uncomfortable with. If there's no proven methodology for demonstrating a number's validity then you've essentially moved away from the gold standard and declared that the value of your dollar is based entirely on what others are going to value it at (pretty much determined by your political-military-industrial capabilities or in this case, those guarding the riverbank). Your numbers only have meaning so far as others are willing to accept them as valid and if lots of money is being paid for an opinion, that opinion is going to be gold regardless if it's based on invalid assumptions or documentable facts.

The online analytics field is partially ordered — it's been around long enough for a hierarchy to appear — so only those willing to expend the energy are going to attempt fixing it for the sake of getting it fixed rather than changing it to suit their own objectives.

And this is where

The detritus encounters the many winged whirling object

NSE was seeing so many erroneous tool results (my favorite example was the company that was getting 10k visitors/day and only 3 conversions/month. Their online analyst swore by the numbers) that it lead us to come up with a reliable y = x 2db that we could prove, repeat and document. It relied solely on First Principles. This led to our in-house analytics tools, which is why we're analytics tool agnostic. We really don't care what tools clients use. If we don't believe the numbers we'll use our own tools to determine them because we know and can validate how our tools work. As a result we now often use our tools to validate the accuracy of other tools.

I have no dog in this fight (both the “Web Analytics is…” and whether or not a promise existed and has gone unfulfilled fights because I'm a recognized industry outsider) and won't be dragged into it (I mean, would you really want me involved?). My agenda is making sure that those coming to NextStage for help either bring with them some mathematical rigor or allow NextStage to invoke it. There is little that can be done when a tool lacks internal consistency (given a consistent input it generates different outputs).

It really is that simple, folks. This is First Principles and they always work. Don't believe me? Ask Ockham. First Principles have to work. As long as the sun rises in the east and sets in the west, as long as there are stars up in the sky, as long as the recognized laws of reality are valid, …

And because mathematics is a universal language, the stars are in the sky, etc., etc., these rules have to apply to online analytics and the tools used therein.

Unless you're happy with high variability in results sets given a known and highly defined set of inputs.

Which is fine, if that's what your values are based on.

And I doubt it is, so be prepared for companies to use HiPPOs only for political purposes (“Our methods are valid because they were installed/given to us/updated/validated/… by HiPPO du jour“), not for accuracy purposes.

How fast are you going?I mean, people make a living out of these things, right? When someone talks about a regression curve and that a decision was made because the probabilities were such and so, does it matter if they know what they're talking about?

Or is being able to use a tool the same as understanding what the tool is doing?

And I know there are online analysts out there who take high variability and weave it into gold. Good for them (truly!). They have a skill I lack. And they're performing art, not science, and as someone who walks in both worlds I will share my opinion that science is lots easier than art. Science has rules. Art is governed by what the buying public is willing to spend and on whom.


That offered, HiPPOs du jour should be prepared for highly defined and validatable game-changing methods and technologies to un-du jour them because such methods and technologies will, given time and regardless of where they originate and how they emerge. In this, like stars shining in the sky, there is no option, no way out. The laws of evolutionary dynamics apply in everything from rainstorm puddles on the pavement to galactic clustering (I can demonstrate their validity in the online analytics world very quickly and easily; start with the first online analytics implementation at UoH in the early 1990s and follow the progression to today. Simple, clean and neat. I love it when things work. Don't you? Gives me confidence in what I think, do and say).

My suggestion (note the italics) is that the online community create an unbiased, product agnostic experimental group. All empirical sciences that I know of have experimental disciplines within them (physics has “experimental physics”, immunology has “experimental immunology”, …). NextStage is not part of this community so again, we have no dog in this fight. Let me offer NextStage as an example, though — we do regularly publish our experimental methods and their results in our own papers and in business-science journals and in scientific conference papers. This allows others to determine for themselves if our methods are valid and worthy. Granted, NextStage comes from a scientific paradigm and perhaps taking on some of science's disciplines would benefit the industry as a whole, or at least bring more confidence and comfort to those within it.

But what about the Third Semiotic Question?

Answering “What happened to me?” follows the trail of asking trusted others (my thanks to Susan, Charles, Barb, Mike, Warner, Lewis, Todd, Little-T and the Girls, M, Gladys and Dolph) many questions to bridge holes in my understandings.

All the ills referenced in parts 1 and 2 demonstrated themselves to their full — people who didn't like what I wrote triangulated. They contacted others whom they thought were socially closer to me or “might have an in” but heaven forbid they contact me directly. Others focused their frustration at me because (probably in their minds) I was something concrete and tangible, something they could point at, instead of something they felt powerless against; the industry as a whole. Still others because they consider me an industry leader (I'm not. I'm an outsider, remember? I can't lead an industry I'm not a part of. Or will Moses start telling Buddhists how to behave?). And (I'm told) I became the subject of klatch-talk on at least two continents (obviously, I need to start charging more for my time).

All of these things add up to determining the human cost of the unfulfilled promise of online analytics. As I quoted before, Coca-Cola Interactive Marketing Group Manager Tom Goodie said “Metrics are ridiculously political.” He was correct and not by half. The cost is high. It is highest amongst

  • those unsure of the validity of their methods, their measurements and their meanings who want to be accepted and acknowledged as doing valuable work yet are unable to concisely and consistently document what they're doing to the satisfaction of executives signing their checks
  • and those who are cashing those checks to buy new clothes.

Do I think online analytics industry will change because of my research and its publication?

See this tool? I must know what I'm doing because I use this tool.Did you read what I wrote about accountability in The Unfulfilled Promise of Online Analytics, Part 1? People are being paid without being accountable for what they're being paid to do. The sheer human inertia put forth to not change that model has got to be staggering, don't you think?

And I doubt anything I could do would bring such a change about. My work may contribute, it may be a drop in the bucket helping that bucket to fill and that's all.

The industry itself will change regardless (surprise!). As a WAWB colleague recently wrote, “For a field that's changing rapidly, based on rapidly changing technologies, I personally feel that holding any expectations for the future is a set up for disappointment. The expectation of change is the only realistic expectation I can hold today.” and I agree. Things will change. They always do. To promise anything else is to lie first to one's self then to others.

Final Thoughts

This is the end of the Unfulfilled Promise arc for me, folks. Please feel free to continue it on your own and give me a nod if you wish.

(my thanks to readers of Questions for my Readers who suggested this footnoting format over my usual <faux html> methods and to participants in the First NH WAW who, knowing nothing about this post, covered much the same topics during our lunch conversation)

1 – A constant promise to myself regarding my work — perform honest research, report results accurately and unbiasedly and (when possible) determine workable solutions to any challenges that presented themselves in either research or results.


2 – For those who don't know, much of ET is based on anthrolingualsemiotics — how humans communicate via signs. “Signs” means things like “No Parking”, true, and also means language, movement, symbols, art, music, … . According to Thomas Carlyle, it is through such things “that man consciously or unconsciously lives, works and has his being.” You can find more about semiotics in the following bibliography:

Aho, Alfred V. 2004 27 Feb Software and the Future of Programming Languages, .Science V 303 , I 5662 , DOI: 10.1126/science.1096169

Balter, Michael 2004 27 Feb Search for the Indo-Europeans, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1323

Balter, Michael 2004 27 Feb Why Anatolia?, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1324

Benson J.; Greaves W.; O'Donnell M.; Taglialatela J. 2002 Evidence for Symbolic Language Processing in a Bonobo (Pan paniscus), .Journal of Consciousness Studies V 9 , I 12

Bhattacharjee, Yudhijit 2004 27 Feb From Heofonum to Heavens, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1326

Carrabis, Joseph 2006 Chapter 4 “Anecdotes of Learning”, Reading Virtual Minds Volume I: Science and History, V 1 , Northern Lights Publishing , Scotsburn, NS 978-0-9841403-0-5

Carrabis, Joseph 2006 Reading Virtual Minds Volume I: Science and History, V 1 , Northern Lights Publishing , Scotsburn, NS

Chandler, Daniel 2007 Semiotics: The Basics, , Routledge 978-0415363754

Crain, Stephen; Thornton, Rosalind 1998 Investigations in Universal Grammar, , MIT Press 0-262-03250-3

Fitch, W. Tecumseh; Hauser, Marc D. 2004 16 Jan Computational Constraints on Syntactic Processing in a Nonhuman Primate, .Science V 303 , I 5656

Gergely, Gyorgy; Bekkering, Harold; Kiraly, Ildiko 2002 14 Feb Rational imitation in preverbal infants, .Nature V 415 , I 6873 , DOI:

Graddol, David 2004 27 Feb The Future of Language, .Science V 303 , I 5662 , DOI: 10.1126/science.1096546

Holden, Constance 2004 27 Feb The Origin of Speech, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1316

Montgomery, Scott 2004 27 Feb Of Towers, Walls, and Fields: Perspectives on Language in Science, .Science V 303 , I 5662 , DOI: 10.1126/science.1095204

Pennisi, Elizabeth 2004 27 Feb The First Language?, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1319

Pennisi, Elizabeth 2004 27 Feb Speaking in Tongues, .Science V 303 , I 5662 , DOI: 10.1126/science.303.5662.1321


3 – There is (in my opinion) no greater demonstration of this principle than in The Book of the Wounded Healers, a long forgotten book that I hope will become available again sometime soon.


4 – Aleksander, Igor; Dunmall, Barry 2003 Axioms and Tests for the Presence of Minimal Consciousness in Agents I: Preamble, .Journal of Consciousness Studies V 10 , I 4-5


5 – Carrabis, Joseph 2004, 2006, 2009 A Primer on Modality Engineering, 18 Pages, , Northern Lights Publishing , Scotsburn, NS

Carrabis, Joseph 2009 18 Aug I'm the Intersection of Four Statements, , BizMediaScience

Carrabis, Joseph 2009 8 Sep Addendum to “I'm the Intersection of Four Statements”, , BizMediaScience

Nabel, Gary J. 2009 2 Oct The Coordinates of Truth, .Science V 326 , I 5949


6 – The simplest things often have the most power. The semioticist's A + B = C demonstrates itself with three questions to form equations of meaning such as:

(what happened) + (what do I think happened) = (what happened to me)

(what happened to me) – (what do I think happened) = (what happened)

(what happened to me) – (what happened) = (what do I think happened)

Know any two and the last reveals itself to you.

But only if you're willing.


7 – Note to Jacques Warren: Un et un est troi. Ha!


8 – Note to Ben Robison: Nope, ET wouldn't detect the sarcasm. The string was too short. We're working on it.


9 – Note to Ben Robison: Still working on that sarcasm thing. We have what we think is a good go at it in the NS Sentiment Analysis tool we'll be making public either this week or next (still waiting for the interface and may decide to go without it just to learn what happens).


10 – As Jacques Warren, Stephane Hamel and Rene can tell you, my best French is laughable. My attempt at “My gosh, what a beautiful day” usually comes out as “Joli jour heureux je”. (C'est rire, n'est-ce pas?)


11 – Carrabis, Joseph 2007 10 Jan Standards and Noisy Data, Part 1, , BizMediaScience

Carrabis, Joseph 2007 11 Jan Standards and Noisy Data, Part 2, , BizMediaScience

Carrabis, Joseph 2007 12 Jan Standards and Noisy Data, Part 3, , BizMediaScience

Carrabis, Joseph 2007 14 Jan Standards and Noisy Data, Part 4, , BizMediaScience

Carrabis, Joseph 2007 27 Jan Standards and Noisy Data, Part 5, , BizMediaScience

Carrabis, Joseph 2007 28 Jan Standards and Noisy Data, Part 10, , BizMediaScience

Carrabis, Joseph 2007 28 Jan Standards and Noisy Data, Part 6, , BizMediaScience

Carrabis, Joseph 2007 28 Jan Standards and Noisy Data, Part 7, , BizMediaScience

Carrabis, Joseph 2007 28 Jan Standards and Noisy Data, Part 8, , BizMediaScience

Carrabis, Joseph 2007 28 Jan Where Noisy Data Meets Standards (The Noisy Data arc, Part 9), , BizMediaScience

Carrabis, Joseph 2007 29 Jan For Angie and Matt, and The Noisy Data Finale, , BizMediaScience

Carrabis, Joseph 2007 29 Jan Standards and Noisy Data, Part 11, , BizMediaScience


12 – Periodic relearnings are part of my training and makeup. I put myself through periodic re-educations because I question my knowledge, not because I question someone else's. My goal is to find the flaws in my understanding, not to pronounce someone else's in error. Periodic re-educations keep subject matter knowledge fresh within me, brings new understandings to old educations, increases wisdom, all sorts of good things. Admittedly, this has enabled me to recognize flaws in other people's reasonings. Two examples that the online community may be familiar with are Eric Peterson's engagement equation (flawed definitions and mathematical logic) and Stephane Hamel's WAMM (frame confusion).

To respond to some comments made on the (now dead) TheFutureOf blog, I had to study other people's work. One such work was Eric Peterson's engagement equation. Other people had contacted me about his equation with some questions about it's validity (for the record, I had no intention of looking at Eric's engagement equation until he mentioned it in response to something I'd written. Once he mentioned it, my belief was he'd “placed it in the game” so to speak, hence opened it up to inspection).

In any case, the result of my own and others' questioning was that I studied how that equation was derived (was the mathematical logic viable and consistent, were the variables defined and used consistently, …) and found it flawed. Eric asked if it would be possible for us to simply work together on the equation to remove some ambiguities and make it more generally applicable, thereby removing any questions of mathematical validity and provide business value.

The public response to my reworking of Eric's original equation both confused and concerned me. My reworking was nothing more than turning it into a multiple regression model with the b0 and e terms set to 0 and all bn assumed to 1 (they could be changed as needs dictated). This allowed people using the reworking to determine by simple variance which models/methods weren't valid in their business setting and ignore them. I kept thinking people would laugh at how simplistic my reworking was and the response was quite the opposite. It was at this point my concerns about basic mathematical knowledge among online analysts flared.

I read through Stephane Hamel's WAMM paper (also because others entered it into a discussion) and recognized that by adding some consistent variable definitions that tool would have a great deal of power across disciplines. I asked Stephane if he'd mind my tinkering and so the story goes.

The challenge with Eric Peterson's engagement equation and Stephane Hamel's WAMM is (in my current understanding) that there is no “standard”, itself a theme I'll return to in this post. As an example, my current work with WAWB involves applying some standard modeling techniques so a “normal” can be determined. This would allow Company A to measure itself against a normal rather than comparing itself to bunches of other companies (that might not be good exemplars based on differing business and market conditions) and determine upon which vector Company A should place its efforts to insure cost-efficient gains along all WAMM vectors. The first aspect (my opinion) would be organizational. Without people accepting recognized truth there is no truth (again, my opinion).

And each time I take on such a task I require myself to relearn the necessary disciplines so I can be confident that my understandings are as close to the original author's as possible.

My method for learning and re-learning anything is to go back to First Principles (as mentioned earlier in this post). Some people may have heard or seen me talk about learning theory and how it can be applied everywhere. That's a lot of what First Principles are about. Start with the most basic elements you can, understand them as completely as possible, build upon that. One thing this provides me is the ability and confidence to discuss my ideas openly, the freedom to ask questions honestly and truthfully, and to understand and accept conflicting views easily and graciously. Put another way, the more you know, the wider your field of acceptance and understanding, and the more fluid and dynamic you become in your ability to respond to others.

So I started relearning statistics by going back to First Principles, studying Gauss, Galton, Fisher and Wright, giving myself the time to understand how the discipline evolved, how the concepts of regression, regression to the mean, ANOVA, ANCOVA, trait analysis, path analysis, structural equations modeling, causal analysis, least squares analysis, …, came about, how they're applied to different sciences (agriculture, eugenics, medicine, …), how bias, efficiency, optimality, sufficiency, ancillarity, robustness, … came about and how they are solved.

I also learned that the advent of fast, inexpensive computing power tended to focus people's attentions to problems that could be solved via fast, inexpensive computing rather than problems that needed to be solved. This was (to me) a point of intersection with the Unfulfilled Promise posts; “gathered data that [we] knew how to gather rather than asking what data would be useful to gather and figuring out how to gather it.”

So I shifted my focus a bit. I decided to use online analytics as the groundwork for teaching myself statistics.


13 – Somebody remind me to publish The Augmented Man. It covers Preparation Sets, EEGSLs and all that stuff in detail.

And it's another darn good read. Phphttt!


14 – Carrabis, Joseph 2006 Chapter 2, “What The Reading Virtual Minds Series Is About”, Reading Virtual Minds Volume I: Science and History, , Northern Lights Publishing , Scotsburn, NS 978-0-9841403-0-5

Carrabis, Joseph 2006 Chapter 4 section 2, “The Investors Heard the Music”, Reading Virtual Minds Volume I: Science and History, V 1 , Northern Lights Publishing , Scotsburn, NS 978-0-9841403-0-5

Carrabis, Joseph 2006 10 Nov Mapping Personae to Outcomes,

Carrabis, Joseph 2007 23 Mar Websites: You've Only Got 3 Seconds, , ImediaConnections

Carrabis, Joseph 2007 11 May Make Sure Your Site Sells Lemonade, , iMediaConnections

Carrabis, Joseph 2007 29 Nov Adding sound to your brand website, , ImediaConnections

Carrabis, Joseph 2008/9 28 Jan/1 Jul From TheFutureOf (22 Jan 08): Starting the discussion: Attention, Engagement, Authority, Influence, , , The Analytics Ecology

Carrabis, Joseph 2008 26 Jun Responding to Christopher Berry's “A Vexing Problem, Part 4” Post, Part 3, , BizMediaScience

Carrabis, Joseph 2008 2 Jul Responding to Christopher Berry's “A Vexing Problem, Part 4” Post, Part 2, , BizMediaScience

Carrabis, Joseph 2008/9 11 Jul/3 Jul From TheFutureOf (10 Jul 08): Back into the fray, , The Analytics Ecology

Carrabis, Joseph 2008/9 18 Jul/7 Jul From TheFutureOf (16 Jul 08): Responses to Geertz, Papadakis and others, 5 Feb 08, , The Analytics Ecology

Carrabis, Joseph 2008/9 18 Jul/7 Jul From TheFutureOf (16 Jul 08): Responses to Papadakis 7 Feb 08, , The Analytics Ecology

Carrabis, Joseph 2008/9 29 Aug/9 Jul From TheFutureOf (28 Aug 08): Response to Jim Novos 12 Jul 08 9:40am comment, , The Analytics Ecology

Carrabis, Joseph 2008 1 Oct Do McCain, Biden, Palin and Obama Think the Way We Do? (Part 1), , BizMediaScience

Carrabis, Joseph 2008 6 Oct Do McCain, Biden, Palin and Obama Think the Way We Do? (Part 2), , BizMediaScience

Carrabis, Joseph 2008 30 Oct Me, Politics, Adam Zand's Really Big Shoe, How Obama's and McCain's sites have changed when we weren't looking, , BizMediaScience

Carrabis, Joseph 2008 31 Oct Governor Palin's (and everybody else's) Popularity, , BizMediaScience

Carrabis, Joseph 2008/9 10 Nov/15 Jul From TheFutureOf (7 Nov 08): Debbie Pascoe asked me to pontificate on What are we measuring when we measure engagement?, , The Analytics Ecology

Carrabis, Joseph 2009 A Demonstration of Professional Test-Taker Bias in Web-Based Panels and Applications, 20 Pages, , NextStage Evolution , Scotsburn, NS

Carrabis, Joseph 2009 Machine Detection of and Response to User Non-Conscious Thought Processes to Increase Usability, Experience and Satisfaction – Case Studies and Examples, . , Towards a Science of Consciousness: Hong Kong 2009, University of Arizona, Center for Consciousness Studies , Tucson, AZ

Carrabis, Joseph 2009 5 Jun Sentiment Analysis, Anyone? (Part 1), , BizMediaScience

Carrabis, Joseph 2009 12 Jun Canoeing with Stephane (Sentiment Analysis, Anyone? (Part 2)), , BizMediaScience

Carrabis, Joseph; 2007 30 Mar Technology and Buying Patterns, , BizMediaScience

Carrabis, Joseph; 2007 9 Apr Notes from UML's Strategic Management Class – Saroeung, 3 Seconds Applies to Video, too, , BizMediaScience

Carrabis, Joseph; 2007 16 May KBar's Findings: Political Correctness in the Guise of a Sandwich, Part 1, , BizMediaScience

Carrabis, Joseph; 2007 16 May KBar's Findings: Political Correctness in the Guise of a Sandwich, Part 2, , BizMediaScience

Carrabis, Joseph; 2007 16 May KBar's Findings: Political Correctness in the Guise of a Sandwich, Part 3, , BizMediaScience

Carrabis, Joseph; 2007 16 May KBar's Findings: Political Correctness in the Guise of a Sandwich, Part 4, , BizMediaScience

Carrabis, Joseph; 2007 Oct The Importance of Viral Marketing: Podcast and Text, ,

Carrabis, Joseph; 2007 9 Oct Is Social Media a Woman Thing?, ,

Carrabis, Joseph; Bratton, Susan; Evans, Dave; 2008 9 Jun Guest Blogger Joseph Carrabis Answers Dave Evans, CEO of Digital Voodoos Question About Male Executives Weilding Social Media Influence on Par with Female Executives, , PersonalLifeMedia

Carrabis, Joseph; Carrabis, Susan; 2009 Designing Information for Automatic Memorization (Branding), 35 Pages, , NextStage Evolution , Scotsburn, NS

Carrabis, Joseph; 2009 Frequency of Blog Posts is Best Determined by Audience Size and Psychological Distance from the Author, 25 Pages, , NextStage Evolution , Scotsburn, NS

Daw, Nathaniel D.; Dayan, Peter 2004 18 Jun Matchmaking, .Science V 304 , I 5678

Draaisma, Douwe 2001 8 Nov The tracks of thought, .Nature V 414 , I 6860 , DOI:

Ferster, David 2004 12 Mar Blocking Plasticity in the Visual Cortex, .Science V 303 , I 5664

Harold Pashler; Mark McDaniel; Doug Rohrer; Robert Bjork 2008 Learning Styles: Concepts and Evidence, .Psychological Science in the Public Interest V 9 , I 3 1539-6053 %+ University of California, San Diego; Washington University in St. Louis; University of South Florida; University of California, Los Angeles

Hasson, Uri; Nir, Yuval; Levy, Ifat; Fuhrmann, Galit; Malach, Rafael 2004 12 Mar Intersubject Synchronization of Cortical Activity During Natural Vision, .Science V 303 , I 5664

Kozlowski, Steve W.J.; Ilgen, Daniel R. 2006 Dec Enhancing the Effectiveness of Work Groups and Teams, .Psychological Science in the Public Interest V 7 , I 3 , DOI:

Matsumoto, Kenji; Suzuki, Wataru; Tanaka, Keiji 2003 11 Jul Neuronal Correlates of Goal-Based Motor Selection in the Prefrontal Cortex, .Science V 301 , I 5630

Ohbayashi, Machiko; Ohki, Kenichi; Miyashita, Yasushi 2003 11 Aug Conversion of Working Memory to Motor Sequence in the Monkey Premotor Cortex, .Science V 301 , I 5630

Otamendi, Rene Dechamps; Carrabis, Joseph; Carrabis, Susan 2009 Predicting Age & Gender Online, 8 Pages, , NextStage Analytics , Brussels, Belgium

Otamendi, Rene Dechamps; 2009 22 Oct NextStage Announcements at eMetrics Marketing Optimization Summit Washington DC, , NextStage Analytics

Otamendi, Rene Dechamps; 2009 24 Nov NextStage Rich PersonaeTM classification, , NextStage Analytics

Paterson, S. J.; Brown, J. H.; Gsdl, M. K.; Johnson, M. H.; Karmiloff-Smith, A. 1999 17 Dec Cognitive Modularity and Genetic Disorders, .Science V 286 , I 5448

Pessoa, Luiz 2004 12 Mar Seeing the World in the Same Way, .Science V 303 , I 5664

Richmond, Barry J.; Liu, Zheng; Shidara, Munetaka 2003 11 Jul Predicting Future Rewards, .Science V 301 , I 5630

Sugrue, Leo P.; Corrado, Greg S.; Newsome, William T. 2004 18 June Matching Behavior and the Representation of Value in the Parietal Cortex, .Science V 304 , I 5678

Tang, Tony Z.; DeRubeis, Robert J.; Hollon, Steven D.; Amsterdam, Jay; Shelton, Richard; Schalet, Benjamin 2009 1 Dec Personality Change During Depression Treatment: A Placebo-Controlled Trial, .Arch Gen Psychiatry V 66 , I 12


15 – And before I get another flurry of emails that I'm attacking one person or another, no, I'm not. An almost identical process occurs when someone says “(something) is Easy”. I describe the “(something) is Hard” version because it's easier for people to understand. One of the wonders of AmerEnglish and American cultural training, that — it is easier to accept that something can be hard and harder to accept that something could be easy.

Human neural topography. Gotta love it.


16 – This understanding of what happens during teachings and trainings is why all NextStage trainings are done the way they are (see Eight Rules for Good Trainings (Rules 1-3) and Eight Rules for Good Trainings (Rules 4-8)) and could be why our trainings get the responses they do (see Comments from Previous Participants and Students).


17 – Bloom, Paul 2001 Precis of How Children Learn the Meanings of Words, .Behavioral and Brain Sciences V 24

Burnett, Stephanie; Blakemore, Sarah-Jayne 2009 6 Mar Functional connectivity during a social emotion task in adolescents and in adults, .European Journal of Neuroscience V 29 , I 6 , DOI: 10.1111/j.1460-9568.2009.06674.x

Frith, Chris D.; Frith, Uta 1999 26 Nov Interacting Minds–A Biological Basis, .Science V 286 , I 5445

Gallagher, Shaun 2001 The Practice of Mind (Theory, Simulation or Primary Interaction), .Journal of Consciousness Studies V 8 , I 5-7

Senju, Atsushi; Southgate, Victoria; White, Sarah; Frith, Uta 2009 14 Aug Mindblind Eyes: An Absence of Spontaneous Theory of Mind in Asperger Syndrome, .Science V 325 , I 5942

Tooby, J.; Cosmides, L. 1995 'Foreward' to S. Baron-Cohen, “MindBlindness: An Essay on Autism and Theory of Mind”, . , MIT Press , Cambridge, Mass.

Zimmer, Carl 2003 16 May How the Mind Reads Other Minds, .Science V 300 , I 5622


18 – I'll use myself as an example. I've often become emotional when talking about research and results. But (But!) regardless of my emotionalism, the work stands or doesn't. I can clarify, elucidate, explain, divulge, describe, … and in the end, the work stands or it doesn't.


19 – If your model is a linear variation (all regression analyses are linear in nature) then you have something like y = mx + b, y = b0 + b1x + e, … and every change in one unit of x will cause a one unit change in y. Using the above equations as examples we get the textbook definition of the regression coefficient (either m or b1 in the above); the effect that a one unit change in x has on y.


20 – I have experience working with large data sets. Some of you might know I worked for NASA in my younger years. I was responsible for downloading and analyzing satellite data. The downloads came every fifteen minutes and reported atmospheric phenomena the world over. My job was to catch the incongruous data and discard it. I got to a point where I could look at this hexidecimal data stream and determine weather conditions any where in the world before it got sent on for analysis.

Amazing that I got dates back then, isn't it?


Posted in , , , , ,

2 thoughts on “The Unfulfilled Promise of Online Analytics, Part 3 – Determining the Human Cost

Comments are closed.