From TheFutureOf (10 Nov 08): Responding to Jim Novo’s 9 Sept 08 1013am comment

Jim’s Comment

What is the behavior we’re segmenting here?

Visits, and likelihood to Visit again.

So we take some level of Activty – 50 visits – to define “Best Visitors” and then ask, How are we doing with best Visitors, what percent have Visited in the past 60 days? You can segment this by Campaigns, by Content, etc.

For example, a study like this is extremely effective when you launch new Product / Content Categories, or change Policies. What effect did the changes have on Best Visitors? If Best Visitors = 80% of monetization, then that’s the right question to ask. The question that doesn’t give you the right answer is to randomly survey all Visitors – the “majority”, who only contribute 20% of the monetization. They will overrule the Visitors who contribute 80% because there are fewer of these Best Visitors.

Make sense?

The tricky thing with Visitors – especially if you don’t have an e-mail address – is you have to do something *now* when you see certain patterns that historically lead to dis-engagement, for example, does not sign up for RSS feed or newsletter.

If you can hook dis-engagement to something tangible like lack of feed behavior – those who don’t subscribe tend not to come back – you might be able to do something in CMS to address them. The bare bones version of this approach is the old exit pop newsletter subscribe.

If you can’t hook it to something tangible and have to rely on visit patterns, then you’re into something like Joseph’s NextStage.

Either way, using Recency allows you to act more quickly against the testing – you don’t have to wait for a segment to PROVE they’re not coming back, you can PREDICT it.

My Response

Excellently stated. I’m wondering what the max-min number of visitors necessary for “accuracy” is. If I’m reading your http://blog.jimnovo.com/2007/04/25/engagement-customers/ correctly then the min is somewhere on the other side if 15k visitors, correct? And the time variable is about six months?

Also, based on the concepts you’re demonstrating, historical records play heavily in your model, correct? If so, the length of time required to create accurate models is…?

Just asking for better understanding. Also touching back to my previous about replicability and transferability. If you’ve covered this elsewhere then my apologies for not noting it and could you please provide some pointers? Thanks.

I ask because of your caveats “…when you launch new Product / Content Categories, or change Policies.”

I’ll admit to some amusement at “If you can’t hook it to something tangible and have to rely on visit patterns, then you’re into something like Joseph’s NextStage.” Tangible? How are we defining tangible?

And in the end I agree, PREDICTion is what it’s all about.