Novelty results.
Also known as a false positive, false negative, or even a false tie because the conclusion is made too early or it was just a flawed test setup across the board. And a lot of times it coincides with confirmation bias.
I’ve seen this a lot in the course of my career.
A few common examples:
Launch an updated newsletter design and strategy and do just two-three weeks of side-by-side testing and then make an assumption that the result will scale
Also, you might look at the combined results rather than how things are trending
Update something automated towards the top of the funnel and only look at direct results vs. longitudinal/downstream impact
If you do so, you could be missing out on some very compelling gains (my fav A/B test case study gets into this)
Do offer testing over the course of a few campaigns and make long-term decisions based on which versions win
You get the idea.
Sometimes the hard truth is that when a new thing is introduced it oftentimes conjures up some type of short-term reaction because of the novelty of the thing itself. This does not imply long-term, sustaining results. Even if you have stat sig and 95% confidence.
You are simply proving that at that point in time with those audience cohorts with all the micro and macro conditions that existed when the test occured that you achieved. a specific result.
So, be practical around time-frames and monitoring results over the long-term to see if things shift. And definitely be aware of the relationship b/w upstream and downstream.
Effective Novelty
Novelty as a tool can be quite powerful in growth and marketing.
This time, I’m not talking about a short-term result from a test. I’m talking about conjuring up something so unique, but also contextual to your brand and product that it actually improves important business results and/or sentiment and can become a scalable feature or addition to the overall bag of tricks.
Some Examples:
Spotify Unwrapped
Fitbit creating a new category for a few years when it first launched
Slack leveraging gif generators and Slackbot to improve sentiment and housekeeping
Marvel movies leaning into the post-credit scenes
The not-so-secret menu at In-N-Out Burger
I’ve got another example not from the tech world that may surpise you. It’s from one of the OGs of using novelty to its advantage.
Disney World (and Disney Land/etc) have found countless ways to use gamification and novelty to increase loyalty and engagement.
What do I mean?
Well, there is the core experience of going to one of their parks a few times with your family and all the magic (and overstimulation) that goes with it.
But, like the first season of Westworld, there is the core experience and there are the easter eggs, collectibles, and fan service experiences.
Some examples:
Coin pressing machines at all the park with their own unique designs
Unique, collectble pins are all over the place (Parks and resorts) and they produce new ones every year so there’s never an endpoint for collectors
Same with buttons, merch, treats and pretty much everything they produce
Joffrey’s coffee stands are littered across all the parks and each have their very own / exclusive specialty drink, some of which are only for season pass holders
Private/secret VIP locations for vacation club members in every park
Lots of lore/easter eggs that can become their own mini game when you visit
I’m sure this list could be 5x longer. But you get the point. Disney does not rely soley on the core product experience (aka the rides and character experiences). They continuously find new ways to hook/intrigue/satisfy an itch for people so they keep coming back for more. The rabbit hole is deep (pun intended) and you’ll probably never make it to the center of the maze.
But it’s all in service of the brand and in trying to ensure there’s something for everyone and plenty of reasons to return.
So, remember. When running tests don’t fall into the short-term novelty result trap.
But when creating compelling lifecycle experiences, do pursuse unique game-changing ideas that have the potential to become beloved product experiences.