My Favorite Email A/B Test, Ever
The strategy was good. But the decisions on how to measure impact were even better.
No matter what anybody says, A/B testing is both an art and a science. You might use objective methodologies and statistics to properly confirm a particular result. But there are so many choices that we make every time we run a test.
Not just how many variables to test. Not just the audience sizing for the control vs. variant group along with the corresponding creative. And not just the primary KPIs you are trying to move the needle on.
The art comes down to the following:
The strength of your hypothesis
The strength of your execution
And your decisions on how to measure the impact
The first two bullets above point to your ability to truly understand your audience and your business and the likelihood that you are shipping something that is scaleable and not just a novelty result.
And the last bullet—measurement—is the one most people get wrong.
And with that preamble out of the way, I’m going to share in detail—including creative screenshots, actual % lift, and the measurement methodology—my favorite A/B test I ever shipped at Grammarly.
Setting the Stage
The following are important foundational details before contextualizing the test:
Grammarly utilizes a freemium model on the consumer/B2C side of the house and it’s been that way since 2014
The strong majority of new users sign up for free and stay that way indefinitely since Grammarly has chosen to offer a very solid product for free and it’s not a bait-and-switch to upgrade
The Lifecycle Team (from 2014-early 2019 was just me, but is now around nine FTEs) has two parent focus areas or KPIs for business impact:
Monetize a portion of our free consumer base by getting them to upgrade to a paid subscription
Drive more product engagement and/or retention
For the case study at hand, it was a pure monetization play.
Just like many other brands, Grammarly has an onboarding flow. The flow itself varies depending on customer segment and behaviors. But the core flow is for new, free users who are using the product.
From roughly 2015 to 2018, I had an onboarding flow in place that was driving meaningful results but the truth is that my scope was so huge—all marketing and transactional emails across the entire lifecycle—and I did not experiment with onboarding as much as I would have liked and it really needed some love.
Where was my attention mostly focused on during this time? It was building out our weekly Grammarly Insights program (personalized stats), a monthly promo program, and product updates communications on top of just a growing number of internal servicing requests. In other words, I was focused more so on the ongoing relationships with our ever-growing database and was keeping the onboarding flow on auto-pilot.
But the truth of the matter is that those early parts of the lifecycle provide a lot of leverage and are instrumental in trust-building, brand sentiment, and business impact.
Here’s a birds-eye-view on that 1.0 onboarding flow for free, engaged users:
Day 1 - Welcome Email
Day 2 - Feature Education & X-Sell Email
Day 4 - Upsell to Grammarly Premium (no discount)
Day 10 - Upsell to Grammarly Premium (discount)
Day 12 - Discount Reminder
There were some edge cases for behavioral triggers in this time frame but are not worth getting into here. I should note that these users were also falling into our flagship program during this time - the weekly Grammarly Insights email - which people truly love and is seen more as a core product feature rather than an email.
Anyways, the focus of today’s case study was to try and improve upon results for our Day 4 email - the very first dedicated email that contextualizes our paywall and tries to get these free, engaged users to upgrade at no discount.
Day 4 Upsell Email: Creative Comparison
Keep reading with a 7-day free trial
Subscribe to Scaling CRM to keep reading this post and get 7 days of free access to the full post archives.