Group items tagged

which is just a fancy way of saying we tested all the combinations of buttons and media against each other at the same time.

Every visitor to the splash page was randomly shown one of these combinations and we tracked whether they signed up or not.

The best-performing combination of button and media was “Combination 11″ which was the “Learn More” button and the “Family” image:

The winning variation had a sign-up rate of 11.6%. The original page had a sign-up rate of 8.26%. That’s an improvement of 40.6% in sign-up rate. What does an improvement of 40.6% translate into?

f we hadn’t run this experiment and just stuck with the original page that number would be closer to 7,120,000 signups. That’s a difference of 2,880,000 email addresses.

Lessons Learned Every visitor to your website is an opportunity. Take advantage of that opportunity through website optimization and A/B testing. Question assumptions. Everyone on the campaign loved the videos. All the videos ended up doing worse than all the images. We would have never known had we not questioned our assumptions. Experiment early and often. We ran this experiment in December of 2007 and reaped the benefits for the rest of the campaign. Because this first experiment proved to be so effective we continued to run dozens of experiments across the entire website throughout the campaign.