The last time I ran an A/B test, I learned a very valuable lesson that I think anyone doing landing page testing should know.
The A/B Test
The original landing page had 4 sign up options (Facebook, LinkedIn, Twitter and email). I had a hypothesis that this was extremely confusing to the user and therefore drove down the sign up rate. In addition for the company, a Facebook or LinkedIn sign up was much more valuable due to the information we could collect, so it made sense to encourage those sign up options. I fired up Optimizely and promptly set up a variant of the landing page that just had two sign up options. My variant beat the original and I decided to take my A/B test further by creating variants with just Facebook or LinkedIn.
To my surprise, the LinkedIn only variant beat out the Facebook only and two option variants, by a 15%+ in sign up rate. Satisfied with the results, I made the homepage of the site to have just the LinkedIn sign up option, however, just to be safe I also set up a variant with Facebook only. Once again, I was surprised by the results, this time, the Facebook only variant outperformed the LinkedIn only variant by a decent margin. Upon further evaluation, I realized that the traffic that was being driven to the landing page was very different from traffic that was going to the homepage.
All Traffic Are Not Made Equal
For the landing page, where the LinkedIn only variant won out, all the traffic was paid traffic from Stumbleupon (5 cents per 1000 impressions!). With the purpose of Stumbleupon being being the discovery of beautiful and insightful content, I had anticipated that Facebook sign up would be more appealing, since Facebook is more where the casual internet users dwell. Interestingly, Stumbleupon users much preferred signing up with LinkedIn, perhaps this was because what they were signing up for was for professional development.
Traffic to the homepage was a whole different story, with the traffic being from mainly direct traffic and press. In addition, during the time of the homepage test we had a 300%+ increase of Indonesian users (2nd largest FB market in the world), which favored the Facebook only variant. In the end, for the homepage I decided to keep both sign up options as a way to hedge against the various types of traffic that would hit the homepage.
The Traffic Lesson
If I had simply followed the results of my landing page test without running the same test on the homepage, I would have a lost a ton of sign ups. Make sure the traffic that you are using to A/B test is representative of the traffic of where you will be implementing the results. In order to maximize conversion rates, I would suggest that you run the same A/B tests with different types of traffic.
Sobering Lessons From My 1st Product Building Hackathon
My first product building hackathon last weekend, did not go so hot, in fact, it was...