#214 A/B Testing with A/Bingo

A/B testing, also known as split testing, is a great way to experiment with variations in an application in order to determine which one is the most effective. Below is the sign-up form for an application. At the top of the form is a paragraph that explains what the user needs to do. We want to determine how effective this paragraph is in persuading users to sign up and we can do this by creating an A/B test.

The way this works is as follows: each user that visits the page will either see or not see the paragraph at the top of the page and we can use A/B testing to determine which option yields the better results. To do this we’ll need an event that, when triggered, marks the option as having been successful. In this case the event will fire when a user fills in the form and submits it.

Picking The Right Testing Tool

There are a variety of A/B testing tools available. One we could use is Google’s Website Optimizer. This isn’t Rails-specific, it can be used with any web application even a static site. This tool allows us to provide different URLs for site or use JavaScript to track the conversions for you. This is a nice solution but if you want test a Rails application then you might want to use something that integrates into the application and handles the tracking itself.

One of the most popular Rails-specific solutions is Vanity. This provides some beautiful results and is quite full featured. We won’t be covering it here, though it may be covered in a future episode. It can be a little difficult to set up as it requires a Redis database engine for performance reasons but it’s worth taking a look at to see if it fits your needs.

The plugin we are going to use is A/Bingo. It has a good DSL for defining experiments and it is easy to get started with it. The application we’ll be running the tests on is written with Rails 2 so we can install it by running

script/plugin install git://git.bingocardcreator.com/abingo.git

from our application’s directory. When that has run we’ll need to generate a migration with

script/generate abingo_migration

and then run it with

rake db:migrate

This will generate two new database tables: experiments and alternatives and it’s in these tables that the results of our tests will be stored.

In a production environment it’s best to configure A/Bingo’s caching so that it uses something like memcached but we can skip that here as we’ll just be using the development environment.

Writing The First Test

Now that we have A/Bingo installed we can write the first test. We want to toggle the paragraph at the top of the signup form so that it shows for some users and not for others. We can do this by using the ab_test method.

<% title ("Sign up") %>

<% if ab_test "signup_intro" %>
<p>Complete the form below to create a new user account. You will then be able 
to create projects and tasks, and mark them as complete when finishing them.</p>
<% end %>
<p>Already have an account? <%= link_to "Log in", new_user_session_path %>.</p>

<% form_for @user do |form| %>
 <!-- form omitted -->
<% end %>

The first argument passed to ab_test is the name of the test which in this case is “signup-intro”. If we don’t provide any further arguments this method will randomly return true or false for different users so that the paragraph will be shown or hidden depending on who loads the page.

Next we’ll add a trigger event so that we can track the results. We want the event to be tracked when the user is successfully saved and we can do this by modifying the create action in the UsersController.

def create
  @user = User.new(params[:user])
  if @user.save
    bingo! "signup_intro"
    session[:user_id] = @user.id
    flash[:notice] = "Thank you for signing up. You are now logged in."
    redirect_to root_url
  else
    render :action => 'new'
  end
end

In the create action all we need to do is call the bingo! method and pass it in the name of the test when the user is successfully saved.

If we reload the signup form now we can see that for us ab_test has returned false so we don’t see the paragraph at the top of the page. We can try reloading the page again but the paragraph won’t be shown as our identity is bring remembered (just how will be explained shortly).

If we complete the form and submit it we’ll have signed up successfully and therefore the event will be triggered.

Viewing The Results

It’s currently difficult to see the results of the test but we can do so by creating a controller to view them. To do this we’ll have to generate a controller that we’ll call abingo_dashboard.

script/generate controller abingo_dashboard

Inside this new controller we just need to include the A/Bingo dashboard module.

class AbingoDashboardController < ApplicationController
  # TODO add authorization.
  include Abingo::Controller::Dashboard
end

Obviously if this application was going into production we wouldn’t want everyone to be able to view the dashboard so we’d have to add some authorization to this controller. For now we’ll just leave a comment in there as a reminder.

We also need to add a new route so that we can access the dashboard controller.

map.abingo_dashboard "/abingo/:action/:id", :controller => :abingo_dashboard

With everything in place we can visit http://localhost:3000/abingo and we’ll see the dashboard.

There’s currently no styling applied to the dashboard but it’s easy enough to modify the view templates and style them to fit in with the rest of the application. With the default styling we can still see the results of the experiments so there’s no need to style it now. Looking at the results we can see that we have one experiment with one participant, which was my visit to the signup page, and one conversion which happened when I successfully submitted the form. The paragraph at the top of the form wasn’t showing when we signed up so the participant and conversion are shown in the false (f) results.

Identifying Users

If we visit the signup form again we still won’t see the paragraph above the form because as far as A/Bingo knows our identity is constant. We need to instruct it how to handle user identity and give it a way to differentiate one user from another. This is done inside the application controller by writing a before_filter.

# Filters added to this controller apply to all controllers in the application.
# Likewise, all the methods added will be available for all controllers.

class ApplicationController < ActionController::Base
  helper :all # include all helpers, all the time
  protect_from_forgery # See ActionController::RequestForgeryProtection for details
  before_filter :set_abingo_identity

  private
  def set_abingo_identity
    session[:abingo_identity] ||= rand(10 ** 10)
    Abingo.identity = session[:abingo_identity]
  end
end

In the application controller we’ve added a before filter called set_abingo_identity and in the set_abingo_identity method is the code that determines each user’s identity. The method checks first for the existence of a session variable called abingo_identity and if it doesn’t find one creates it with a random numeric value. This means that as long as a user maintains their session they will always be treated as the same user by A/Bingo and they will consistently either see or not see the paragraph at the top of the signup form.

If we had user authentication in our application we’d want a logged-in user to see the same thing when they were logged in even if their session changed or if they logged in with a different browser or computer. We can change the set_abingo_identity method so that it uses the user’s unique id if the current user is logged in and fall back to the session-based approach for anonymous users.

def set_abingo_identity
  if current_user
    Abingo.identity = current_user.id
  else
    session[:abingo_identity] ||= rand(10 ** 10)
    Abingo.identity = session[:abingo_identity]
  end
end

We can also check to see if the site is being viewed by a web crawler or bot and give each bot the same identity so that the results aren’t skewed by these non-human visitors.

def set_abingo_identity
  if request.user_agent =~ /\b(Baidu|Gigabot|Googlebot|libwww-perl|lwp-trivial|msnbot|SiteUptime|Slurp|WordPress|ZIBB|ZyBorg)\b/i
     Abingo.identity = "robot"
   elsif current_user
     Abingo.identity = current_user.id
   else
     session[:abingo_identity] ||= rand(10 ** 10)
     Abingo.identity = session[:abingo_identity]
   end
end

We check for crawlers and bots by matching their user agent string against a list of known names and if we find a match set the identity to “robot”. This way all of the bots that visit are considered to be a single user.

During the course of writing this application I’ve visited the signup page with a number of different identities and signed up successfully a couple of times. This is now reflected in the A/Bingo dashboard. There are now eight participants, two of who saw the paragraph on the signup page and six who didn’t.

More Complex Tests

We’ll finish this episode by showing you how to provide multiple given options for a single test. The signup_into test is a simple boolean test but if we want to provide more than two different options, for example if we want to provide a number of different page titles, then we can do so.

It’s worth bearing in mind that if we have multiple tests on a single page this can mess with the results as they can be skewed towards those options that show up the most frequently. For a production application this is important but as this is just an example application this doesn’t really matter.

To create a test with multiple options we can call ab_test with two arguments, the second one being an array of the different options we want to test. We want to test three different page titles and we can change the signup page to show one of three different titles by using:

<% title ab_test("signup_title", ["Sign up", "Registration", "Free Sign up"]) %>

With this in place each user that visits the page will see one of these titles at random. This is straightforward enough for something as simple as setting the page title but if you want to use the random value more than once on the page you can do so by using ab_test with a block.

<% ab_test("signup_title", ["Sign up", "Registration", "Free Sign up"]) do |signup_title| %>
<% title signup_title %>
<% end %>

When you try this you may see the error “can’t modify frozen array” if you’re not using the latest version of A/Bingo. Ryan Bates has submitted a patch for this problem which seems to have been added to the codebase so if you see this error try upgrading the plugin and trying again.

When we visit the signup page now we’ll one of the title options and the page will either show or not show the paragraph at the top. There’s one more thing left to do, though. Although we have set up the test in the view, we aren’t recording its success in the controller. We could do this by just adding another call to bingo! in the create action:

def create
  @user = User.new(params[:user])
  if @user.save
    bingo! "signup_intro"
    bingo! "signup_title"
    session[:user_id] = @user.id
    flash[:notice] = "Thank you for signing up. You are now logged in."
    redirect_to root_url
  else
    render :action => 'new'
  end
end

If we have a large number of tests in a given action this can become unwieldy. Instead we can make a single call to bingo! with the name of a conversion and use that conversion name as an extra argument passed to ab_test in the view.

So, we can replace the two bingo! calls with just one in the controller:

def create
  @user = User.new(params[:user])
  if @user.save
    bingo! "signup"
    session[:user_id] = @user.id
    flash[:notice] = "Thank you for signing up. You are now logged in."
    redirect_to root_url
  else
    render :action => 'new'
  end
end

While in the view we can pass the name of the conversion as part of a hash of arguments.

<% ab_test("signup_title", ["Sign up", "Registration", "Free Sign up"], :conversion => "signup") do |signup_title| %>
  <% title signup_title %>
<% end %>

<% if ab_test "signup_intro", nil, :conversion => "signup" %>
<!-- rest of the view -->

Note that as the second test is a simple boolean one we’ve passed nil as the second argument.

That’s it for this episode. A/B testing provides an excellent way to experiment with variations in a web application and track the success of the results. I encourage you to try it in your applications either by using Google Website Optimizer, Vanity or A/Bingo.