#315 Rollout and Degrade pro

In this episode we’ll show you how to roll out a new feature to a subset of users so that they can try it before it goes out live to everyone. The application we’ll be using for this is shown below. It’s a simple to-do list app that shows a list of tasks and which has a text box for adding a new task.

This application also has some basic authentication so that we can log in or switch users. Currently that’s all there is to the app but we do have a new feature waiting in a new Git branch. We’re currently in the master branch but we can checkout the branch with the new feature in it, which is called phone, by running git checkout.

$ git checkout phone
Switched to branch 'phone'

When we reload the page now it has a “Send to Phone” link at the bottom. If we click this link we’re taken to a new page where we can enter a phone number.

Entering a phone number on this page and clicking the button would theoretically send our list to a phone number as a text message.

Rolling Out Features With Rollout

This feature is difficult to test fully in development so we’re going to roll it out slowly to production starting with a subset of users. There’s a Ruby gem called Rollout that can help us to do this. This uses a Redis backend to manage the users that can see each feature so the first thing we’ll need to do it install that. If you’re using OS X the easiest way to do this is with Homebrew.

$ brew install redis
``` terminal

<p>Next we&rsquo;ll need to add the Redis and Rollout gems to our application and run <code>bundle</code> to install them.</p>

``` /Gemfile
source 'http://rubygems.org'

gem 'rails', '3.1.3'

gem 'sqlite3'


# Gems used only for assets and not required
# in production environments by default.
group :assets do
  gem 'sass-rails',   '~> 3.1.5'
  gem 'coffee-rails', '~> 3.1.1'
  gem 'uglifier', '>= 1.0.3'
end

gem 'jquery-rails'

gem 'bcrypt-ruby'
gem 'redis'
gem 'rollout'

We set up Rollout in a new initializer file.

$redis = Redis.new
$rollout = Rollout.new($redis)
$rollout.define_group(:admin) do |user|
  user.admin?
end

The documentation recommends using global variables and we’ve used them here to set up a new instance of Redis and a new Rollout instance which uses that Redis instance. We can configure Rollout in this file, too and we’ve defined a group called admin. The define_group method takes a block that accepts a user object and which should return a boolean value based on whether that user is a member of that group. For our admin group we check that the admin boolean column is true for that user.

Now that we have Rollout set up we can use it to define the portions of the application that we want to roll our slowly. For example we want only a select group to see the “Send to Phone” link. We can do this by wrapping the link in a if clause so that it’s only displayed if we have it specified in Rollout.

<% if $rollout.active? :phone, current_user %>
  <p><%= link_to "Send to Phone", new_phone_request_path %></p>
<% end %>

We do this by calling $rollout.active?, passing it a the name of a feature and the current user. The link will now only be displayed if we’ve selected to show that feature to that user. Before we reload the page we’ll need to restart the server and ensure that Redis is running. To start Redis we run:

$ redis-server /usr/local/etc/redis.conf

Now when we reload the page the link has gone as by default all features are off.

Next we need to tell Rollout to activate this feature for a given subset of users. We’ll do this through the console but we could do it through a Rake task or even a Web user interface.

1.9.2p290 :001 > $rollout.activate_group(:phone, :admin)
 => true 

To activate a feature we call activate_group on the $rollout variable we defined in the initializer and pass it the name of the feature we want to activate and the group we want to activate it for. If we reload the page now the link has gone again but it we login as an admin user we’ll see it again.

If we want to activate a feature for every user we can use the group :all as this group is set up by default to include all users.

1.9.2p290 :001 > $rollout.activate_group(:phone, :all)

If we want to activate a feature for an individual user we can use activate_user.

1.9.2p290 :002 > $rollout.activate_user(:phone, User.find_by_name('eifion'))

As we’re currently logged in as that user we can see the “Send to Phone” link now if we refresh the page.

We can activate a percentage of users by calling activate_percentage and passing in a feature and a number.

1.9.2p290 :003 > $rollout.activate_percentage(:phone, 20)

This will activate our feature for 20% of users.

Each of these methods has an equivalent deactivate method for removing that activation. If we want to deactivate a feature completely we can use deactivate_all.

1.9.2p290 :003 > $rollout.deactivate_all(:phone)
 => 0 

Restricting Access To Actions

We’re back where we started now and we don’t have access to the “Send to Phone” link. If we know the URL, though, we can access the page directly. We can disable this in the controller.

class PhoneRequestsController < ApplicationController
  before_filter :authenticate

  def new
  end

  def create
    # Send to do list to phone number here
    redirect_to tasks_url, notice: "Sent list to #{params[:phone_number]}"
  end
end

The controller already has a before_filter for managing authentication; we’ll add another one handling the rollout so that only users who have access to this feature can view these action.

class PhoneRequestsController < ApplicationController
  before_filter :authenticate
  before_filter :rollout

  def new
  end

  def create
    # Send to do list to phone number here
    redirect_to tasks_url, notice: "Sent list to #{params[:phone_number]}"
  end
  
  private
  def rollout
    unless $rollout.active? :phone, current_user
      redirect_to root_url, alert: "Access denied"
    end
  end
end

The code in the new rollout method is essentially the same that we have in the view where we check that the current user can access the phone feature. If they aren’t we’ll redirect them back to the home page. If we visit http://localhost:3000/phone_requests/new now we’ll be redirected as currently no-one has access to the phone feature.

Tidying Up

We’ll be calling $rollout.active a lot as we develop our application so it’s a good idea to move it into a new rollout? method in our ApplicationController. We can then call this method elsewhere by just calling rollout?.

def rollout?(name)
  $rollout.active? name, current_user
end
helper_method :rollout?

We can now modify the code in the PhoneRequestsController so that it uses our new method.

def rollout
  unless rollout? :phone
    redirect_to root_url, alert: "Access denied"
  end
end

As we’ve made rollout? a helper method we can modify the view code, too.

<% if rollout? :phone %>
  <p><%= link_to "Send to Phone", new_phone_request_path %></p>
<% end %>

Sometimes it can be difficult to find all of the changes we’ve made to a codebase for a given feature so that it can be wrapped in a rollout condition. This is where Git branches come in handy. If we keep a separate branch for each feature we can call git diff master to find the differences in the codebase between a given branch and the master branch. We can then wrap these differences in a rollout condition so that they only appear if the user has access to them.

Degrade

That wraps up how to use the Rollout gem to enable features for a specific subset of users. There’s another gem that goes hand-in-hand with it called Degrade, also written by James Golick. This gem allows us to automatically disable features when exceptions occur.

For example, let’s say that while our new Send to Phone feature is sending the list it throws an exception. Maybe too many people are using the feature at once and the SMS gateway is beginning to creak under the strain. It would be nice if we could automatically disable this feature and fall back to the old behaviour in these circumstances. The Degrade gem helps us to do this.

As with Rollout, the first step to adding Degrade to an application is to add it to the gemfile and run bundle.

gem 'degrade'

In the initializer we created for Rollout we can set up Degrade.

$degrade_phone = Degrade.new($redis,
        name: :phone,
        minimum: 1,
        failure_strategy: lambda { $rollout.deactivate_all(:phone)}
)

We’ve created a global degrade_phone variable here and assigned it to a new instance of Degrade. As well as passing in our Redis instance we’ve specified some other options such as the name of the feature. We’ve set the minimum option to 1 so that we can test this, as it defaults to 100. This is the number of times that the code should be run before the failure rate is checked. The failure_strategy option is a lambda which is triggered when the failure rate is exceeded. In it we disable the “Send to Phone” feature.

Now we have this set up we can call perform on this Degrade instance and pass it a block and put our feature in there. If an exception is raised inside the block it will go through that instance. In our case the feature is handled inside two controller actions and we can use an around filter to handle them.

class PhoneRequestsController < ApplicationController
  before_filter :authenticate
  before_filter :rollout
  around_filter :degrade

  def new
  end

  def create
    # Send to do list to phone number here
    redirect_to tasks_url, notice: "Sent list to #{params[:phone_number]}"
  end
  
  private
  def rollout
    unless rollout? :phone
      redirect_to root_url, alert: "Access denied"
    end
  end

  def degrade
    $degrade_phone.perform { yield }
  end
end

The around filter is called before an action is triggered and will then yield to that action so in the degrade method we get the $degrade_phone variable that we created in the initializer, call perform on it and pass it yield. This will yield to the action so that it’s executed inside the perform block and any exceptions raised are handled through it. To test this out we’ll raise an exception in the PhoneRequestsController’s create action.

def create
  # Send to do list to phone number here
  raise "foo"
  redirect_to tasks_url, notice: "Sent list to #{params[:phone_number]}"
end

We can try this feature out in the browser now. As we’ve modified an initializer we’ll need to restart the server and we’ll need to re-enable the feature, too. Once we have we can visit the Send to Phone page, enter a phone number and press the button. When we do we see the exception.

If we reload the page now we’ll see the “Access Denied” message as Degrade will have disabled the feature. Note that the link has now gone from the page too as this is part of the feature.

We should improve this error message so that its more user-friendly but the feature does now work.

A Custom Solution

Both of the Rollout and Degrade gems work really well but what should we do if we want a more customized solution? Maybe we don’t want to use a Redis backend or our application doesn’t have a User model. In these cases we can create similar functionality from scratch and it’s not too difficult to do so. The Rollout gem itself is less than 100 lines of code. Below is some Ruby code that recreates the Rollout feature from scratch so that we can get an idea as to what is involved.

# Generated with:
# rails g model rollout name:string group:string user_id:integer percentage:integer failure_count:integer
class Rollout < ActiveRecord::Base
  def match?(user)
    enabled? && match_group?(user) && match_user?(user) && match_percentage?(user)
  end

  def enabled?
    failure_count.to_i < 1
  end

  def match_group?(user)
    case group
    when "admin" then user.admin?
    else true
    end
  end

  def match_user?(user)
    user_id ? user_id == user.id : true
  end

  def match_percentage?(user)
    percentage ? user.id % 100 < percentage : true
  end
end

We’ve removed the Rollout gem from our application now and generated a new model called Rollout using the generator you can see in the comments at the top of the file. We’re using an ActiveRecord database but if performance is an issue we can easily swap this out for a different backend. Most of the logic in this file is for determining whether a rollout should apply to a given user based on a group, a specific user or a percentage just like the Rollout gem does. We can also specify a failure rate based off a failure count so that if part of our application fails a certain number of times the rollout is disabled.

We’ve also added two new methods to the ApplicationController. The first of these, rollout?, behaves like method we wrote to wrap the $rollout.active? functionality earlier. It finds all of the rollouts that match a given name and, if so, sees if those rollouts match the current user. The second, degrade_feature, which will yield to the block and if an exception is raised will increment the failure count for the matching rollouts.

def rollout?(name)
  Rollout.where(name: name).any? do |rollout|
    rollout.match?(current_user)
  end
end
helper_method :rollout?

def degrade_feature(name)
  yield
rescue StandardError => e
  Rollout.where(name: name).each do |rollout|
    rollout.increment!(:failure_count)
  end
  raise e
end

Our view code is the same as it was when we were using the Rollout gem so that the “Send to Phone” link is only shown if the phone feature is currently rolled-out.

<% if rollout? :phone %>
  <p><%= link_to "Send to Phone", new_phone_request_path %></p>
<% end %>

There PhoneRequestsController hasn’t changed much either.

class PhoneRequestsController < ApplicationController
  before_filter :authenticate
  before_filter :rollout
  around_filter :degrade

  def new
  end

  def create
    # Send to do list to phone number here
    raise "foo"
    redirect_to tasks_url, notice: "Sent list to #{params[:phone_number]}"
  end
  
  private
  def rollout
      redirect_to root_url, alert: "Feature unavailable" unless rollout? :phone
    end
  end

  def degrade
    degrade_feature(:phone) { yield }
  end
end

We still have a rollout before filter and a degrade around filter. The rollout method is much as it was before, redirecting if a feature is unavailable, while degrade now calls the ApplicationController’s degrade_feature method and passes it through the block so that it wraps around the action.

Our application now works much as it did before. By default the “Send to Phone” link will be disabled so we won’t see it but we can enable it in the console by creating a new Rollout record.

1.9.2p290 :001 > Rollout.create! name: >phone", group: "admin"

Now, if we’re logged in as an admin we’ll see the link again. If we click it and fill in a telephone number on the form and submit it we’ll see the exception that we deliberately left in the create action to test the degrade functionality. If we reload the page again we’ll be redirected to the home page and told that the feature is unavailable as the failure threshold will have been passed.