Autoscale Resque Workers on Heroku

27 November 2011

A common technique for autoscaling Resque workers on Heroku uses the after_enqueue and after_perform hooks that run within each Resque worker to adjust the current worker scale up and down, respectively. The after_perform hook watches the pending job count. When no jobs remain, this hook will scale all workers down.

When scaling workers up, the Heroku API is additive; thus, additional workers can be scaled while others are actively processing jobs. As the Heroku API does not currently allow individually specified workers to scale down, however, it is necessary to wait until all workers are inactive before performing a scale down operation.

While the after_perform scale down approach mitigates against shutting an active worker down while pending jobs exist, a race condition can occur where the pending job count hits zero, and a scale down operation is invoked on the Heroku API prior to a subsequent job being enqueued from the app and reserved on a worker which will be shutdown in the current scale operation.

Recently, I have been working on the resque-heroku-scaler gem which autoscales Resque workers through a separate monitor process. The scaler monitor process polls for pending jobs against the specified Resque Redis backend. When scaling down, this scaler process locks each worker from attempting to reserve any future pending jobs before a scale down operation is performed.

To get started, include the scaler tasks in lib/tasks/scaler.rake

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment

In your Procfile, configure the scaler process using:

scaler: bundle exec rake resque:heroku_scaler

To run the scaler process, use the following command.

heroku scale scaler=1

Require the worker extensions within the app running your workers. For example, in lib/tasks/resque.rake. These extensions to the worker run loop poll for a shared flag within the Redis backend set by the scaler process which indicates the worker should enter an inactive scaling state. After a worker has entered this scaling state, it will cease attempting to check for incoming pending jobs. Once all workers have indicated they are inactive, the scaler process issues the scale down operation using the Heroku API.

require 'resque/tasks'

task "resque:setup" => :environment do
    require 'resque-heroku-scaler'
    ENV['QUEUE'] = '*'
end

In your development environment, the scaler process can run local worker processes using the rush library. To configure, use the following in your scaler rake file.

require 'resque/tasks'
require 'resque/plugins/heroku_scaler/tasks'

task "resque:setup" => :environment do
    if Rails.env.development?
        require 'resque-heroku-scaler'
        ENV["RUSH_PATH"] ||= File.expand_path('/path/to/app', __FILE__)
        Resque::Plugins::HerokuScaler.configure do |c|
            c.scale_manager = :local
        end
    end
end

By Aaron Dunnington

Related Articles