---
title: "How we migrated from Sidekiq to Solid Queue"
description: "How we migrated to Solid Queue from Sidekiq"
canonical_url: "https://www.bigbinary.com/blog/migrating-to-solid-queue-from-sidekiq"
markdown_url: "https://www.bigbinary.com/blog/migrating-to-solid-queue-from-sidekiq.md"
---

# How we migrated from Sidekiq to Solid Queue

How we migrated to Solid Queue from Sidekiq

- Author: Chirag Shah
- Published: March 5, 2024
- Categories: Rails, Solid Queue

BigBinary is building a suite of products under [neeto](https://neeto.com). We
currently have around 22 products under development, and all of the products are
using [Sidekiq](https://github.com/sidekiq/sidekiq). After the
[launch of Solid Queue](https://dev.37signals.com/introducing-solid-queue/), we
decided to migrate [NeetoForm](https://neeto.com/neetoform) from Sidekiq to
Solid Queue.

Please note that Solid Queue currently doesn't support cron-style or recurring
jobs. There is a [PR open](https://github.com/basecamp/solid_queue/pull/155)
regarding this issue. We have only partially migrated to Solid Queue. For
recurring jobs, we are still using Sidekiq. Once the PR is merged, we will
migrate completely to Solid Queue.

## Migrating to Solid Queue from Sidekiq

Here is a step-by-step migration guide you can use to migrate your Rails
application from Sidekiq to Solid Queue.

## 1. Installation

- Add `gem "solid_queue"` to your Rails application's Gemfile and run
  `bundle install`.
- Run `bin/rails generate solid_queue:install` which copies the config file and
  the required migrations.
- Run the migrations using `bin/rails db:migrate`.

## 2. Configuration

The installation step should have created a `config/solid_queue.yml` file.
Uncomment the file and modify it as per your needs. Here is how the file looks
for our application.

```yaml
default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: "auth"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "urgent"
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: "low"
      threads: 3
      processes: 1
      polling_interval: 2
    - queues: "*"
      threads: 3
      processes: 1
      polling_interval: 1

development:
  <<: *default

staging:
  <<: *default

heroku:
  <<: *default

test:
  <<: *default

production:
  <<: *default
```

## 3. Starting Solid Queue

On your development machine, you can start Solid Queue by running the following
command.

```
bundle exec rake solid_queue:start
```

This will start Solid Queue's supervisor process and will start processing any
enqueued jobs. The supervisor process forks
[workers and dispatchers](https://github.com/basecamp/solid_queue?tab=readme-ov-file#workers-and-dispatchers)
according to the configuration provided in the `config/solid_queue.yml` file.
The supervisor process also controls the heartbeats of workers and dispatchers,
and sends signals to stop and start them when needed.

Since we use [foreman](https://github.com/ddollar/foreman), we added the above
command to our Procfile.

```ruby
# Procfile
web:  bundle exec puma -C config/puma.rb
worker: bundle exec sidekiq -C config/sidekiq.yml
solidqueueworker: bundle exec rake solid_queue:start
release: bundle exec rake db:migrate
```

## 4. Setting the Active Job queue adapter

You can set the Active Job queue adapter to `:solid_queue` by adding the
following line in your `application.rb` file.

```ruby
# application.rb
config.active_job.queue_adapter = :solid_queue
```

The above change sets the queue adapter at the application level for all the
jobs. However, since we wanted to use Solid Queue for our regular jobs and
continue using Sidekiq for cron jobs, we didn't make the above change in
`application.rb`.

Instead, we created a new base class that inherited from `ApplicationJob` and
set the queue adapter to `:solid_queue` inside that.

```ruby
# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue
end
```

Then we made all the classes implementing regular jobs inherit from this new
class `SqBaseJob` instead of `ApplicationJob`.

```diff
# send_email_job.rb
- class SendEmailJob < ApplicationJob
+ class SendEmailJob < SqBaseJob
  # ...
end
```

By making the above change, all our regular jobs got enqueued via Solid Queue
instead of Sidekiq.

But we realized later that emails were still being sent via Sidekiq. On
debugging and looking into Rails internals, we found that `ActionMailer` uses
`ActionMailer::MailDeliveryJob` for enqueuing or sending emails.

`ActionMailer::MailDeliveryJob` inherits from `ActiveJob::Base` rather than the
application's `ApplicationJob`. So even if we set the queue_adapter in
`application_job.rb`, it won't work. `ActionMailer::MailDeliveryJob` fallbacks
to using the adapter defined in `application.rb` or environment-specific
(production.rb / staging.rb / development.rb) config files. But we can't do that
because we still want to use Sidekiq for cron jobs.

To use Solid Queue for mailers, we needed to override the queue_adapter for
mailers. We can do that in `application_mailer.rb`.

```ruby
# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.queue_adapter = :solid_queue
end
```

This change is only until we use both Sidekiq and Solid Queue. Once cron style
jobs feature lands in Solid Queue, we can remove this override and set the
queue_adapter directly in `application.rb`, which will enforce the setting
globally.

## 5. Code changes

For migrating from Sidekiq to Solid Queue, we had to make the following changes
to the syntax for enqueuing a job.

- Replaced `.perform_async` with `.perform_later`.
- Replaced `.perform_at` with `.set(...).perform_later(...)`.

```diff
- SendMailJob.perform_async
+ SendMailJob.perform_later

- SendMailJob.perform_at(1.minute.from_now)
+ SendMailJob.set(wait: 1.minute).perform_later
```

At some places we were storing the Job ID on a record, for querying the job's
status or for cancelling the job. For such cases, we made the following change.

```diff
def disable_form_at_deadline
- job_id = DisableFormJob.perform_at(deadline, self.id)
- self.disable_job_id = job_id
+ job_id = DisableFormJob.set(wait_until: deadline).perform_later(self.id)
+ self.disable_job_id = job.job_id
end

def cancel_form_deadline
- Sidekiq::Status.cancel(self.disable_job_id)
+ SolidQueue::Job.find_by(active_job_id: self.disable_job_id).destroy!
  self.disable_job_id = nil
end
```

## 6. Error handling and retries

Initially, we thought the
[`on_thread_error` configuration](https://github.com/basecamp/solid_queue?tab=readme-ov-file#other-configuration-settings)
provided by Solid Queue can be used for error handling. However, during the
development phase, we noticed that it wasn't capturing errors. We raised
[an issue with Solid Queue](https://github.com/basecamp/solid_queue/issues/120)
as we thought it was a bug.

[Rosa Gutiérrez](https://github.com/rosa)
[responded](https://github.com/basecamp/solid_queue/issues/120#issuecomment-1894413948)
on the issue and clarified the following.

> `on_thread_error` wasn't intended for errors on the job itself, but rather
> errors in the thread that's executing the job, but around the job itself. For
> example, if you had an Active Record's thread pool too small for your number
> of threads and you got an error when trying to check out a new connection,
> on_thread_error would be called with that.
>
> For errors in the job itself, you could try to hook into Active Job's itself.

Based on the above information, we modified our `SqBaseJob` base class to handle
the exceptions and report it to [Honeybadger](https://www.honeybadger.io/).

```ruby
# sq_base_job.rb
class SqBaseJob < ApplicationJob
  self.queue_adapter = :solid_queue

  rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end
```

Remember we mentioned that `ActionMailer` doesn't inherit from `ApplicationJob`.
So similarly, we would have to handle exceptions for Mailers separately.

```ruby
# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.rescue_from(Exception) do |exception|
    context = {
      error_class: self.class.name,
      args: self.arguments,
      scheduled_at: self.scheduled_at,
      job_id: self.job_id
    }
    Honeybadger.notify(exception, context:)
    raise exception
  end
end
```

For retries, unlike Sidekiq, Solid Queue doesn't include any automatic retry
mechanism, it
[relies on Active Job for this](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs).
We wanted our application to retry sending emails in case of any errors. So we
added the retry logic in the `ApplicationMailer`.

```ruby
# application_mailer.rb
class ApplicationMailer < ActionMailer::Base
  # ...
  ActionMailer::MailDeliveryJob.retry_on StandardError, attempts: 3
end
```

Note that, although the queue adapter configuration can be removed from
`application_mailer.rb` once the entire application migrates to Solid Queue,
error handling and retry override cannot be removed because of the way
`ActionMailer::MailDeliveryJob` inherits from `ActiveJob::Base` rather than
application's `ApplicationJob`.

## 7. Testing

Once all the above changes were done, it was obvious that a lot of tests were
failing. Apart from fixing the usual failures related to the syntax changes,
some of the tests were failing inconsistently. On debugging, we found that the
affected tests were all related to controllers, specifically tests inheriting
from `ActionDispatch::IntegrationTest`.

We tried debugging and searched for solutions when we stumbled upon
[Ben Sheldon's](https://github.com/bensheldon)
[comment on one of Good Job's issues](https://github.com/bensheldon/good_job/issues/846#issuecomment-1432375562).
Ben points out that this is actually
[an issue in Rails](https://github.com/rails/rails/issues/37270) where Rails
sometimes inconsistently overrides ActiveJob's queue_adapter setting with
TestAdapter. A [PR is already open](https://github.com/rails/rails/pull/48585)
for the fix. Thankfully, Ben, in the same comment, also mentioned a workaround
for it until the fix has been added to Rails.

We added the workaround in our test `helper_methods.rb` and called the method in
each of our controller tests which were failing.

```ruby
# test/support/helper_methods.rb
def ensure_consistent_test_adapter_is_used
  # This is a hack mentioned here: https://github.com/bensheldon/good_job/issues/846#issuecomment-1432375562
  # The actual issue is in Rails for which a PR is pending merge
  # https://github.com/rails/rails/pull/48585
  (ActiveJob::Base.descendants + [ActiveJob::Base]).each(&:disable_test_adapter)
end
```

```ruby
# test/controllers/exports_controller_test.rb
class ExportsControllerTest < ActionDispatch::IntegrationTest
  def setup
    ensure_consistent_test_adapter_is_used
    # ...
  end

  # ...
end
```

## 8. Monitoring

Basecamp has released
[mission_control-jobs](https://github.com/basecamp/mission_control-jobs), which
can be used to monitor background jobs. It is generic, so it can be used with
any compatible ActiveJob adapter.

Add `gem "mission_control-jobs"` to your Gemfile and run `bundle install`.

Mount the mission control route in your `routes.rb` file.

```ruby
# routes.rb
Rails.application.routes.draw do
  # ...
  mount MissionControl::Jobs::Engine, at: "/jobs"
```

By default, mission control would try to load the adapter specified in your
`application.rb` or individual environment-specific files. Currently, Sidekiq
isn't compatible with mission control, so you will face an error while loading
the dashboard at `/jobs`. The fix is to explicitly specify `solid_queue` to the
list of mission control adapters.

```ruby
# application.rb
# ...
config.mission_control.jobs.adapters = [:solid_queue]
```

Now, visiting `/jobs` on your site should load a dashboard where you can monitor
your Solid Queue jobs.

But that isn't enough. There is no authentication. For development environments,
it is fine, but the `/jobs` route would be exposed on production too. By
default, Mission Control's controllers will extend the host app's
`ApplicationController`. If no authentication is enforced, `/jobs` will be
available to everyone.

To implement some kind of authentication, we can specify a different controller
as the base class for Mission Control's controllers and add the authentication
there.

```ruby
# application.rb
# ...
MissionControl::Jobs.base_controller_class = "MissionControlController"
```

```ruby
# app/controllers/mission_control_controller.rb
class MissionControlController < ApplicationController
  before_action :authenticate!, if: :restricted_env?

  private

    def authenticate!
      authenticate_or_request_with_http_basic do |username, password|
        username == "solidqueue" && password == Rails.application.secrets.mission_control_password
      end
    end

    def restricted_env?
      Rails.env.staging? || Rails.env.production?
    end
end
```

Here, we have specified that `MissionControlController` would be our base
controller for mission control related controllers. Then in
`MissionControlController` we implemented basic authentication for staging and
production environments.

## Observations

We haven't had any complaints so far. Solid Queue offers simplicity, requires no
additional infrastructure and provides visibility for managing jobs since they
are stored in the database.

In the coming days, we will migrate all of our 22 Neeto products to Solid Queue.
And once cron-style job support lands in Solid Queue, we will completely migrate
from Sidekiq.

## Links

- [Human page](https://www.bigbinary.com/blog/migrating-to-solid-queue-from-sidekiq)