How to mitigate DDoS using Rack::Attack

Recently, we faced a DDoS attack in one of the clients' projects. There were many requests from different IPs to root and login paths, and we were running thrice the usual number of servers to keep the system alive.

We are using Cloudflare's HTTP proxy and it was doing a great job preventing malicious requests, but we wanted to check if we can avoid the loading/captcha pages which Cloudflare uses to filter requests. We came to a conclusion that we would be able to mitigate the ongoing attack if we could throttle requests by IP.

Cloudflare has an inbuilt Rate Limiting feature to throttle requests, but it would be a little expensive in our case since Cloudflare charges by the number of good requests and it was a high traffic website. On further analysis, we found that throttling at application level would be enough in that situation and the gem Rack::Attack helped us with that.

Rack::Attack is a Rack middleware from Kickstarter. It can be configured to throttle requests based on IP or any other parameter.

To use Rack::Attack, include the gem in Gemfile.

gem "rack-attack"

After bundle install, configure the middleware in config/application.rb:

config.middleware.use Rack::Attack

Now we can create the initializer config/initializers/rack_attack.rb to configure Rack::Attack.

By default, Rack::Attack uses Rails.cache to store requests information. In our case, we wanted a separate cache for Rack::Attack and it was configured as follows.

redis_client = Redis.connect(url: ENV["REDIS_URL"])
Rack::Attack.cache.store = Rack::Attack::StoreProxy::RedisStoreProxy.new(redis_client)

If the web server is behind a proxy like Cloudflare, we have to configure a method to fetch the correct remote_ip address. Otherwise, it would block based on proxy's IP address and would result in blocking a lot of legit requests.

class Rack::Attack
  class Request < ::Rack::Request
    def remote_ip
      # Cloudflare stores remote IP in CF_CONNECTING_IP header
      @remote_ip ||= (env['HTTP_CF_CONNECTING_IP'] ||
                      env['action_dispatch.remote_ip'] ||
                      ip).to_s
    end
  end
end

Requests can be throttled based on IP address or any other parameter. In the following example, we are setting a limit of 40rpm/IP for "/" path.

class Rack::Attack
  throttle("req/ip", :limit => 40, :period => 1.minute) do |req|
    req.remote_ip if req.path == "/"
  end
end

The downside of this configuration is that after the 1 minute period, the attacker can launch another 40 requests/IP simultaneously and it would exert pressure on the servers. This can be solved using exponential backoff.

class Rack::Attack
  # Exponential backoff for all requests to "/" path
  #
  # Allows 240 requests/IP in ~8 minutes
  #        480 requests/IP in ~1 hour
  #        960 requests/IP in ~8 hours (~2,880 requests/day)
  (3..5).each do |level|
    throttle("req/ip/#{level}",
               :limit => (30 * (2 ** level)),
               :period => (0.9 * (8 ** level)).to_i.seconds) do |req|
      req.remote_ip if req.path == "/"
    end
  end
end

If we want to turn off throttling for some IPs (Eg.: Health check services), then those IPs can be safelisted.

class Rack::Attack
  class Request < ::Rack::Request
    def allowed_ip?
      allowed_ips = ["127.0.0.1", "::1"]
      allowed_ips.include?(remote_ip)
    end
  end

  safelist('allow from localhost') do |req|
    req.allowed_ip?
  end
end

We can log blocked requests separately and this is helpful for analyzing the attack.

ActiveSupport::Notifications.subscribe('rack.attack') do |name, start, finish, request_id, req|
  if req.env["rack.attack.match_type"] == :throttle
    request_headers = { "CF-RAY" => req.env["HTTP_CF_RAY"],
                        "X-Amzn-Trace-Id" => req.env["HTTP_X_AMZN_TRACE_ID"] }

    Rails.logger.info "[Rack::Attack][Blocked]" <<
                      "remote_ip: \"#{req.remote_ip}\"," <<
                      "path: \"#{req.path}\", " <<
                      "headers: #{request_headers.inspect}"
  end
end

A sample initializer with these configurations can be downloaded from here.

Application will now throttle requests and will respond with HTTP 429 Too Many Requests response for the throttled requests.

We now block a lot of malicious requests using Rack::Attack. Here's a graph with % of blocked requests over a week.

EDIT: Updated the post to add more context to the situation.