Mime type resolution in Rails

This is a long blog. If you want a summary then José Valim has provided a summary in less than 140 characters.

It is common to see following code in Rails

respond_to do |format|
  format.html
  format.xml  { render :xml => @users }
end

If you want output in xml format then request with .xml extension at the end like this localhost:3000/users.xml and you will get the output in xml format.

What we saw is only one part of the puzzle. The other side of the equation is HTTP header field Accept defined in HTTP RFC.

HTTP Header Field Accept

When browser sends a request then it also sends the information about what kind of resources the browser is capable of handling. Here are some of the examples of the Accept header a browser can send.

text/plain

image/gif, images/x-xbitmap, images/jpeg, application/vnd.ms-excel, application/msword,
application/vnd.ms-powerpoint, */*

text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

application/vnd.wap.wmlscriptc, text/vnd.wap.wml, application/vnd.wap.xhtml+xml,
application/xhtml+xml, text/html, multipart/mixed, */*

If you are reading this blog on a browser then you can find out what kind of Accept header your browser is sending by visiting this link. Here is list of Accept header sent by different browsers on my machine.

Chrome: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Firefox: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8,application/json
Safari: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
IE: application/x-ms-application, image/jpeg, application/xaml+xml, image/gif,
image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, */*

Let's take a look at the Accept header sent by Safari.

Safari: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5

Safari is saying that I can handle documents which are xml (application/xml), html (text/html) or plain text (text/plain) documents. And I can handle images such as image/png. If all else fails then send me whatever you can and I will try to render that document to the best of my ability.

Notice that there are also q values. That signifies the priority order. This is what HTTP spec has to say about q.

Each media-range MAY be followed by one or more accept-params, beginning with the "q" parameter for indicating a relative quality factor. The first "q" parameter (if any) separates the media-range parameter(s) from the accept-params. Quality factors allow the user or user agent to indicate the relative degree of preference for that media-range, using the qvalue scale from 0 to 1 (section 3.9). The default value is q=1.

The spec is saying is that each document type has a default value of q as 1. When q value is specified then take that value into account. For all documents that have same q value give high priority to the one that came first in the list. Based on that this should be the order in which documents should be sent to safari browser.

application/xml (q is 1)
application/xhtml+xml (q is 1)
image/png (q is 1)
text/html (q is 0.9)
text/plain (q is 0.8)
\*/\* (q is 0.5)

Notice that Safari is nice enough to put a lower priority for */*. Chrome and Firefox also puts */* at a lower priority which is a good thing. Not so with IE which does not declare any q value for */* .

Look at the order again and you can see that application/xml has higher priority over text/html. What it means is that safari is telling Rails that I would prefer application/xml over text/html. Send me text/html only if you cannot send application/xml.

And let's say that you have developed a RESTful app which is capable of sending output in both html and xml formats.

Rails being a good HTTP citizen should follow the HTTP_ACCEPT protocol and should send an xml document in this case. Again all you did was visit a website and safari is telling rails that send me xml document over html document. Clearly HTTP_ACCEPT values being sent by Safari is broken.

HTTP_ACCEPT is broken

HTTPACCEPT attribute concept is neat. It defines the order and the priority. However the implementation is broken by all the browser vendors. Given the case that browsers do not send proper HTTP_ACCEPT what can rails do. One solution is to ignore it completely. If you want _xml output then request http://localhost:3000/users.xml . Solely relying on formats make life easy and less buggy. This is what Rails did for a long time.

Starting this commit ,by default, rails did ignore HTTP_ACCEPT attribute. Same is true for Twitter API where HTTP_ACCEPT attribute is ignored and twitter solely relies on format to find out what kind of document should be returned.

Unfortunately this solution has its own sets of problems. Web has been there for a long time and there are a lot of applications who expect the response type to be RSS feed if they are sending application/rss+xml in their HTTP*ACCEPT attribute. It is not nice to take a hard stand and ask all of them to request with extension *.rss_ .

Parsing HTTP_ACCEPT attribute

Parsing and obeying HTTP_ACCEPT attribute is filled with many edge cases. First let's look at the code that decides what to parse and how to handle the data.

BROWSER_LIKE_ACCEPTS = /,\s*\*\/\*|\*\/\*\s*,/

def formats
  accept = @env['HTTP_ACCEPT']

  @env["action_dispatch.request.formats"] ||=
    if parameters[:format]
      Array(Mime[parameters[:format]])
    elsif xhr? || (accept && accept !~ BROWSER_LIKE_ACCEPTS)
      accepts
    else
      [Mime::HTML]
    end
end

Notice that if a format is passed like http://localhost:3000/users.xml or http://localhost:3000/users.js then Rails does not even parse the HTTP_ACCEPT values. Also note that if browser is sending */* along with other values then Rails totally bails out and just returns Mime::HTML unless the request is ajax request.

Next I am going to discuss some of the cases in greater detail which should bring more clarity around this issue.

Case 1: HTTP_ACCEPT is */*

I have following code.

respond_to do |format|
  format.html { render :text => 'this is html' }
  format.js  { render :text => 'this is js' }
end

I am assuming that HTTP_ACCEPT value is */* . In this case browser is saying that send me whatever you got. Since browser is not dictating the order in which documents should be sent Rails will look at the order in which Mime types are declared in respond_to block and will pick the first one. Here is the corresponding code

def negotiate_mime(order)
  formats.each do |priority|
    if priority == Mime::ALL
      return order.first
    elsif order.include?(priority)
      return priority
    end
  end

  order.include?(Mime::ALL) ? formats.first : nil
end

What it's saying is that if Mime::ALL is sent then pick the first one declared in the respond_to block. So be careful with order in which formats are declared inside the respond_to block.

The order in which formats are declared can be real issue. Checkout case (Link is not available) where the author ran into issue because of the order in which formats are declared.

So far so good. However what if there is no respondto block. If I don't have respond_to block and if I have _index.html.erb, index.js.erb and index.xml.builder files in my view directory then which one will be picked up. In this case Rails will go over all the registered formats in the order in which they are declared and will try to find a match . So in this case it matters in what order Mime types are registered. Here is the code that registers Mime types.

Mime::Type.register "text/html", :html, %w( application/xhtml+xml ), %w( xhtml )
Mime::Type.register "text/plain", :text, [], %w(txt)
Mime::Type.register "text/javascript", :js, %w( application/javascript application/x-javascript )
Mime::Type.register "text/css", :css
Mime::Type.register "text/calendar", :ics
Mime::Type.register "text/csv", :csv
Mime::Type.register "application/xml", :xml, %w( text/xml application/x-xml )
Mime::Type.register "application/rss+xml", :rss
Mime::Type.register "application/atom+xml", :atom
Mime::Type.register "application/x-yaml", :yaml, %w( text/yaml )

Mime::Type.register "multipart/form-data", :multipart_form
Mime::Type.register "application/x-www-form-urlencoded", :url_encoded_form

# http://www.ietf.org/rfc/rfc4627.txt
# http://www.json.org/JSONRequest.html
Mime::Type.register "application/json", :json, %w( text/x-json application/jsonrequest )

# Create Mime::ALL but do not add it to the SET.
Mime::ALL = Mime::Type.new("*/*", :all, [])

As you can see text/html is first in the list, text/javascript next and then application/xml. So Rails will look for view file in the following order: index.html.erb , index.js.erb and index.xml.builder .

Case 2: HTTP_ACCEPT with no */*

I am going to assume that in this case HTTP_ACCEPT sent by browser looks really simple like this

text/javascript, text/html, text/plain

I am also assuming that my respond_to block looks like this

respond_to do |format|
  format.html { render :text => 'this is html' }
  format.js  { render :text => 'this is js' }
end

So browser is saying that I prefer documents in following order

 js
 html
 plain

The order in which formats are declared is

html (format.html)
js (format.js)

In this case rails will go through each Mime type that browser supports from top to bottom one by one. If a match is found then response is sent otherwise rails tries find match for next Mime type. First in the list of Mime types supported by browser is js and Rails does find that my respondto block supports .js . Rails executes _format.js block and response is sent to browser.

Case 3: Ajax requests

When an AJAX request is made the Safari, Firefox and Chrome send only one item in HTTP_ACCEPT and that is */*. So if you are making an AJAX request then HTTP_ACCEPT for these three browsers will look like

Chrome: */*
Firefox: */*
Safari: */*

and if your respond_to block looks like this

respond_to do |format|
  format.html { render :text => 'this is html' }
  format.js  { render :text => 'this is js' }
end

then the first one will be served based on the formats order. And in this case html response would be sent for an AJAX request. This is not what you want.

This is the reason why if you are using jQuery and if you are sending AJAX request then you should add something like this in your application.js file

$(function () {
  $.ajaxSetup({
    beforeSend: function (xhr) {
      xhr.setRequestHeader("Accept", "text/javascript");
    },
  });
});

If you are using a newer version of rails.js then you don't need to add above code since it is already take care of for you through this commit .

Trying it out

If you want to play with HTTP_ACCEPT header then put the following line in your controller to inspect the HTTP_ACCEPT attribute.

puts request.headers['HTTP_ACCEPT']

I used following rake task to set custom HTTP_ACCEPT attribute.

require "net/http"
require "uri"

task :custom_accept do
  uri = URI.parse("http://localhost:3000/users")
  http = Net::HTTP.new(uri.host, uri.port)

  request = Net::HTTP::Get.new(uri.request_uri)
  request["Accept"] = "text/html, application/xml, */*"

  response = http.request(request)
  puts response.body
end

Thanks

I got familiar with intricacies of mime parsing while working on ticket #6022 . A big thanks to José Valim for patiently dealing with me while working on this ticket.