Not using https might be breaking file uploads

Recently we came across following error when mobile app was trying to upload images using backend API.

==> nginx-error.log <==

2015/12/16 07:34:13 [error] 1440#0: *1021901
sendfile() failed (32: Broken pipe) while sending request to upstream,
client: xxx.xx.xxx.xx, server: app-name.local,
request: "POST /api_endpoint HTTP/1.1",
upstream: "http://unix:/data/apps/app-name/shared/tmp/sockets/unicorn.sock:/api_endpoint",
host: "appname.com"

==> nginx-error.log <==

xxx.xx.xxx.xx - - [16/Dec/2015:07:34:13 - 0500]
"POST /api_endpoint HTTP/1.1" 502 1051 "-"
"Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13B143 (366691552)"

The backend API was developed using Ruby on Rails and was powered using NGINX HTTP server along with unicorn application server.

Notice the http response received was 502 which is documented in RFC as shown below.

10.5.3 502 Bad Gateway The server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request.

Debugging the problem

From the first look it seemed that the client_max_body_size parameter for nginx was too less, which might cause the request to fail if the uploaded filesize is greater than the allowed limit.

nginx.conf
http {
  ...
  client_max_body_size 250M;
  ...
}

But that was not the case with our NGINX configuration. The NGINX was configured to accept file size up to 250 megabytes, while the file being uploaded was around 6 megabytes.

In our Rails code we are forcing all traffic to use HTTPS by having production.rb to have following setting.

config.force_ssl = true

What this setting does is, it forces HTTPS by redirecting HTTP requests to their HTTPS counterparts. So any request accessing http://domain.com/path will be redirected to https://domain.com/path

Forcing all traffic to HTTPS still does not explain the vague error sendfile() failed (32: Broken pipe) while sending request to upstream and why the HTTP 502 error which means Bad Gateway

Here is how our nginx.conf is configured.

server {
  ...
  listen 80;
  listen 443 ssl;
  ...
}

NGINX is configured to accept both HTTP and HTTPS requests. This is a pretty standard setting so that application can support both HTTP and HTTPS. What it means is that nginx will accept both HTTP and HTTPS request. It would not force HTTP request to be HTTPS. It would forward HTTP request to rails backend api.

Here is what happens when you upload a file

When we upload a file then NGINX takes that request and passes that request to Rails. NGINX expects a response from Rails only when NGINX is done sending the whole request.

Let's see what happens when a user visits login page using HTTP. NGINX takes the request and passes the request to Rails. When NGINX is done handing over the request to Rails then NGINX expects a response from Rails. However in this case rather than sending 200 Rails sends a redirect over HTTPS. So now NGINX takes up the request again over HTTPS and then hands over request to Rails. Rails processes the request and returns 200. Everything works out.

Now let's see what happens when a file is uploaded over HTTP.

User uploads a file over HTTP. NGINX takes up the request and starts sending the file to Rails. More details about the file upload process can be see at RFC (Link is not available). The data is sent to server in chunks. When first chunk is sent to server then server notices that data is being sent over HTTP and it immediately sends a response that data should be sent over HTTPS.

In the meantime on the client side , client is still pushing rest of the data to the server. Client expects a response from the server only when it is done pushing all the data. However in this case server sends a response when client is not expecting it.

NGINX at this time is all confused and thinks something has gone wrong with the gateway and returns HTTP/1.1 502 Bad Gateway and aborts the upload operation. And that how we get the 502 error.

So next time you see an error like this make sure that you are using https to upload the file if server is enforcing https.