Upload direct to S3 with Pre-signed POST request

It's easy to create a form in Rails which can upload a file to the backend. The backend, can then take the file and upload it to S3. We can do that by using gems like paperclip or carrierwave. Or if we are using Rails 5.2, we can use Active Storage

But for applications, where Rails is used only as an API backend, uploading via a form is not an option. In this case, we can expose an endpoint which accepts files, and then Rails can handle uploading to S3.

In most of the cases, the above solution works. But recently, in one of our applications which is hosted at Heroku we faced time-out related problems while uploading large files. Here is what heroku's docs says about how long a request can take.

The router terminates the request if it takes longer than 30 seconds to complete.

Pre-signed POST request

An obvious solution is to upload the files directly to S3. However in order to do that, the client needs AWS credentials, which is not ideal. If the client is a Single Page Application, the AWS credentials would be visible in the javascript files. Or if the client is a mobile app, someone might be able to reverse engineer the application, and get hold of the AWS credentials.

Here's where Pre-signed POST request comes to the rescue. Here is official docs from AWS on this topic.

Uploading via Pre-signed POST is a two step process. The client first requests a permission to upload the file. The backend receives the request, generates the pre-signed URL and returns the response along with other fields. The client can then upload the file to the URL received in the response.

Implementation

Add the AWS gem to you Gemfile and run bundle install.

1gem 'aws-sdk'
Copy

Create a S3 bucket with the AWS credentials.

1
2aws_credentials = Aws::Credentials.new(
3  ENV['AWS_ACCESS_KEY_ID'],
4  ENV['AWS_SECRET_ACCESS_KEY']
5)
6
7s3_bucket = Aws::S3::Resource.new(
8  region: 'us-east-1',
9  credentials: aws_credentials
10).bucket(ENV['S3_BUCKET'])
11
Copy

The controller handling the request for getting the presigned URL should have following code.

1def request_for_presigned_url
2  presigned_url = s3_bucket.presigned_post(
3    key: "#{Rails.env}/#{SecureRandom.uuid}/${filename}",
4    success_action_status: '201',
5    signature_expiration: (Time.now.utc + 15.minutes)
6  )
7
8  data = { url: presigned_url.url, url_fields: presigned_url.fields }
9
10  render json: data, status: :ok
11end
Copy

In the above code, we are creating a presigned url using the presigned_post method.

The key option specifies path where the file would be stored. AWS supports a custom ${filename} directive for the key option. This ${filename} directive tells S3 that if a user uploads a file named image.jpg, then S3 should store the file with the same name. In S3, we cannot have duplicate keys, so we are using SecureRandom to generate unique key so that 2 files with same name can be stored.

If a file is successfully uploaded, then client receives HTTP status code under key success_action_status. If the client sets its value to 200 or 204 in the request, Amazon S3 returns an empty document along with 200 or 204 as HTTP status code. We set it to 201 here because we want the client to notify us with the S3 key where the file was uploaded to. The S3 key is present in the XML document which is received as a response from AWS only when the status code is 201.

signature_expiration specifies when the signature on the POST will expire. It defaults to one hour from the creation of the presigned POST. This value should not exceed one week from the creation time. Here, we are setting it to 15 minutes.

Other configuration options can be found here.

In response to the above request, we send out a JSON which contains the URL and the fields required for making the upload.

Here's a sample response.

1{
2  "url": "https://s3.amazonaws.com/<some-s3-url>",
3  "url_fields": {
4    "key": "development/8614bd40-691b-4668-9241-3b342c6cf429/${filename}",
5    "success_action_status": "201",
6    "policy": "<s3-policy>",
7    "x-amz-credential": "********************/20180721/us-east-1/s3/aws4_request",
8    "x-amz-algorithm": "AWS4-HMAC-SHA256",
9    "x-amz-date": "201807021T144741Z",
10    "x-amz-signature": "<hexadecimal-signature>"
11  }
12}
Copy

Once the client gets the above credentials, it can proceed with the actual file upload.

The client can be anything. An iOS app, android app, an SPA or even a Rails app. For our example, let's assume it's a node client.

1var request = require("request");
2function uploadFileToS3(response) {
3  var options = {
4    method: 'POST',
5    url: response.url,
6    formData: {
7      ...response.url_fields,
8      file: <file-object-for-upload>
9    }
10  }
11
12  request(options, (error, response, body) => {
13    if (error) throw new Error(error);
14    console.log(body);
15  });
16}
Copy

Here, we are making a POST request to the URL received from the earlier presigned response. Note that we are using the spread operator to pass url_fields in formData.

When the POST request is successful, the client then receives an XMLresponse from S3 because we set the response code to be 201. A sample response can be like the following.

1<?xml version="1.0" encoding="UTF-8"?>
2<PostResponse>
3    <Location>https://s3.amazonaws.com/link-to-the-file</Location>
4    <Bucket>s3-bucket</Bucket>
5    <Key>development/8614bd40-691b-4668-9241-3b342c6cf429/image.jpg</Key>
6    <ETag>"32-bit-tag"</ETag>
7</PostResponse>
Copy

Using the above response, the client can then let the API know about where the file was uploaded by sending the value from the Key node. Although, this can be optional in some cases, depending on the API, if it actually needs this info.

Advantages

Using AWS S3 presigned-urls has a few advantages.

• The main advantage of uploading directly to S3 is that there would be considerably less load on your application server since the server is now free from handling the receiving of files and transferring to S3.

• Since the file upload happens directly on S3, we can bypass the 30 seconds Heroku time limit.

• AWS credentials are not shared with the client application. So no one would be able to get their hands on your AWS keys.

• The generated presigned-url can be initialized with an expiration time. So the URLs and the signatures generated would be invalid after that time period.

• The client does not need to install any of the AWS libraries. It just needs to upload the file via a simple POST request to the generated URL.