--- title: "Upload direct to S3 with Pre-signed POST request" description: "Pre-signed POST request allows for securely uploading large files directly to S3 via a signed expirable url, bypassing the 30 seconds Heroku time limit" canonical_url: "https://www.bigbinary.com/blog/uploading-files-directly-to-s3-using-pre-signed-post-request" markdown_url: "https://www.bigbinary.com/blog/uploading-files-directly-to-s3-using-pre-signed-post-request.md" --- # Upload direct to S3 with Pre-signed POST request Pre-signed POST request allows for securely uploading large files directly to S3 via a signed expirable url, bypassing the 30 seconds Heroku time limit - Author: Chirag Shah - Published: September 4, 2018 - Categories: Rails It's easy to create a form in Rails which can upload a file to the backend. The backend, can then take the file and upload it to S3. We can do that by using gems like [paperclip](https://github.com/thoughtbot/paperclip) or [carrierwave](https://github.com/carrierwaveuploader/carrierwave). Or if we are using Rails 5.2, we can use [Active Storage](https://github.com/rails/rails/tree/master/activestorage) But for applications, where Rails is used only as an API backend, uploading via a form is not an option. In this case, we can expose an endpoint which accepts files, and then Rails can handle uploading to S3. In most of the cases, the above solution works. But recently, in one of our applications which is hosted at [Heroku](https://www.heroku.com/) we faced time-out related problems while uploading large files. Here is what heroku's [docs](https://devcenter.heroku.com/articles/request-timeout) says about how long a request can take. > The router terminates the request if it takes longer than 30 seconds to > complete. ## Pre-signed POST request An obvious solution is to upload the files directly to S3. However in order to do that, the client needs AWS credentials, which is not ideal. If the client is a Single Page Application, the AWS credentials would be visible in the javascript files. Or if the client is a mobile app, someone might be able to reverse engineer the application, and get hold of the AWS credentials. Here's where Pre-signed POST request comes to the rescue. Here is [official docs](https://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObject.html) from AWS on this topic. Uploading via Pre-signed POST is a two step process. The client first requests a permission to upload the file. The backend receives the request, generates the pre-signed URL and returns the response along with other fields. The client can then upload the file to the URL received in the response. ## Implementation Add the AWS gem to you Gemfile and `run bundle install`. ```ruby gem 'aws-sdk' ``` Create a S3 bucket with the AWS credentials. ```ruby aws_credentials = Aws::Credentials.new( ENV['AWS_ACCESS_KEY_ID'], ENV['AWS_SECRET_ACCESS_KEY'] ) s3_bucket = Aws::S3::Resource.new( region: 'us-east-1', credentials: aws_credentials ).bucket(ENV['S3_BUCKET']) ``` The controller handling the request for getting the presigned URL should have following code. ```ruby def request_for_presigned_url presigned_url = s3_bucket.presigned_post( key: "#{Rails.env}/#{SecureRandom.uuid}/${filename}", success_action_status: '201', signature_expiration: (Time.now.utc + 15.minutes) ) data = { url: presigned_url.url, url_fields: presigned_url.fields } render json: data, status: :ok end ``` In the above code, we are creating a presigned url using the `presigned_post` method. The **key** option specifies path where the file would be stored. AWS supports a custom ${filename} directive for the key option. This ${filename} directive tells S3 that if a user uploads a file named `image.jpg`, then S3 should store the file with the same name. In S3, we cannot have duplicate keys, so we are using `SecureRandom` to generate unique key so that 2 files with same name can be stored. If a file is successfully uploaded, then client receives HTTP status code under key `success_action_status`. If the client sets its value to `200` or `204` in the request, Amazon S3 returns an empty document along with `200` or `204` as HTTP status code. We set it to `201` here because we want the client to notify us with the [S3 key](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#BasicsKeys) where the file was uploaded to. The S3 key is present in the XML document which is received as a response from AWS only when the status code is `201`. signature_expiration specifies when the signature on the POST will expire. It defaults to one hour from the creation of the presigned POST. This value should not exceed one week from the creation time. Here, we are setting it to 15 minutes. Other configuration options can be found [here](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Bucket.html#presigned_post-instance_method). In response to the above request, we send out a JSON which contains the URL and the fields required for making the upload. Here's a sample response. ```json { "url": "https://s3.amazonaws.com/", "url_fields": { "key": "development/8614bd40-691b-4668-9241-3b342c6cf429/${filename}", "success_action_status": "201", "policy": "", "x-amz-credential": "********************/20180721/us-east-1/s3/aws4_request", "x-amz-algorithm": "AWS4-HMAC-SHA256", "x-amz-date": "201807021T144741Z", "x-amz-signature": "" } } ``` Once the client gets the above credentials, it can proceed with the actual file upload. The client can be anything. An iOS app, android app, an SPA or even a Rails app. For our example, let's assume it's a node client. ```javascript var request = require("request"); function uploadFileToS3(response) { var options = { method: 'POST', url: response.url, formData: { ...response.url_fields, file: } } request(options, (error, response, body) => { if (error) throw new Error(error); console.log(body); }); } ``` Here, we are making a POST request to the URL received from the earlier presigned response. Note that we are using the [spread operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax) to pass `url_fields` in formData. When the POST request is successful, the client then receives an XMLresponse from S3 because we set the response code to be 201. A sample response can be like the following. ```xml https://s3.amazonaws.com/link-to-the-file s3-bucket development/8614bd40-691b-4668-9241-3b342c6cf429/image.jpg "32-bit-tag" ``` Using the above response, the client can then let the API know about where the file was uploaded by sending the value from the `Key` node. Although, this can be optional in some cases, depending on the API, if it actually needs this info. ## Advantages Using AWS S3 presigned-urls has a few advantages. - The main advantage of uploading directly to S3 is that there would be considerably less load on your application server since the server is now free from handling the receiving of files and transferring to S3. - Since the file upload happens directly on S3, we can bypass the 30 seconds Heroku time limit. - AWS credentials are not shared with the client application. So no one would be able to get their hands on your AWS keys. - The generated presigned-url can be initialized with an expiration time. So the URLs and the signatures generated would be invalid after that time period. - The client does not need to install any of the AWS libraries. It just needs to upload the file via a simple POST request to the generated URL. ## Links - [Human page](https://www.bigbinary.com/blog/uploading-files-directly-to-s3-using-pre-signed-post-request)