2 ways to upload files to Amazon S3 in Flask

Many times, you need the ability to allow users to upload files, mainly images, to your WebApp. One option is you can store those images in your database using binary objects (blobs), which is what I tried while developing WeTalk, which used a PostgreSQL database on Heroku. However, as I discovered later, storing user-uploaded images in your database as blobs is a bad idea. Processing the binary object from the database as an image took a lot of time. And, if the user uploaded a large image (5MB, 10MB), it would take so much time that the app would crash due to timeout errors. So, there has to be a better way to handle user-uploaded images.

You might have heard of Amazon S3, which is a popular and reliable storage option. So, what we will do is when the user uploads the image, store it in an S3 bucket, and store the image’s location in the database. Later, when you need to load that image, you load it from the URL stored. In a flask web app, there are two possible ways to upload a file to S3.

1. Using Flask to upload the file to S3 — Using this method, we pass the file to our Flask application and then transfer the file to the S3 bucket.
2. Uploading directly to S3 — Here, we use client-side JavaScript to upload the file directly to S3 without our Flask application receiving it. This saves bandwidth and reduces the load on your server considerably.

Before we start, you will need an AWS Account and Amazon S3 Access Key ID and a Secret Access Key, which acts as a username and password. To get them, log in to your AWS Management Console and on the top right, under your name, select “My Security Credentials” then open the “Access Keys” tab and finally click “Create New Access Key”. To set up AWS and S3 bucket, this documentation by AWS is helpful.

Using Flask to upload the file to S3

pip install boto3 Boto3 is a AWS SDK for Python. It provides a high-level interface to interact with AWS API. Now, we specify the required config variables for boto3

app.config['S3_BUCKET'] = "S3_BUCKET_NAME"
app.config['S3_KEY'] = "AWS_ACCESS_KEY"
app.config['S3_SECRET'] = "AWS_ACCESS_SECRET"
app.config['S3_LOCATION'] = 'http://{}.s3.amazonaws.com/'.format(S3_BUCKET)

Using boto3.client, we will connect to our AWS account.

import boto3, botocore

s3 = boto3.client(
   "s3",
   aws_access_key_id=app.config['S3_KEY'],
   aws_secret_access_key=app.config['S3_SECRET']
)

Here, we will take the file from the user’s computer to our server and call send_to_s3() function. This code is a standard code for uploading files in flask

@app.route("/", methods=["POST"])
def upload_file():
    if "user_file" not in request.files:
        return "No user_file key in request.files"

    file = request.files["user_file"]

    if file.filename == "":
        return "Please select a file"

    if file:
        file.filename = secure_filename(file.filename)
        output = send_to_s3(file, app.config["S3_BUCKET"])
        return str(output)

    else:
        return redirect("/")

This code simply takes the file from user’s computer and calls the function send_to_s3() on it.

def upload_file_to_s3(file, bucket_name, acl="public-read"):
    """
    Docs: http://boto3.readthedocs.io/en/latest/guide/s3.html
    """
    try:
        s3.upload_fileobj(
            file,
            bucket_name,
            file.filename,
            ExtraArgs={
                "ACL": acl,
                "ContentType": file.content_type    #Set appropriate content type as per the file
            }
        )
    except Exception as e:
        print("Something Happened: ", e)
        return e
    return "{}{}".format(app.config["S3_LOCATION"], file.filename)

In the last line, we are returning the location of the uploaded file, as the public url of a file hosted on a S3 bucket usually looks like https://bucketname.s3.amazonaws.com/filename.extension. We can then use this URL later to load the uploaded image.

That’s it. This way, you can upload your files to S3 through Flask. Here is the full code for it.

Now lets look at the second way, i.e. to upload directly to S3. 

Uploading File Directly to S3

Here, we upload the file directly without passing it through our webserver. The process happens in following steps:

1. User selects a file to upload
2. JavaScript makes a request to the webserver. The webserver produces a temporary signature with which to sign the upload request and returns it to the browser as JSON.
3. JavaScript (client-side) then uploads the file directly to Amazon S3 using the signed request supplied by our webserver. The upload takes place asynchronously, and user is displayed his/her uploaded image. On clicking submit, only the URL of the uploaded image gets sent to our Flask application.

So, on the client side, we need to-

1. Handle file selection
2. Obtain the signed request from the Flask app with which the image can be uploaded to S3
3. Finally, upload the image to S3 with a asynchronous POST request.

The HTML for the form -

<input type="file" id="file_input"/>
<p id="status">Please select a file</p>
<img id="preview" src="/static/default.png" />

<form method="POST" action="/submit_form/">
  <input type="hidden" id="avatar-url" name="avatar-url" value="/static/default.png">
  <input type="text" name="username" placeholder="Username">
  <input type="text" name="full-name" placeholder="Full name">
  <input type="submit" value="Update profile">
</form>

JS function for file input -

(function() {
  document.getElementById("file_input").onchange = function(){
    var files = document.getElementById("file_input").files;
    var file = files[0];
    if(!file){
      return alert("No file selected.");
    }
    getSignedRequest(file);
  };
})();

JS function to accept the file object and retrieve signed request from our Flask app -

function getSignedRequest(file){
  var xhr = new XMLHttpRequest();
  xhr.open("GET", "/sign_s3?file_name="+file.name+"&file_type="+file.type);
  xhr.onreadystatechange = function(){
    if(xhr.readyState === 4){
      if(xhr.status === 200){
        var response = JSON.parse(xhr.responseText);
        uploadFile(file, response.data, response.url);
      }
      else{
        alert("Could not get signed URL.");
      }
    }
  };
  xhr.send();
}

JS function to upload the actual file to S3 using the signed request -

function uploadFile(file, s3Data, url){
  var xhr = new XMLHttpRequest();
  xhr.open("POST", s3Data.url);

  var postData = new FormData();
  for(key in s3Data.fields){
    postData.append(key, s3Data.fields[key]);
  }
  postData.append('file', file);

  xhr.onreadystatechange = function() {
    if(xhr.readyState === 4){
      if(xhr.status === 200 || xhr.status === 204){
        document.getElementById("preview").src = url;
        document.getElementById("avatar-url").value = url;
      }
      else{
        alert("Could not upload file.");
      }
   }
  };
  xhr.send(postData);
}

Flask route to generate and respond with a signed request -

@app.route('/sign_s3/')
def sign_s3():
  S3_BUCKET = os.environ.get('S3_BUCKET')

  file_name = request.args.get('file_name')
  file_type = request.args.get('file_type')

  s3 = boto3.client('s3')

  presigned_post = s3.generate_presigned_post(
    Bucket = S3_BUCKET,
    Key = file_name,
    Fields = {"acl": "public-read", "Content-Type": file_type},
    Conditions = [
      {"acl": "public-read"},
      {"Content-Type": file_type}
    ],
    ExpiresIn = 3600
  )

  return json.dumps({
    'data': presigned_post,
    'url': 'https://%s.s3.amazonaws.com/%s' % (S3_BUCKET, file_name)
  })

As you can see, we use boto3’s generate_presigned_post to generate the request and then send it as JSON to the client.request

Conclusion

That’s it for this post. These are the two ways to upload files to Amazon S3 using Flask. S3 is a really cool service and there’s a free tier as well, so you should definitely use it in your projects. I hope this post was helpful, see you in the next one!

Further Reading

1. Heroku's documentation for S3 Uploads
2. S3 Boto3 Documentation