Upload asynchronously to Amazon S3 using Tornado

TornadoWeb is a great non-blocking web server written in Python and Boto3 is the Amazon Web Services (AWS) SDK for Python, which allows developers to write in a very easy manner software that makes use of Amazon services like S3. Unfortunately boto3 S3 wrapper is blocking and if you would just use it out of the box in a Tornado application it will block the main thread because it uses a synchronous HTTP client.

The solution is to use Tornado’s AsyncHTTPClient and do manually all the work which boto does for you under the hood.

I built a small replace of boto3 mthods for upload and delete (the only ones I need for the moment) which uses tornado’s AsyncHTTPClient and I published the code on github.

The main idea around this replacement is to use botocore to build the request (AWS wants the requests to be signed using different algorithms based on AWS zones and request data) and only to use the AsyncHTTPClient for the actual asynchronous call.

In order to use the S3AsyncManager you need to define an AWS profile, for example as

# cat /home/razvan/.aws/config
[profile upload]
region=eu-west-1

and the credentials file:

# cat /home/oem/.aws/credentials
[upload]
aws_access_key_id=<YOUR KEY>
aws_secret_access_key=<YOUR SECRET>

You can obtain your access_key and secret_access_key from your aws account.

To install the S3AsyncManager system wide (or in your virtualenv), clone this repo

git clone https://github.com/raztud/tornados3

and install it with pip:

pip install .

Afterwords, you can use it as shown in the example folder.

from tornado import gen
from tornado import ioloop

from tornados3 import S3AsyncManager

@gen.coroutine
def run():
    fh = open('./gnu.png')  # the file needs to exists on the disk
    data = fh.read()
    fh.close()

    client = S3AsyncManager(profile_name='upload', bucket='<YOUR_BUCKET>')
    yield client.upload(body=data, path='path/to/gnu.png')

    # and to delete
    yield client.delete('path/to/gnu.png')


if __name__ == '__main__':
    ioloop.IOLoop.instance().run_sync(run)

 

 

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.