Upload asynchronously to Amazon S3 using Tornado

TornadoWeb is a great non-blocking web server written in Python and Boto3 is the Amazon Web Services (AWS) SDK for Python, which allows developers to write in a very easy manner software that makes use of Amazon services like S3. Unfortunately boto3 S3 wrapper is blocking and if you would just use it out of the box in a Tornado application it will block the main thread because it uses a synchronous HTTP client.

The solution is to use Tornado’s AsyncHTTPClient and do manually all the work which boto does for you under the hood.

I built a small replace of boto3 mthods for upload and delete (the only ones I need for the moment) which uses tornado’s AsyncHTTPClient and I published the code on github.

The main idea around this replacement is to use botocore to build the request (AWS wants the requests to be signed using different algorithms based on AWS zones and request data) and only to use the AsyncHTTPClient for the actual asynchronous call.

In order to use the S3AsyncManager you need to define an AWS profile, for example as

and the credentials file:

You can obtain your access_key and secret_access_key from your aws account.

To install the S3AsyncManager system wide (or in your virtualenv), clone this repo

and install it with pip:

Afterwords, you can use it as shown in the example folder.

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *