Tag Archives: python

Upload asynchronously to Amazon S3 using Tornado

TornadoWeb is a great non-blocking web server written in Python and Boto3 is the Amazon Web Services (AWS) SDK for Python, which allows developers to write in a very easy manner software that makes use of Amazon services like S3. Unfortunately boto3 S3 wrapper is blocking and if you would just use it out of the box in a Tornado application it will block the main thread because it uses a synchronous HTTP client.

The solution is to use Tornado’s AsyncHTTPClient and do manually all the work which boto does for you under the hood.

I built a small replace of boto3 mthods for upload and delete (the only ones I need for the moment) which uses tornado’s AsyncHTTPClient and I published the code on github.

The main idea around this replacement is to use botocore to build the request (AWS wants the requests to be signed using different algorithms based on AWS zones and request data) and only to use the AsyncHTTPClient for the actual asynchronous call.

In order to use the S3AsyncManager you need to define an AWS profile, for example as

and the credentials file:

You can obtain your access_key and secret_access_key from your aws account.

To install the S3AsyncManager system wide (or in your virtualenv), clone this repo

and install it with pip:

Afterwords, you can use it as shown in the example folder.



Monitor an error log with python and RabbitMQ

Nowadays there are many professional solutions to monitor your application for the errors. Some web frameworks have even build-in tools or support plugins to catch the programming exceptions and act accordingly.

Anyway, I wanted just to build a simple proof of concept how to monitor the web server error file and, when an event occurs and the file is changed, the monitoring script should send out an email. To monitor the log file I used pyinotify python module. This is an implementation on top of inotify, offering an easy interface to interact with the changes of the filesystem.

Continue reading

Example for singleton decorator pattern in python

I know there is not very common in Python to use the singleton pattern, but I found a nice implementation of this pattern in Python 3 Patterns, Recipes and Idioms book. Starting with that example I implemented an equivalent of the well known PHP getConnection example.

You have the code below:

This is the class that implements the Singleton pattern.

Now, we create a class and we decorate it with the Singleton class. Let’s import also MySQLdb module *.

Let’s test this:

You will see something like:

As you can see there is only one object.

For fun, let’s remove the line “@Singleton” and re-run the example. This time you will see different objects:

You can find the fully example here.

* If you don’t know how to install MySQLdb, you can check the previous post.

Install MySQLdb for Python 2.x

If you want to install MySQLdb for Python 2.x, without using the package manager of your Linux distro (Ubuntu in our case), you can do it through pip like this:

First you must to have installed libmysqlclient-dev (and, of course pip)

After that execute this two commands:

Test if is working:

Type in the command line:

and in the python console:

That’s all. Enjoy!