Upload asynchronously to Amazon S3 using Tornado

TornadoWeb is a great non-blocking web server written in Python and Boto3 is the Amazon Web Services (AWS) SDK for Python, which allows developers to write in a very easy manner software that makes use of Amazon services like S3. Unfortunately boto3 S3 wrapper is blocking and if you would just use it out of the box in a Tornado application it will block the main thread because it uses a synchronous HTTP client.

The solution is to use Tornado’s AsyncHTTPClient and do manually all the work which boto does for you under the hood.

I built a small replace of boto3 mthods for upload and delete (the only ones I need for the moment) which uses tornado’s AsyncHTTPClient and I published the code on github.

The main idea around this replacement is to use botocore to build the request (AWS wants the requests to be signed using different algorithms based on AWS zones and request data) and only to use the AsyncHTTPClient for the actual asynchronous call.

In order to use the S3AsyncManager you need to define an AWS profile, for example as

and the credentials file:

You can obtain your access_key and secret_access_key from your aws account.

To install the S3AsyncManager system wide (or in your virtualenv), clone this repo

and install it with pip:

Afterwords, you can use it as shown in the example folder.

 

 

Deploy cherrymusic using Nginx, Supervisord and Virtualenv on Ubuntu

Cherrymusic is a music streaming server written in python.

We assume the deployment is done in /home/user/music.domain.com

Cherrymusic

1. Go to the deployment folder and clone the cherrymusic repo

2. Create and enable the virtualenv

3. Test if the cherrymusic server starts and stop it afterwords

4. If you executed this commands under another user than the one under which you want to run cherrymusic

(eg: you ran the commands as root but you want to run under the user user)

5. Edit cherrymusic.conf from the user‘s home and set the basedir with the path where your music collection is stored.
eg: /var/music

Supervisord

1. Install supervisord

2. Create the file /etc/supervisor/conf.d/music.conf with this content

Ajust the path /home/user/music.domain.com and the value of the user with your values.

3. Reload supervisor service:

4. Check if the music service shows in the supervisor status:

You should see something like

5. Start the music service

6. Check if there is any error in the logs and if you can access the service

(ctrl+c)

Try to check the connectivity:

Should see something like:

( press: ctrl \ )

Also, if there is no firewall, should work to access from the browser: http://music.domain.com:8081/

Nginx

1. Install nginx webserver

2. Create the configuration file in /etc/nginx/sites-enabled/music.domain.com.conf

 

(Adjust the domain name and paths)

3. Reload nginx web server:

Test if everythings works from your browser: http://music.domain.com/

Firewall

Do not allow direct access to the application, but only through nginx.

 

You can find this tutorial on github.

Clear swap space and move to RAM

 

Google Chrome UI extremely large

Today I updated google-chrome-stable to version 43.0.2357.124-1 and I had an unpleasant surprise. Everything was looking bigger: the top bar, the bookmark bar, the menus, the font on the website. Something like this:

Screenshot from 2015-06-11 09:50:04
image-201

It seems the fix is to start the chrome browser with –force-device-scale-factor option.

Screenshot from 2015-06-11 09:55:01
image-202

And the result is:

Screenshot from 2015-06-11 09:57:14
image-203

The UI looks again like before.

Create a Cassandra cluster with OpsCenter on Amazon EC2

Today I played a little with Cassandra on Amazon EC2. It was a very user friendly and pleasant experience to deploy a cluster with 2 nodes in one region using DataStax OpsCenter.

First I started a m1.small instance in Amazon EC2 where I installed OpsCenter. For this I chose Centos 6, the official AMI. Before starting to install OpsCenter, we need to configure the firewall in order to be able to access it. In AWS console, under the Security group, there is “CentOS 6 -x86_64- – with Updates-6 – 2014-09-29-AutogenByAWSMP-“. We need to righ-click on it and Edit inbound rules. Here we add a new Custom TCP Rule with port 8888 and the Source IP: My IP.

Anyway, I noticed that the instance has also an iptables firewall and the port 8888 is not open. So, on the instance I did:

Now, we can install OpsCenter. All you need to do is to follow the installation guide for RPM package from DataStax:

1. Edit the file:

2. Add the repository for OpsCenter

3. Install and start OpsCenter

After the installation is finished and the service started, write in your browser: http://<YOUR_INSTANCE_IP>:8888 and you will see this nice screen.

Welcome to DataStax OpsCenter
image-154

From now on, it is pretty easy to setup a cluster with multiple nodes.

Just click “Create Brand New Cluster” and follow the steps.

You will need to add some information as in the image below:

create-cluster
image-155

  • The cluster Name
  • Your DataStax Credentials. If you do not know what these are, then you need to go to DataStax Registration page, fill your data and click “Download Now”. Don’t worry, nothing will be downloaded, but you will get an email with your username and password. These are your credentials you need to put in the form from OpsCenter.
  • The total number of nodes to be created (and installed with Cassandra) – be aware,  the current instance where OpsCenter is running is not counted. I created 2 nodes initially and I added another one later on.
  • The Amazon EC2 Credentials – these are needed because OpsCenter will launch the instances for you. You need only to select the Availability Zone and the Size of the instances.

The job is almost done. Now you need to click Build Cluster and wait while all the necessary software is installed.

cassandra-installing
image-156

After few minutes, you will have a Cassandra cluster with 2 nodes.

In the next tutorial I will describe how to add an extra node through OpsCenter to the current cluster.

Good luck!

Install SSL certificate for Nginx

Recently I bought an SSL certificate for this blog from MegaSSLStore. My website is hosted on a FreeBSD machine and served by Nginx web server. In order to install the certificate on this machine, I downloaded from MegaSSLStore the certificate and CSR+private key and I copied them on my server in /usr/local/etc/nginx/ssl

Because I have an .crt certificate and also a ca-bundle I need to combine these two files in one certificate:

After this, I changed the nginx website configuration file, in order to redirect all the traffic that is coming on http (port 80) on https (port 443).

In my website .conf file, I added a new server section in which I specified to redirect all the traffic that comes on port 80 to https, using the http response code 301 (Moved Permanently). Also in the old server section I removed the “listen 80” directive and I added “listen 443 ssl”.

The next step is to add the certificates into the configuration file. So, again in the nginx configuration of the website:

If you use the a default nginx config file, probably you will have a line like:

I replaced this line with:

in order to avoid some vulnerabilities old versions of SSL and I removed the old line ‘ssl_ciphers’ that was containing some weak ciphers and I replaced with:

After this I reloaded the nginx config file with:

In my case, I was using a CDN to deliver some assets (js, css files or images), but because it was over http I disabled it in order to not have mixed content on the same page, until I will add the certificates also to the CDN subdomain.

Monitor an error log with python and RabbitMQ

Nowadays there are many professional solutions to monitor your application for the errors. Some web frameworks have even build-in tools or support plugins to catch the programming exceptions and act accordingly.

Anyway, I wanted just to build a simple proof of concept how to monitor the web server error file and, when an event occurs and the file is changed, the monitoring script should send out an email. To monitor the log file I used pyinotify python module. This is an implementation on top of inotify, offering an easy interface to interact with the changes of the filesystem.

Continue reading