Monitor an error log with python and RabbitMQ

Nowadays there are many professional solutions to monitor your application for the errors. Some web frameworks have even build-in tools or support plugins to catch the programming exceptions and act accordingly.

Anyway, I wanted just to build a simple proof of concept how to monitor the web server error file and, when an event occurs and the file is changed, the monitoring script should send out an email. To monitor the log file I used pyinotify python module. This is an implementation on top of inotify, offering an easy interface to interact with the changes of the filesystem.

The code to monitor the file is very simple:

wm = pyinotify.WatchManager()
mask = pyinotify.IN_MODIFY
handler = LogMonitor(config.log_file, channel)
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch(config.log_file, mask, rec=True)
notifier.loop()

As you can see, I use a separate class, LogMonitor, that get as parameters the file to be watched and the another parameter, channel. About this last parameter I will discuss later on, below. The class LogMonitor needs to extend pyinotify.ProcessEvent class in order to implement our own event handlers by defining the appropriate methods. You can read more about Subclassing ProcessEvent in the pyinotify’s documentation. 

The LogMonitor class is fairly simple:

class LogMonitor(pyinotify.ProcessEvent):

    def __init__(self, filename, channel):
        self.fh = open(filename, 'r')
        self.channel = channel
        file_size = os.stat(filename).st_size
        self.fh.seek(file_size)

    def process_IN_MODIFY(self, event):
        line = self.fh.readline()
        self.channel.basic_publish(exchange='logs', routing_key='', body=line)

    # TODO - should handle logrotate, deletion, creation of a new file

You can notice that in the __init__ method, I just seek at the end of the file, so I skip the previous events and the monitor script will get the new lines in the log.

The code implements only IN_MODIFY event. This one is triggered when the monitored file is changed, but the code should also handle:

IN_CLOSE_WRITE - the monitored file was closed (it can happen, for example, when the web server (or another tool) rotates the log file) 
IN_CLOSE_NOWRITE -  an unwritable file was closed
IN_MOVE_SELF - the monitored file was moved. 
IN_DELETE_SELF - the monitored file was deleted

You can find a complete list of the Event codes in the pyinotify documentation.

Because the events can occur sometimes very fast, the monitor script will “publish” the event to a RabbitMQ server.  I will not cover in this tutorial the RabbitMQ part. I just mention I used pika python module to talk with the RabbitMQ server.

The publish to RabbitMQ is done with this line of code:

self.channel.basic_publish(exchange='logs', routing_key='', body=line)

After the event is published in RabbitMQ, I used a subscriber (or event receiver) that will monitor the RabbitMQ queue and for every event will send me an email with the error. Of course in a production environment, probably it will be better to send the alert message to a monitoring tool (like Opsview).

The subscriber code is also pretty simple:

def callback(ch, method, properties, body):
    print " [x] %r" % (body,)
    send_email(config.report['email'], 'Error', body)

if __name__ == "__main__":
    connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
    channel = connection.channel()

    channel.exchange_declare(exchange='logs', type='fanout')

    result = channel.queue_declare(exclusive=True)
    queue_name = result.method.queue

    channel.queue_bind(exchange='logs', queue=queue_name)

    print ' [*] Waiting for logs. To exit press CTRL+C'

    channel.basic_consume(callback, queue=queue_name, no_ack=True)

    channel.start_consuming()

As you can see, I created a connection to the RabbitMQ server on localhost and I subscribe to the ‘logs’ channel. The line of code:

channel.basic_consume(callback, queue=queue_name, no_ack=True)

will call my callback function every time when a new message is coming in this queue. The callback function will print the message and it will call the send_email() function.

You can find the complete code on my github account. Feel free to use it or to contribute to it!

If you found useful this blog post, please click some of the share buttons below. Thanks! 🙂

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.