Grabbing Hanuman's Long Tail – Jacob Singh online

A while back, I wrote a daemon. No, I’m not a satanist mom, it’s a program which will basically stick around and manage a bunch of other little minions as they server content via unix sockets to a webserver (nginx).

The point here is to take in traffic from nginx via python and do something with it. For this I found an excellent tutorial which got me started:

http://www.p16blog.com/p16/2008/11/quick-demo-of-python-wsgi-nginx.html

This worked great, but then you needed to write something to manage all the little unix sockets, start them when they died, etc.

So I had to custom write something (at least I thought) as nothing in existence seemed suited for the task. It has worked “okay” but is having some mysterious problems under heavy real world load, and I needed to find something more robust for the task.

I recently stumbled across:
http://just-another.net/2009/01/18/byteflowdjangosupervisordnginx-win/

This thing looks perfect, but I can’t quite get it work… Basically, supervisord is a python application which has a very usefull and usable configuration file to specify programs you would like to run as services. It replaces 90% of the init.d scripts in existence I imagine.

In theory, you create a block like this:

; Production setup
[fcgi-program:gate]
socket=tcp://127.0.0.1:1212  ; We reference this later in nginx
command = /usr/local/solrflare/bin/gate.py  ; Calls the above code

This means, that when I run supervisord, it starts a daemon which will fire up my python script (which currently looks like this):

#!/usr/bin/python
from flup.server.fcgi import WSGIServer
import time, os, sys

def app(environ, start_response):
        status = "200 OK"
        response_headers = [('Content-type', 'text/plain')]
        start_response(status, response_headers)
        return ["If a thread dies in the middle of a request, and noone is a around to hear it, does it give a status code?\n"]
WSGIServer(app).run()

And Supervisord provides a nice little web interface to monitor and manage the daemon, also provides a nice interactive shell program and XML-RPC! (among many other cool features).

When this works, it will be awesome because I can throw out a lot of code (which I love to do). However, currently it just kinda sits there when I curl the port… doesn’t do anything, doesn’t log anything.

Update: I got it working! And it is awesome. I just needed to deal w/ an nginx config issue I created and do some permissions wrangling. It is running great so far!

Python has a great Logging Engine which is at once powerful and simple (like python). One of the features added in 2.5 was the ability to provide a dictionary of extra params to the logging functions to allow for custom replacements in your formatter string. Unfortunately for me, I had to move my entire infrastructure for my Solr hosting project over to RightScale, and RightScale plays well with CentOS, and Python 2.5 doesn’t

If you are wondering how to get this same functionality in 2.4, here is how:

import logging
from logging import _srcfile, LogRecord
# Custom logger that uses our log record
class Python25_Logger(logging.getLoggerClass()):
    def makeRecord(self, name, level, fn, lno, msg, args, exc_info, extra):
        """
        A factory method which can be overridden in subclasses to create
        specialized LogRecords.
        """
        rv = LogRecord(name, level, fn, lno, msg, args, exc_info)
        if extra:
            for key in extra:
                if (key in ["message", "asctime"]) or (key in rv.__dict__):
                    raise KeyError("Attempt to overwrite %r in LogRecord" % key)
                rv.__dict__[key] = extra[key]
        return rv

    def _log(self, level, msg, args, exc_info=None, extra={}):
        """
        Low-level logging routine which creates a LogRecord and then calls
        all the handlers of this logger to handle the record.
        """
        if _srcfile:
            fn, lno, func = self.findCaller()
        else:
            fn, lno, func = "(unknown file)", 0, "(unknown function)"
        if exc_info:
            if type(exc_info) != types.TupleType:
                exc_info = sys.exc_info()
        record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, extra)
        self.handle(record)

    def findCaller(self):
        """
        Unfortunately, this also needs to be overridden because the trace is one level up now.
        (HACK)
        """

        try:
            raise Exception
        except:
            f = sys.exc_traceback.tb_frame.f_back.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename == _srcfile:
                f = f.f_back
                continue
            rv = (filename, f.f_lineno, co.co_name)
            break
        return rv

And to implement it:

logging.setLoggerClass(Python25_Logger)
log = logging.getLogger('testlogger')
formatter = logging.Formatter('%(name)s: level=%(levelname)s module=%(module)s special=%(special)s: %(message)s')

console = logging.StreamHandler()
console.setFormatter(formatter)
console.setLevel(logging.INFO)
log.addHandler(console)

#Here we are passing along the extra param "special"
log.error('hi mom', extra={'special':'cake'})

This code will output:

testlogger: level=ERROR module=WHATEVER special=cake: hi mom