September 20th, 2008

On the Josh Howard Media Frenzy

I’m a big hoops fan, so I picked up on this one.

If you’re interested, I was responding to this post here:
http://www.celticsblog.com/index.php?option=com_content&task=view&id=3934&Itemid=189
But got to writing so damn much that I decided it was more of a blog post than a comment:

Your points are fine, but I think you are missing the crucial and more interesting one: Why is the media threatened to see a Black athlete Howard (who is obviously an ass) protesting a given ritual in a White owned sports complex where 99% of the players are Black? It is good to handle things maturely, etc as you mention, but I don’t think you got on Bird for saying “we’re playing like women out there” did you? It’s also about if and how much you are personally offended. How did you feel when Marvin Gaye sang at the all-star game? The country flipped their lids worse then in this case and it is now being used on Team USA commercials!

This nation has been nothing but hate and violence against African Americans. I do not expect any African American to be patriotic. If they are, well that is their choice; but I think any white person should be ashamed to criticize the cynicism of an African American in regards to what this country has done for(to) them. That being said, I guess we should all be polite and protest respectfully, not tred on others, etc. Don’t get me wrong, Howard is no Jesse Owens, Tommie Smith, John Carlos or Muhammad Ali. But that doesn’t mean his point is not valid. Why can’t we discuss letting the players have some political freedom to dictate the rituals? Sure there are 80% white fans in the audience, but the players are almost all Black. Why don’t they sing “Lift Every Voice” or something else?

The National Anthem is flawed for a few reasons. Primarily, it is about a war fought by good old boys (many of whom were involved in the slave trade) for good old boys, not for poor Europeans, and certainly not for enslaved Africans or Native Americans. And secondarily, if has no relevance to African American history in the U.S. If anything, it only hearkened in an economic expansion which caused a brutal acceleration of slavery. I’d be angry too if I had to shut up and not say anything about a song written and revered by some guys who enslaved my family and who’s kids are still attacking me. Maybe I wouldn’t be respectful either.

September 19th, 2008

Adding custom placeholders to your log in python2.4

Python has a great Logging Engine which is at once powerful and simple (like python). One of the features added in 2.5 was the ability to provide a dictionary of extra params to the logging functions to allow for custom replacements in your formatter string. Unfortunately for me, I had to move my entire infrastructure for my Solr hosting project over to RightScale, and RightScale plays well with CentOS, and Python 2.5 doesn’t :(

If you are wondering how to get this same functionality in 2.4, here is how:


import logging
from logging import _srcfile, LogRecord
# Custom logger that uses our log record
class Python25_Logger(logging.getLoggerClass()):
    def makeRecord(self, name, level, fn, lno, msg, args, exc_info, extra):
        """
        A factory method which can be overridden in subclasses to create
        specialized LogRecords.
        """
        rv = LogRecord(name, level, fn, lno, msg, args, exc_info)
        if extra:
            for key in extra:
                if (key in ["message", "asctime"]) or (key in rv.__dict__):
                    raise KeyError("Attempt to overwrite %r in LogRecord" % key)
                rv.__dict__[key] = extra[key]
        return rv

    def _log(self, level, msg, args, exc_info=None, extra={}):
        """
        Low-level logging routine which creates a LogRecord and then calls
        all the handlers of this logger to handle the record.
        """
        if _srcfile:
            fn, lno, func = self.findCaller()
        else:
            fn, lno, func = "(unknown file)", 0, "(unknown function)"
        if exc_info:
            if type(exc_info) != types.TupleType:
                exc_info = sys.exc_info()
        record = self.makeRecord(self.name, level, fn, lno, msg, args, exc_info, extra)
        self.handle(record)

    def findCaller(self):
        """
        Unfortunately, this also needs to be overridden because the trace is one level up now.
        (HACK)
        """

        try:
            raise Exception
        except:
            f = sys.exc_traceback.tb_frame.f_back.f_back
        rv = "(unknown file)", 0, "(unknown function)"
        while hasattr(f, "f_code"):
            co = f.f_code
            filename = os.path.normcase(co.co_filename)
            if filename == _srcfile:
                f = f.f_back
                continue
            rv = (filename, f.f_lineno, co.co_name)
            break
        return rv

And to implement it:


logging.setLoggerClass(Python25_Logger)
log = logging.getLogger('testlogger')
formatter = logging.Formatter('%(name)s: level=%(levelname)s module=%(module)s special=%(special)s: %(message)s')

console = logging.StreamHandler()
console.setFormatter(formatter)
console.setLevel(logging.INFO)
log.addHandler(console)

#Here we are passing along the extra param "special"
log.error('hi mom', extra={'special':'cake'})

This code will output:

testlogger: level=ERROR module=WHATEVER special=cake: hi mom

September 12th, 2008

Delhi Drupal Meetup Tomorrow!!

Sorry for the late notice folks, but I figured we should just try to make this happen:

Calling all drupal wale in Delhi!

We’re meeting at the offices of Srijan in Nehru place.

Directions and contact info are here:
http://groups.drupal.org/node/14727#comment-49495

and the node regarding them meetup (organized in about 3 days) is here:
http://groups.drupal.org/node/14727

There will certainly be 4-5 people there, so please do show up!

At present there is no formal agenda, but Srijan will be providing a projector so we can do some lightning talks, or watch a thrilling DrupalCon presentation or two :) Anyway, see you then.

All the best,
Jacob

September 2nd, 2008

How to - gzip compression between Drupal and nginx / apache

I was working on trying to speed up the transmission and reduce the bandwidth of a web service I’ve been building in EC2. I’m using nginx (en-JINN-ex) as my load balancer and the cloud because, it is awesome.

A little overview for people who have lives and think an HTTP Header is a football term:

When you send a request to a server via your browser a series of Headers(little variables) are sent across which tells the server what type of browser you are, what language you prefer, and a whole host of other interesting info.

When you get data back from the webserver, it also sends back some headers like the status of the request (200 is OK, 404 is a url with no content, etc), the length of the content, type of content, etc.

When a server is using mod_deflate or mod_gzip in apache (or The Gzip Module for Nginx), then it is capable of sending back its content in a compressed (gzipped) form. All modern browsers support transparent decryption on the browser side. What this means is that you see a normal HTML page which is 70k, but only 15 or 20k went over the pipe because of compression. Cool, right? Chances are it just happened as you saw this page.

Now how does a web server know you can accept gzip’d content? Well, you send one of those headers that looks like this:


Accept-Encoding: gzip,deflate

and the server responds with


Content-Encoding: gzip

By default, php does not transparently support gzip encoding, but it can be done. See the following:



if (function_exists('gzinflate')) {
        $gz_on = true;
    }

    if ($gz_on) {
    // Tells the drupal_http_request function to send this header over.
        $headers = array (
            'Accept-Encoding' => 'gzip,deflate',
        );
    }

    $return = drupal_http_request($url,$headers,'GET'); 

    // This checks to make sure that we actually got gzip'd content back
    if ($gz_on == true && stristr($return->headers['Content-Encoding'],'gzip')) {
        //First 10 chars are junk
        $string = substr($return->data, 10);
        $output = gzinflate($string);
    } else {
        $output = $return->data;
    }

Go ahead and try it with $gz_on false and true. You'll see that $return->data will have different lengths if it is compressed or not, but $output will be the same.

One caveat when working with nginx (this one hurt after 2 hrs):
From the nginx manual

Turns gzip compression on or off depending on the HTTP request version.

When HTTP version 1.0 is used, the Vary: Accept-Encoding header is not set. As this can lead to proxy cache corruption, consider adding it with add_header. Also note that the Content-Length header is not set when using either version. Keepalives will therefore be impossible with version 1.0, while for 1.1 it is handled by chunked transfers.

Drupal uses HTTP 1.0 for some reason… I don’t know why it uses version 1.0, but I filed an issue about the same, because 1.1 is a lot better and pretty much ubiquitous IIRC.

Happy Header Hacking!

How To find me

Telephone: +1 510.277.0891 | Email: jacobsingh at gmail daht calm

Solution Graphics