Grabbing Hanuman's Long Tail – Jacob Singh online

Drupal 7’s major API code freeze is behind us so time to take stock of the effect of the massive API overhauls and the hotly debated new interfaces and how they effect performance. As part of the last sprint at Acquia, I was tasked with comparing the performance of Drupal 6 and Drupal 7 in similar conditions so we know how much work we all have to do before Drupal 7 is ready for release.

So how does D7 match up with D6?

Of men who have a sense of honor, more come through alive than are slain, but from those who flee comes neither glory nor any help.
Homer, The Iliad

The Legend of the Drupal release cycle
In every Drupal development and release cycle, there is a period of rampant innovation by thousands of people distributed throughout the world. They get on IRC, email lists, in the flesh, etc and just bang out a ton of great code. Then the dust settles and the code freezes. Everyone wakes up after 3 days of recovery sleep, finishes their caffeinated beverage of choice, and tries out their new super-duper-fantastic toy to see how it works. It’s full of creaky limbs and flashy lightbulbs attached to misplaced handles. It gets poked and prodded, and then, we take it around the block to see how it runs and what we need to do to harden the security and make it as fast as possible (two areas Drupal has always excelled in). Then we all pitch in to get the bug fixing and performance tuning done.

Drupal 6 was of course a major refinement over Drupal 5 without creating too many waves for developers or user interfaces. It was also a little bit slower. Drupal 7 looks sexy, has much more consistency in its APIs, a kick ass database abstraction layer, a powerful ORM in fields. However as is expected in this pre-release stage, I have found in my testing that Drupal 7 in the current stage is slower, and now is the time to focus on performance.

Disclaimer: These are very preliminary numbers using a new benchmarking setup (which is described below). Neither the methodology nor the reports are perfect, so please do you own benchmarks (I also cover that later).

Summary (Cliff notes)

• As expected, Drupal 7 is slower as it is pre-release and more feature rich.
• For anonymous (cached) browsing, D7 is close on /taxonomy/1 and /node/1, it is much slower on /node
• Authenticated users browsing is about 2-3x slower
• User operations (login, user page, logout) are about 3-4x slower

What / how are we testing- Check the lanes

Target machine: IBM T60p thinkpad (Ubuntu 9.10, 2.16Ghz Core Duo + 2GB Dual channel RAM).
Testing machine: MacBookPro 2.5ghz 4GB of RAM

Testing machine ran jmeter and the two boxes were connected via LAN cables to minimize the network effect.

Testing platform – Set up the pins

To start with, we need a reproducible environment of fake data to test against. As an attempt to create such a standard, I started the NorthDrop project on Drupal.org. It is an install profile which uses devel_generate (dumy content generation module) to make fake content depending on settings provided during the install.

I got devel_generate mostly working for D7 and I backported NorthDrop to D6 so we could get two installs with almost identical types and amounts of content.

For the purposes of this test and not having someone call the society for the preservation of old and battered laptops, I put it on the “small setting” which includes:

$sample_sizes['small'] = array();
$sample_sizes['small']['nodes'] = 200;
$sample_sizes['small']['comments'] = 4; // Per node
$sample_sizes['small']['users'] = 50;
$sample_sizes['small']['terms'] = 15;
$sample_sizes['small']['vocabs'] = 3;

Testing format – The approach…

For this basic profiling test, I built three thread groups. A thread group is a set of fake users who will do the same routine (hit a few paths / submit forms) for a certain number of loops. In this case, here are my thread groups:

All three thread groups run simultaneously and the results are saved to xml files which are later views / processed.
For verification of results, I used the highly scientific eye-ball method of, “eh… that’s pretty close” after running each test 3-4 times.

Results – The Scorecard

Although I highlighted the important conclusions in the summary, here are the tables it is derived from:
Benchmark Data for Drupal 6 / Drupal 7

And here is a nice histogram built by this awesome script by the folks at Atlassian (makers of JIRA and Fisheye).

This shows the response times for Authenticated users as percentage of requests. (click to see a larger image).

Drupal 6:

Drupal 7:

What next? – The long drive home

Profiling

We need to examine the causes in more detail. This type of basic performance testing gives us some clues as to what pages / content cause load. Next we have to open up xcache and start getting into the nitty gritty to identify what needs to change.

Improved tests, isolation tests

I’ve posted the jmx I used for testing to a new github repo for this. If I get time, I will be writing another post to outline how this jmx was built and what it takes to run it. hint: it’s really easy! . It would be great if we built tests which just test one facet at a time, and also if we profiled more write heavy ops like commenting, content creation, etc.

Automated testing

The framework is in place to automate this, especially in D7 since the northdrop profile can be installed from the CLI. Jmeter can take params, and the jmetergraph.pl program can give us a good visual. Everyone’s eventual goal is something like testbot to run after every commit or perhaps on certain patches to give us an indication of what effect a change will have on general performance. We’re just dev hours and a server farm away from getting this set-up.

Resource profiling

Looking from the outside, we just get response times. We need to also identify what is making it slow from a system level. Is it MySQL? Is it PHP? RAM or CPU? are we I/O limited in some operations? Sometimes we can go backwards in performance, but forwards in scalability. This should be accounted for.

Good bye!

If you want to get the test files used for this report they are here.
A special thanks to greg_harvey and Graham Taylor for sending me a starter jmx they had built previously for functional tests. I hope this post is useful in spurring the discussion of D7 performance, please feel free to leave a comment on pajamadesign.com or on my Acquia blog.

There is still time to fix these performance issues so dive in.

As some of you know, I’ve been doing some performance testing of Drupal recently.

I’ve also started a project on github for this work. I’ll be updating the documentation here, there and elsewhere as more time is invested, but for now, here is a brief intro to what jmeter looks like and how it can be used to profile Drupal.

Video on Drupal profiling with jmeter

Apologies for not making zooms, etc. If it is not viewable, let me know and I’ll spend a bit more time.

Best,
Jacob

(Received in an email… really sounds like a Sufi, but it’s not AFAICT from the name).

Be Drunk
by Charles Baudelaire
Translated by Louis Simpson

You have to be always drunk. That’s all there is to it—it’s the only
way. So as not to feel the horrible burden of time that breaks your
back and bends you to the earth, you have to be continually drunk.
But on what? Wine, poetry or virtue, as you wish. But be drunk.
And if sometimes, on the steps of a palace or the green grass of a
ditch, in the mournful solitude of your room, you wake again,
drunkenness already diminishing or gone, ask the wind, the wave, the
star, the bird, the clock, everything that is flying, everything that
is groaning, everything that is rolling, everything that is singing,
everything that is speaking. . .ask what time it is and wind, wave,
star, bird, clock will answer you: “It is time to be drunk! So as not
to be the martyred slaves of time, be drunk, be continually drunk! On
wine, on poetry or on virtue as you wish.”

In my last post, I showed how updating modules may look in Drupal 7. If you didn’t see it it is here

In this post, I’m going to throw up some “dream” wireframes which may or may not make the ver 1 cut, but are perhaps good start in the right direction.

Hopefully, at least parts of them will be practical to implement, or at least spur discussion.

Without further ado:

Check out the clickable wireframes

And give feedback here:
Plugin Manager in Core: Part 3 (integration with installation system)

The problem: Updates in Drupal require FTP / SSH and a bit of know how

When the average Drupal site owner without ssh, cvs and other geek gadgets wants to update modules on or themes on their Drupal site, they currently have to do the following:

1. Go update status and see the mod is out of date
2. Take the site offline
3. Make a backup (if they can)
4. Know where to find the module on d.o., download the tarball
5. Unzip the tarball
6. Remove the current directory
7. Use FTP to upload the new directory
8. Run update.php

We’re trying to provide a way that users can get the same user friendliness of a package manager like Synaptic. Where updates and new installs are just a few clicks, no geek gadget belt.

I’ve entered the D7 ux fray, specifically focusing my generous amount of Acquia community time on getting a project called the Plugin Manager spruced up and into core.

For more background on the effort, see: Plugin Manager in Core (part 1).

The solution: make Drupal update like everything else.

Here is the issue:
Plugin Manager Part 2 : The update status UI

I’ve been working out some wireframes of how the process might look, and I wanted to share them with the planet to see what people thought of them. So without further ado:

Check out the clickable wireframes

Round 2

I know this blog is about 99% work focused, but I just have to rant in the hopes that my measly page rank will make it known to the world that nba.com’s International League Pass (you pay $200 to watch NBA games on the internet) is a total scam.

They use a player called RayV which is basically crowd sourcing bandwidth using a really badly made p2p player and running the games back on a schedule.

I’ve been fiddling with it for 5 months now with zero support. This product is a piece of garbage. Not even a piece, I can’t insult garbage because at least garbage can, at times be recycled and be of use. Hell, if I had a pile of garbage, I would spend an additional $200 to fedex it to the CEO of RayV and the idiot at nba.com who decided this was a good company to partner with.

I have all the system requirements met and get a 1mbps connection, but constant buffering and studdering (especially during crunch time in the 4th quarter) has made it the worst $200 and 20hrs I have spent this year, and I still can’t watch my beloved Celtics!

Aparently, I’m far from alone in thinking that RayV and NBA’s League pass sucks

As many of you know, I’m working on a Hosted Search product at Acquia. We’re building a pretty cool page where you can get some analytics on your search index and what people are searching for. Here is the deets on Dries’s site (hope he doesn’t mind ;0 )

Uploaded with plasq’s Skitch!

For this, we’re using Splunk which is a tad more pricey than I’d like, but a really amazing tool. Basically, it is grep + awk + a kilo of coke + a dozen redbulls + a Ferrari Testerosa + the same HGH A-Rod has been chewing. I’ll write more about it at some point, but this screen shot should give you an idea:

Uploaded with plasq’s Skitch!

Anyway, we use Splunk’s API to grab data into acquia.com and show the page above. The page was taking 10 seconds to load… I was stumped. Splunk seemed so fast, a couple seconds is reasonable for loading a report from millions of records, but 10 seconds was pretty extreme.

Eventually we discovered it was not Splunk at all, but a separate call in our code to a webservice (Call it Info Server) in EC2 which was being firewalled by Amazon. This caused the request to sit there for 10 seconds, and then timeout.

Here’s how security groups work:

I’ve got 2 servers:
Web Server – Serves static files (:80 and :443) and passes tough stuff to app server
App Server – serves requests back up to web server on :8080

Web Server needs to be able to access App server to push proxied requests through.

In EC2, each server has 1(or more) security groups. A security group is a list of access rights. These can be by Port & IP Range or they can be references to other groups. (wtf?)

Yeah, so the rule for the web server would probably be something like:
IP:any Port:80
IP:any Port:443
IP:111.111.111.111/24 Port:10000 (maybe some admin port for a certain location to access)

For the App Server, we don’t want 8080 world readable. We also don’t know the IP of the web server because this is elastic baby, servers can’t stand still. That’s why we give group permissions. So it looks like:

Group: Web Server

Which means any server launched on your account with the security group “Web Server” will have total access to any server launched with the security group “App Server”. Got it?

If not, here is an FBI style blackout picture which might make it more clear:

Uploaded with plasq’s Skitch!

In our case, we had a problem because we were referencing the external IP of our server(Info Server). See in the depths of the Amazon, each machine has a public IP and a private IP. So when you look for infoserver.acquia.com (made up, btw) it will resolve to 74.x.x.x When you try to look for ec2-10-45-123-41.compute.aws…. it will resolve to 10.24.134.41 and both point to the same place. The difference of course is that the security group settings only apply when you are using the internal IP even if both servers are inside the cloud

Hope you’ve been saved some pain.

Please come checkout Peter Wolanin and I as we present the future of Drupal search (we hope) at DrupalCon!

Hopefully with my new awesome job, I’ll have time to continue blogging here, but a couple quick hits about my impressions thus far at Acquia.

• Acquia has really smart people running the show. Everyone, even the non-techies are really techie.
• They really take care of their people. I’m totally free to purchase whatever technology (within reason) that will help me do my job. They get it, it’s pound foolish to keep gadgets away from geeks.
• Chris really pushes scrum hard. We set our own limits, and we spend a lot of time planning for contingencies. It’s tough, and annoying, but it is reality, and I like that I am forced to be real before problems occur, not after.
• Acquia Drupal needs more adoption. I don’t know when the pickup will happen, but I really encourage Drupal shops to start using it. I fully believe there is a huge ROI here, and it will be healthy for the ecosystem
• Kieran is trouble, don’t go drinking with him (but do go)

I’m a big hoops fan, so I picked up on this one.

If you’re interested, I was responding to this post here:
http://www.celticsblog.com/index.php?option=com_content&task=view&id=3934&Itemid=189
But got to writing so damn much that I decided it was more of a blog post than a comment:

Your points are fine, but I think you are missing the crucial and more interesting one: Why is the media threatened to see a Black athlete Howard (who is obviously an ass) protesting a given ritual in a White owned sports complex where 99% of the players are Black? It is good to handle things maturely, etc as you mention, but I don’t think you got on Bird for saying “we’re playing like women out there” did you? It’s also about if and how much you are personally offended. How did you feel when Marvin Gaye sang at the all-star game? The country flipped their lids worse then in this case and it is now being used on Team USA commercials!

This nation has been nothing but hate and violence against African Americans. I do not expect any African American to be patriotic. If they are, well that is their choice; but I think any white person should be ashamed to criticize the cynicism of an African American in regards to what this country has done for(to) them. That being said, I guess we should all be polite and protest respectfully, not tred on others, etc. Don’t get me wrong, Howard is no Jesse Owens, Tommie Smith, John Carlos or Muhammad Ali. But that doesn’t mean his point is not valid. Why can’t we discuss letting the players have some political freedom to dictate the rituals? Sure there are 80% white fans in the audience, but the players are almost all Black. Why don’t they sing “Lift Every Voice” or something else?

The National Anthem is flawed for a few reasons. Primarily, it is about a war fought by good old boys (many of whom were involved in the slave trade) for good old boys, not for poor Europeans, and certainly not for enslaved Africans or Native Americans. And secondarily, if has no relevance to African American history in the U.S. If anything, it only hearkened in an economic expansion which caused a brutal acceleration of slavery. I’d be angry too if I had to shut up and not say anything about a song written and revered by some guys who enslaved my family and who’s kids are still attacking me. Maybe I wouldn’t be respectful either.

Warning: Code is really in alpha state, however since this module is one of the most important and overlooked ones out there in terms of i18n, I thought I should mention it on the planet if others may want to pick it up and polish what is pretty much working now. Here is the issue for the drupal-5 port.

If you don’t know what Active Translation is, from the maintainer: Andrew aka drewish:

The Active Translation module is designed for internationalized sites where translated versions of content need to be displayed when available but if they’re not the original, untranslated content should be displayed. If several regional versions of a language are available these will be used as substitutions.

By default Drupal’s node listings show all content. So if you translate a node, both the original and translations are displayed. The Internationalization module, another contrib project, tried to improve on this on this. It gives you several options for rewriting queries to list nodes.

Keep in mind this is totally alpha code, but I thought since this is such a useful module, I should share it with everyone and encourage people to contribute to the effort.

Development was mainly sponsored by CivicActions.