Grabbing Hanuman’s Long Tail – Jacob Singh online

(Received in an email… really sounds like a Sufi, but it’s not AFAICT from the name).

Be Drunk
by Charles Baudelaire
Translated by Louis Simpson

You have to be always drunk. That’s all there is to it—it’s the only
way. So as not to feel the horrible burden of time that breaks your
back and bends you to the earth, you have to be continually drunk.
But on what? Wine, poetry or virtue, as you wish. But be drunk.
And if sometimes, on the steps of a palace or the green grass of a
ditch, in the mournful solitude of your room, you wake again,
drunkenness already diminishing or gone, ask the wind, the wave, the
star, the bird, the clock, everything that is flying, everything that
is groaning, everything that is rolling, everything that is singing,
everything that is speaking. . .ask what time it is and wind, wave,
star, bird, clock will answer you: “It is time to be drunk! So as not
to be the martyred slaves of time, be drunk, be continually drunk! On
wine, on poetry or on virtue as you wish.”

Okay, so that’s going a bit too far. But we’re getting ever closer to the dream module and theme updates and installs using a GUI in your browser!

Many thanks to cwgordon, Joshua Rogers, dww and especially chx for kicking some serious arse on this issue and getting us very close.

That’s right, in Drupal 7 you will be able to update your modules and themes without learning FTP, SSH or CVS.

Check out my latest screencast

and get involved.

Also PLEASE vote for my session at DrupalCon Paris.

In my last post, I showed how updating modules may look in Drupal 7. If you didn’t see it it is here

In this post, I’m going to throw up some “dream” wireframes which may or may not make the ver 1 cut, but are perhaps good start in the right direction.

Hopefully, at least parts of them will be practical to implement, or at least spur discussion.

Without further ado:

Check out the clickable wireframes

And give feedback here:
Plugin Manager in Core: Part 3 (integration with installation system)

The problem: Updates in Drupal require FTP / SSH and a bit of know how

When the average Drupal site owner without ssh, cvs and other geek gadgets wants to update modules on or themes on their Drupal site, they currently have to do the following:

1. Go update status and see the mod is out of date
2. Take the site offline
3. Make a backup (if they can)
4. Know where to find the module on d.o., download the tarball
5. Unzip the tarball
6. Remove the current directory
7. Use FTP to upload the new directory
8. Run update.php

We’re trying to provide a way that users can get the same user friendliness of a package manager like Synaptic. Where updates and new installs are just a few clicks, no geek gadget belt.

I’ve entered the D7 ux fray, specifically focusing my generous amount of Acquia community time on getting a project called the Plugin Manager spruced up and into core.

For more background on the effort, see: Plugin Manager in Core (part 1).

The solution: make Drupal update like everything else.

Here is the issue:
Plugin Manager Part 2 : The update status UI

I’ve been working out some wireframes of how the process might look, and I wanted to share them with the planet to see what people thought of them. So without further ado:

Check out the clickable wireframes

Round 2

Hey, stay out of my index!

So when I first joined Acquia, my fledgling Solr hosting service had IP based security. You, the customer could tell me what IPs you were going to connect with, and I would allow access to your search index from those IPs.

One of the first major tasks was to implement HMAC based authentication to the service to ensure against man-in-the-middle attacks and provide a way to use from any IP. Also, it is standard operating procedure for other Acquia services.

Fail first!

In the first iteration, we built something on the load balancers (which run nginx) because it provided a central point of access control, the balancers were under-utilized and we didn’t have to mess with the Solr code.

This worked okay for awhile, and was decently fast but was quite flaky as some stupid developer had the brilliant idea to implement it as python middleware with fcgi (flup). That developer was me.

Don’t fail second!

So to combat the unstable nature of the fcgi protocol, and to make things a little more efficient, I (along with help from Peter Wolanin and Douglas Hubler) rebuilt it in Java using a Servlet Filter. This was a royal pain the butt, as Java is pretty tricky when it comes to input streams and buffers.

Thankfully the results are worth it:

It’s hard to tell from this graph because of the peak, but the median stayed almost the same (blue line), and the average decreases pretty significantly (purple) as does the 90% line (yellow). Click the image to see it larger.

This graph shows the standard deviation (blue) in addition to the previous numbers and describes more acutely what the previous graph suggests, that is, the previous implementation was not any slower really, but less consistent, causing some of the requests to take much longer than others.

So there you have, Acquia Search is both secure and fast and now 200% more reliably fast

A while back, I wrote a daemon. No, I’m not a satanist mom, it’s a program which will basically stick around and manage a bunch of other little minions as they server content via unix sockets to a webserver (nginx).

The point here is to take in traffic from nginx via python and do something with it. For this I found an excellent tutorial which got me started:

http://www.p16blog.com/p16/2008/11/quick-demo-of-python-wsgi-nginx.html

This worked great, but then you needed to write something to manage all the little unix sockets, start them when they died, etc.

So I had to custom write something (at least I thought) as nothing in existence seemed suited for the task. It has worked “okay” but is having some mysterious problems under heavy real world load, and I needed to find something more robust for the task.

I recently stumbled across:
http://just-another.net/2009/01/18/byteflowdjangosupervisordnginx-win/

This thing looks perfect, but I can’t quite get it work… Basically, supervisord is a python application which has a very usefull and usable configuration file to specify programs you would like to run as services. It replaces 90% of the init.d scripts in existence I imagine.

In theory, you create a block like this:

; Production setup
[fcgi-program:gate]
socket=tcp://127.0.0.1:1212  ; We reference this later in nginx
command = /usr/local/solrflare/bin/gate.py  ; Calls the above code

This means, that when I run supervisord, it starts a daemon which will fire up my python script (which currently looks like this):

#!/usr/bin/python
from flup.server.fcgi import WSGIServer
import time, os, sys

def app(environ, start_response):
        status = "200 OK"
        response_headers = [('Content-type', 'text/plain')]
        start_response(status, response_headers)
        return ["If a thread dies in the middle of a request, and noone is a around to hear it, does it give a status code?\n"]
WSGIServer(app).run()

And Supervisord provides a nice little web interface to monitor and manage the daemon, also provides a nice interactive shell program and XML-RPC! (among many other cool features).

When this works, it will be awesome because I can throw out a lot of code (which I love to do). However, currently it just kinda sits there when I curl the port… doesn’t do anything, doesn’t log anything.

Update: I got it working! And it is awesome. I just needed to deal w/ an nginx config issue I created and do some permissions wrangling. It is running great so far!

I know this blog is about 99% work focused, but I just have to rant in the hopes that my measly page rank will make it known to the world that nba.com’s International League Pass (you pay $200 to watch NBA games on the internet) is a total scam.

They use a player called RayV which is basically crowd sourcing bandwidth using a really badly made p2p player and running the games back on a schedule.

I’ve been fiddling with it for 5 months now with zero support. This product is a piece of garbage. Not even a piece, I can’t insult garbage because at least garbage can, at times be recycled and be of use. Hell, if I had a pile of garbage, I would spend an additional $200 to fedex it to the CEO of RayV and the idiot at nba.com who decided this was a good company to partner with.

I have all the system requirements met and get a 1mbps connection, but constant buffering and studdering (especially during crunch time in the 4th quarter) has made it the worst $200 and 20hrs I have spent this year, and I still can’t watch my beloved Celtics!

Aparently, I’m far from alone in thinking that RayV and NBA’s League pass sucks

Just want to make a quick update and say that at long last my search project is on the internets and getting decent uptake.

Peter Wolanin and I presented at DrupalCon (don’t laugh too hard at me).

I think the reception so far has been great, and the servers have been champs We’re getting more and more signups every day.

One really cool one is Bryan from the CMS Report:
http://cmsreport.com/search/apachesolr_search/Drupal%20Search

If any of you out there want to find out how search can change how you build your sites and bring more people together with pages they need. Just click this link now:

http://acquia.com/products-services/acquia-search

A lot of people are excited but worried about trying out the free beta because we haven’t released any pricing, and something this cool will certainly cost more than a few pesos.

Well, fear not Drupalers, we’ve heard your call and are working on releasing some preliminary pricing soon. We wanted to wait until the beta got rolling a bit before we did this, but we want people to know we have a commitment to making this technology available to as wide an audience as possible. Stay tuned to my blog and/or the planet/acquia.com for more updates on this.

In the meantime, signup for the beta, it really only takes 15 minutes to setup and won’t break your site or lock you in at all. Please give us feedback! We know there are a few usability hiccups still in the signup process, and we’d love your input so we can fix ‘em. Thanks!

As many of you know, I’m working on a Hosted Search product at Acquia. We’re building a pretty cool page where you can get some analytics on your search index and what people are searching for. Here is the deets on Dries’s site (hope he doesn’t mind ;0 )

Uploaded with plasq’s Skitch!

For this, we’re using Splunk which is a tad more pricey than I’d like, but a really amazing tool. Basically, it is grep + awk + a kilo of coke + a dozen redbulls + a Ferrari Testerosa + the same HGH A-Rod has been chewing. I’ll write more about it at some point, but this screen shot should give you an idea:

Uploaded with plasq’s Skitch!

Anyway, we use Splunk’s API to grab data into acquia.com and show the page above. The page was taking 10 seconds to load… I was stumped. Splunk seemed so fast, a couple seconds is reasonable for loading a report from millions of records, but 10 seconds was pretty extreme.

Eventually we discovered it was not Splunk at all, but a separate call in our code to a webservice (Call it Info Server) in EC2 which was being firewalled by Amazon. This caused the request to sit there for 10 seconds, and then timeout.

Here’s how security groups work:

I’ve got 2 servers:
Web Server – Serves static files (:80 and :443) and passes tough stuff to app server
App Server – serves requests back up to web server on :8080

Web Server needs to be able to access App server to push proxied requests through.

In EC2, each server has 1(or more) security groups. A security group is a list of access rights. These can be by Port & IP Range or they can be references to other groups. (wtf?)

Yeah, so the rule for the web server would probably be something like:
IP:any Port:80
IP:any Port:443
IP:111.111.111.111/24 Port:10000 (maybe some admin port for a certain location to access)

For the App Server, we don’t want 8080 world readable. We also don’t know the IP of the web server because this is elastic baby, servers can’t stand still. That’s why we give group permissions. So it looks like:

Group: Web Server

Which means any server launched on your account with the security group “Web Server” will have total access to any server launched with the security group “App Server”. Got it?

If not, here is an FBI style blackout picture which might make it more clear:

Uploaded with plasq’s Skitch!

In our case, we had a problem because we were referencing the external IP of our server(Info Server). See in the depths of the Amazon, each machine has a public IP and a private IP. So when you look for infoserver.acquia.com (made up, btw) it will resolve to 74.x.x.x When you try to look for ec2-10-45-123-41.compute.aws…. it will resolve to 10.24.134.41 and both point to the same place. The difference of course is that the security group settings only apply when you are using the internal IP even if both servers are inside the cloud

Hope you’ve been saved some pain.

Please come checkout Peter Wolanin and I as we present the future of Drupal search (we hope) at DrupalCon!

Jeff Noyes (our Simplicity Guru), Linea Rowe, Peter Wolanin and myself sat down to discuss how the install process for our Hosted Search Service would look (yes, we’re getting close – Private Beta is out in two weeks)! Typically, when you have a faceted search engine, there is a set of filters on the left and search results on the right, with the sorting links generally horizontally aligned somewhere near the search box.

Here are a few examples from around the web:

Uploaded with plasq’s Skitch!

Uploaded with plasq’s Skitch!

Uploaded with plasq’s Skitch!

And here is our current implementation:

Uploaded with plasq’s Skitch!

Here is the same shot, but broken down in “drupalish”

Uploaded with plasq’s Skitch!

I think it works okay, but we’re concerned that when people enable the module, they will have a hard time getting this together. Here is a series of screen shots of a user, enabling and setting up the module:

Uploaded with plasq’s Skitch!

This part is simple (if you use Acquia’s hosted search), you just enable one module and you are done configuring the connection to Solr.

However, you standard search ends up looking like this:

Uploaded with plasq’s Skitch!

To get all the nice sorting and facet filters, you need to know (somehow) to go to admin/build/blocks and drag the ApacheSolr: blocks into regions like this:

Uploaded with plasq’s Skitch!

So what do people think? Should we just enable a few blocks “out of the box” and hope you are using garland and have a region named “left” or “left-sidebar”? If so, which blocks? Alternately, how can we provide a good workflow for people to know they need to do that extra step to setup their search. The other option Jeff suggested (which is most usable) is to have one block, where you can select what filters you want in it. The downside is that the user loses flexibility about where to but filters (maybe they want sorting on the right, etc).

I’d like to get some feedback on:

A). How to make this process so simple that it really is just checking that one box on the modules page and letting cron run and it looks great for 90% of users.

B). What the default blocks to enable are, and where they should be on the screen

C). How do we address this problem of multi-step installs which want to setup blocks in a more usable way for newbies?

See ya!
jacob