(for reference, see my last article on workflow configuration in drupal)

So being the programmer who will write something in a day which takes an hour to run to save 5 minutes, I decided to scratch a long standing itch:

The Drupal User System API.

I love the simplicity of the drupal user system, but like many things in drupal, it wants you to use forms to admin it, and the DB model is inherently tied to the controller (not good).

So I decided to contribute this to the helpers module. It provides some simple functions like:

helpers_user_help($section)
permissions_get(unknown_type $rid)
permissions_set(unknown_type $rid, unknown_type $perms = array)
role_add(string $name)
role_get(int $rid = 0, string $name = “”)
user_add_roles($uid, $roles = array)
user_delete_roles($uid, $roles = array)

Simple stuff, but AFAIK, lacking in drupal.

In addition, I also wrote a full set of simplete tests for it which can be found in the tests directory.

The patch can be found at:
http://drupal.org/node/252058
Hope others find this helpful!

-Jacob

For those of you who are unfamiliar with faceted search compare the following:

I did a search for WSXGA because I’m looking for a laptop with decent resolution on two sites.

Laptops Direct

vs.

New Egg

(click to enlarge image in new window)

The New Egg search lets me filter, so I know that I’m looking for a laptop between $750 -> $1000, I’ll get 5 results. After that filter, I’ll know what’s available, and the # per manufacturer etc.

Contrast that with an advanced search form where I have to put in all my criteria, and hope I get a result. I might also miss certain results if my vocabulary is bad, or I don’t understand that the website says “high resolution” instead of WSXGA, so I don’t select it.

I think it’s obvious to anyone why faceted search is a good thing. In my next post, I’ll be exploring why is hasn’t gotten widespread adoption, particularly in the small business / NGO sector, and how I plan to help change that.

This module allows you to show nodes in a view which have the same taxonomy as a node which you pass in as an argument. BUT it goes a step beyond this. By using a weighting table, one can specify the weight of each vocabulary in making the comparison.

Say you have two vocabularies on a movie site “Genre” and “Community Tags”, the latter being a free tagging taxonomy. With this module, you can give Genre a weight of 10 and Community tags a weight of 1. Which means that if I’m looking at “Meet the Parents”, I’m most likely to see a list of similar Romantic Comedies like “Sleepless in Seattle” and “French Kiss” with something like “Taxi” or “The Godfather II” coming lower in the list, because someone tagged both “Robert De niro”

See the Module Page for more information and how to use.

I’m going to be cleaning up the code a bit and providing a sample view soon, but any brave souls out there who would like to test / suggest, please go ahead!

I had a bit of a realization (or rather a resurgence of a recurring realization) that I enjoy writing. It happened this weekend as I was “getting away from it all”.

I’ve been interested in enterprise search for small and medium enterprises for a while now. Having implemented the Google Mini and the GSA, I’ve seen how a good search can really turn your information architecture on its head in a good way. Like any conversation between two entities be they two people or a person and a website, communication is difficult, and many of the same rules apply:

Vocabulary

You have to speak the same language. This doesn’t mean an Thai can’t talk to a Nigerian, but it does mean that when you are communicating, if the same word doesn’t mean the same thing (which it never does), your intentions, delivery and content is worthless. That is why non-verbal communication and communication over phones is so ineffective compared to face to face meetings. The words may be the same, but the interpretation never is.

So what does this have to do with enterprise search? When a user wants something from your website, they are looking for a keyword. Many computer scientists have tried to make linguistic aware search engines which correctly interpret sentences and question. Some of these results are useful, but generally, I believe people don’t come to a site thinking “Do you have any

red Toyota Corolla

.” Internally, they are simply thinking “Corolla” and “Red”. For instance, I could speak only two words of English, Red and Corolla, and chances are, I could walk into any American City and rent a red Toyota Corolla.

When one plans information architecture for a site, they usually start with Persona or stereotypes of users, which have goals. And then you define actions they take to meet those goals, and try to use the same vocabulary and thought process of these users to make an interface which is organized like their brain. But when you have 10 different Persona, how is this possible? And within your 10 stereotypes, there can be huge variation and outside of your assumptions, there may be other users you never thought about. By having an good search engine, even if you have one page about red Mustangs which is buried in your site, people might find it. By having an excellent search engine which has synonyms, facets, spell checking, related results, etc you may be able to help the user not only find what they thought they were looking for, but contextual information about it. What if there is a mechanic looking for parts who types in

1988 Corolla Fuel Pump

into a search, shouldn’t the search engine know what years the fuel pumps for Corollas available are the same and available, and allow him to filter? Shouldn’t it know that in the late eighties the Chevy Nova was a clone of the Corolla, and had the same parts cheaper?

This is the type of high value information which comes from dealing with a real human, and no amount of brilliant forethought in information architecture can pre-assume what the person is actually looking for. If I were doing IA for a parts website sans search, I’d have to have categories by model, by year, etc Even in a straightforward example like that, value is lost. That’s why search engines need to ask the extra question, and today’s search engines that most sites use are not.

Faceted Search

Next whenever, I’ll write something about faceted search (fancy name for search with fields and filters) and how I think the combination of Apache Solr and open source CMSs like Drupal, Typo3 and Joomla, are going to pave the way to an entirely new concept of information architecture and where we spend out usability testing money.

I feel compelled to announce this. Please Please Please read this post if you have done any multilingual drupal development.

Do not use t() outside of a hook or before locale gets to build its cache

I was asked to do some profiling on a colleague’s site, and I found this little doozy in the location_views module:

http://drupal.org/node/253813

The offending line is:

 define('LOCATION_VIEWS_UNKNOWN', t('unknown')); 

Okay, so what is the problem? When drupal starts up, it runs through many “bootstrappping states”:

Booting Drupal’s locale module

The last one is:

 case DRUPAL_BOOTSTRAP_FULL:
      require_once './includes/common.inc';
      _drupal_bootstrap_full();
      break;
  }

Now let’s look at the end of _drupal_bootstap_full:

// Load all enabled modules
  module_load_all();
  // Initialize the localization system.  Depends on i18n.module being loaded already.
  $locale = locale_initialize();

So the modules are getting loaded - module_load_all (they are included) before the locale module is initialized. In location_views, the offending statement is just interpreted upon inclusion because it isn’t wrapped in any hook. So it gets called before the locale_initialize() function.

This last call $locale = locale_initialize() sets to the global $locale variable to the iso language code the user is looking at the site in. If it is not set, see what happens when someone calls t():

function t($string, $args = 0) {
  global $locale;
  if (function_exists('locale') && $locale != 'en') {
    $string = locale($string);
  }

.......

Okay, so look at the first conditional. This will obviously return true every time, because locale is == to null when a module uses t() before locale is initialized. So if we dig into locale, what do we see:

function locale($string) {
  global $locale;
  static $locale_t;

  // Store database cached translations in a static var.
  if (!isset($locale_t)) {
    $cache = cache_get("locale:$locale", 'cache');

    if (!$cache) {
      locale_refresh_cache();
      $cache = cache_get("locale:$locale", 'cache');
    }
    $locale_t = unserialize($cache->data);
  }

So what happens?

We get here, because locale_t doesn’t exist yet:

$cache = cache_get("locale:$locale", 'cache');

We are trying to get a cache for “locale:”. Obviously, this does not exist. Because of this, locale says, okay, let’s refresh the cache.

The locale cache

The locale cache is a massive serialized array of strings and their matches from locales_source and locales_target. Let’s see how it is formed:

function locale_refresh_cache() {
  $languages = locale_supported_languages();

  foreach (array_keys($languages['name']) as $locale) {
    $result = db_query("SELECT s.source, t.translation, t.locale FROM {locales_source} s INNER JOIN {locales_target} t ON s.lid = t.lid WHERE t.locale = '%s' AND LENGTH(s.source) < 75", $locale);
    $t = array();
    while ($data = db_fetch_object($result)) {
      $t[$data->source] = (empty($data->translation) ? TRUE : $data->translation);
    }
    cache_set(”locale:$locale”, ‘cache’, serialize($t));
  }
}

OUCH!

This loops through each available language, and does a join selecting every string from locales source and locales target. It then builds a massive array (the bigger the translated site, the bigger the array) which eats up a huge amount of RAM - For reference, on http://www.amnesty.org, we’ve got over 10k translated strings, so we’re talking a few MB.

Then it serializes the whole thing - which uses a good amount of CPU, and then cache_set writes it to the cache table.

That’s where it gets really bad. When cache_set makes a write to the cache table. It first runs
LOCK TABLES cache;
then INSERT INTO cache…

So the cache table is effectively frozen up for anyone who wants to get data or put data into it. So think about this. If you have a 3MB locale cache array you are inserting into cache, and it takes 200ms to make that insert, any other user trying to access the site who wants to make queries to cache for variables is in line.

This creates a cascading effect where other processes which could have finished quickly are now holding up RAM in apache, waiting for access to the DB. If your server isn’t fast enough, this basically runs mysql’s process limit up to it’s max, and people start being unable to connect, the DB server gets partial inserts, deadlocks, all kinds of ugly stuff.

What’s worse, if t() is used outside of a hook, it will fire on EVERY page load. So on the site I was doing the profiling for, it uses a lot of AJAX. So every AJAX request was actually running this massive insert as well!

In xdebug, running on my sandbox, I was able to bring the avg page load time down from about 3-4seconds to 300-500ms by the simple patch referenced above.

Conclusion

• Don’t ever run t() outside of a hook
• Don’t ever run t() on non-static strings (if you have enough of them, this same thing will happen every time a new one appears in the system
• Watch out for cache_sets in your application. They can be a real silent killer. Everything will work fine, but you are killing yourself performance wise. I suggest using xdebug, and if nothing else, go into cache_set, and add debug_print_backtrace(); to just see everyone who is using it.

For those who don’t know, here is an excerpt from the project page of media mover:

Media Mover is a set of modules which allows website administrators to easily create complex file conversion processes. The core of Media Mover is the media_mover_api module which creates a set of rules allowing multiple modules to interact with a file. Media Mover can take a file emailed to an email account, turn a file attachment into an FLV file, create a new node with the file data, and then save the file on an external file storage like Amazon’s S3 all at once. And that’s just the start.

Wow! So does mm live up the hype? Of course. Arthur wrote it.

What I’ve been looking into the integration of Media Mover with workflow-ng and asset. Media Mover’s strengths right now are harvesting from multiple sources like ftp servers, email accounts, local stores, etc… and processing the video / audio / image via ffmpeg.

I’d say the main weakness it has is a drupal waekness, and that is how do files exist in the drupal node paradigm? Where is their meta-data stored?

There are so many different ways people do this task, generally with modules like image or video (which is basically file as a node). Or with modules like videofield, imagefield, etc which is file in files table, and a reference from a CCK field.

This kinda works, but asset provides a much more robust media integration framework, wherein an asset has metadata, it has formatting options and it can be embeded in the text body, or attached as a cck value.

The first goal of the integration is use the store hook in media mover to store the incoming media as an asset. Secondly, we’re going to build a store function to create a new node, and place the asset in a CCK field for that node.

The ultimate scenario is to create multiple assets for multiple processing instructions, so we have a folder of low res videos and a folder of high res videos, and we are automagically somehow adding the created assets to CCK fields (think I have a store selling video, and I have a “preview” CCK asset field and a “full version” CCK asset field).

There is inherent data model problems here both with asset and MM, but let’s see what we can do.

More next week when I’ve written the module, but if any Media Mover users are reading this, how is your experience with MM? What do you perceive as its lackings (if any) and what can be done to make it simpler / more intuitive for you?

-J

PHPitis is another condition I have been suffering from. It’s not early as painful and has provided me with a decent income and professional growth. It is also not a pussy inflamed organ with no apparent value pressing up against all the nicely formed and functioning organs around it. However, the feeling that there is something not right “down there” remains.

I’ve always known that python is the best language out there. I don’t mean to start yet another stupid language fight, because every language is fine and dandy. But honestly, every programmer I respect has told me the same thing, they wish they could just write python. The only knocks against python are:

• Lack of hosting options
• Lack of Jobs / Skilled personnel

The first one, is obviously stupid, because it is as easy to run as PHP, but is totally chicken and egg. The second is pretty much the same. You have to be a better programmer to write python, and so there are less people who can, so there are less CTOs wanting to risk a language with less people available.

Anyway, for what it’s worth, I love python, and I’m not that good at it yet, because I never get jobs in it, and it’s hard to get clients signed up for something that isn’t hosted anywhere…. Until now.

A day before I went to the hospital for an examination into yet another bout of my awful stomach pains which turned into the aforementioned surgery, I got my App Engine key in the mail.

I am affraid of the big G as much as anyone, and this offering is too new to judge. I won’t bore everyone with the questions everyone is asking about if your code can be moved or not, what is the guarantee they will keep it around, etc. I just want to say that the data api and the security of knowing that such terse syntax with so much power will scale till the end of the googleverse is exciting.

I highly recommend everyone give it a whirl, and discover / rediscover beautiful code. I was thinking of “killer apps” to build on it. My first inclination is that the “killer app” for the gapjinn (as I have now started calling it) is something with the following characteristics:

Uses Google Apps data apis

As much as possible, the application should store it’s data in google sites, google docs, google calendar, etc. This is because google will always have a commitment to making these two platforms work together AND you will not be made irrelevant because you can’t integrate.

Uses Google Apps provisioning APIs

Because SaaS is in, and you will want to make setup super easy.

Uses Google for authentication

Because you will not survive in the enterprise if you can’t take advantage of single sign-on services. As long as you are using all the google apps stuff, you should authenticate the same way.

Is niche enough that it does something better / easier / more focused than native google apps
OR

Provides an integration between different google apps and a 3rd party service

On the first count, it’s about creating a wrapper, basically a new interface layer to an existing google apps platform. Sometimes simple is too simple. For instance, using a series of tags and and a small amount of data store in the Google BigTable, I bet you create a really quick and dirty CRM using google contacts + gmail + google sites + google calendar.

On the second, I think the potential to provide bridge functionality between more mature best of breed apps like basecamp, unfuddled, salesforce, will make quick enterprise dashboards possible. Something like a dashboard / middleware layer between this stuff.

Let’s see…

http://www.polarion.org/index.php?page=overview&project=subtrain

It is a fantastic set of training materials generously provided under CC 2.5 by Polarion.org.

it features ppts for admins as well as users, evaluation and feedback materials, as well as sets of exercises to simulate normal working environments. I can’t say enough about these materials, they are really top notch.

Major cred goes to the good folks providing them. Please check them out!

Getting Started

Who are we and what are we doing here? Backgrounds and goals (personal and technical).

• Get to know everyone’s development background and professional history

• Where do we come from as programmers? How does that influence our current approach and approach to learning drupal?

• What do we want to learn and specialize in?

General goals of this training

• Make developing in Drupal fun

• Increase the speed at which you can develop solutions

• Decrease the amount of time you have to spend on donkey work

• Introduce you to the Drupal community and how to leverage / participate in it

• Make the connections between requirements and stock solutions

• Know how and when to hack it

Specific Objectives for today

• Know how to set up drupal sites using multisite deployment

• Understand how to use update status, CVS and drush

• Learn the devel module, masquerade module, and other development tools

• Understand the drupal architecture:

  • What IOC is

  • Why is successful

  • Why is it annoying

Planned Format

• Devs spend as much time in front of computers practicing as possible

• Topics are discussed, examples are shown in existing modules, and practice on a custom module

• Every days ends with ½hr of Q&A

Tools of the trade

• Devel Module

• Coder

• Masquerade

• Project Set-Up and sandbox creation

• admin_menu

• Drush, CVS and update_status

The Drupal Community

• IRC

• Issue Trackers

• groups.drupal.org

• Forums

• drupalmodules.com

• drupal.org/planet

Drupal Architecture

• IOC (Inversion of control model)

• The Hook system

• Drupal Bootsrap process

• Theming Engines and themes

Tea Party

• Learning the hook system via the drink module

Main Core Hooks and their usage

http://api.drupal.org/api/group/hooks/5

hook_menu

Example: contact

hook_nodeapi

Example: path module

hook_help

Example: contact module

hook_user

Example: contact module

To Hack or Not to Hack, that is the question

Form API

Building an admin settings form

• callback to drupal_get_form

• validate hook

• submit hook

Implementing form_alter

Implementing a custom submit hook on a form_alter

Real site planning and development in Drupal

Modules you can’t live without

• CCK

• Views

• Panels 2

• nodequeue

• adminrole

• pathauto

• nodereference

• imagecache

• asset or image + imagefield

• tinymce

How do you build this page:

• List of Nodes w/ RSS & Pager

  • Views

• Custom content type which has related nodes in a heirarchy

  • CCK + nodereference

• Thumbnails in node listing

  • Asset + imagecache

• Cultivated (admin chosen) list of nodes

  • nodequeue

  • views

• Sortable table of nodes and their values

  • Views + Table style plugin

• Complex front page containing any of the above

  • Panels 2

PHPitis is another condition I have been suffering from. It’s not early as painful and has provided me with a decent income and professional growth. It is also not a pussy inflamed organ with no apparent value pressing up against all the nicely formed and functioning organs around it. However, the feeling that there is something not right “down there” remains.

I’ve always known that python is the best language out there. I don’t mean to start yet another stupid language fight, because every language is fine and dandy. But honestly, every programmer I respect has told me the same thing, they wish they could just write python. The only knocks against python are:

• Lack of hosting options
• Lack of Jobs / Skilled personnel

The first one, is obviously stupid, because it is as easy to run as PHP, but is totally chicken and egg. The second is pretty much the same. You have to be a better programmer to write python, and so there are less people who can, so there are less CTOs wanting to risk a language with less people available, so they go with PHP, and get speghetti code, because 90% of PHP developers aren’t that good.

Anyway, for what it’s worth, I love python, and I’m not that good at it yet, because I never get jobs in it, and it’s hard to get clients signed up for something that isn’t hosted anywhere…. Until now.

A day before I went to the hospital for an examination into yet another bout of my awful stomach pains which turned into the aforementioned surgery, I got my App Engine key. During my few days of bed rest without net, I took it for a spin.

I’m as afraid of the big G as everyone else, and I hate the idea of locking my code up in a proprietary platform too. But suspend those doubts for a minute and let me just tell you, that this thing is pretty cool. Being able to write really elegant code while being able to scale to the end of the googleverse is pretty exciting.

I’m not going to repeat the questions everyone is asking about portability, sustainability, etc. I’m looking at what would be a “killer app” that could be developed and deployed in 6 months and that would be successful on this platform. Here are the attributes I would expect it to have:

Uses Google Apps data apis

If you’ve got the power of google calendar behind you, why spend forever building the database schema to support a calendaring application, it will never be as good. If you need a wiki, use sites. It’s all there, and google will always provide more and more support for integration. Plus, you reap all the benefits of your data being interoperable with other formats these services export to. No brainer.

Uses Google Apps Provisioing apis

SaaS is in forever. This means setup must be really simple for users. They shouldn’t have to do a run around to setup google apps before using your service.

Uses Google Authentication

To survive in the enterprise, you need the single signon stuff. Plus, they need it already if you want to do the first two.

Provides a better / enhanced interface to existing google apps

OR

Provides a 3rd party middleware / conversion layer to other software

On the first count, I think that you could provide something which makes it easier to use google apps, or provided some value added service there. For instance, a simple CRM system which is a mashup of Contacts, gmail, sites and calendar would probably not be too much of a stretch.

On the second, there are lots of opportunities here, to be the glue, which makes you an attractive meal for both google, and the 3rd party you are integrating with. good for short term wins.

Makes Real Money (is not ad supported)

I feel that the architecture of google apps is not for making content rich sites. It’s really more for building tools which do something. Ad supported sites which ACTUALLY make any money (as in profit, as in real money) are so rare. All of them are either content producers or content organizers. If you want to do something like that, go for a CMS like Drupal or something, and make your .02 per click on viagra ads. I don’t think it’s a very exiting place to be.

The Sew-up

What is this blasphemy? I thought you were a linux free software guy? You are encouraging me to place all my data in google and make my app totally dependent on google? Yeah, but I’m also a husband, and a son and I have rent + huge medical bills. I think there is a golden opportunity here, and the tech is fun to play with. There is no point doing this IMO, if you don’t want to marry google for this app. It won’t make sense for everything you build, but if you’re going to do it, go whole hog and get bought. Of course, there are millions thinking the same thing, but this platform has the potential to have lots of really small apps which get in, get a pain point solved and get out - bought - or at least with a low recovery time.