Generalizations are always wrong

I’m as bad as anyone when it comes to generalizations and simplifications.  Yet even as I state that nothing is ever that simple I still believe there’s a lot of value in generalizations.  They help us see patterns and anti-patterns.  I see even more value in the effort to simplify problems.  Much of what we do in building software is about creating a software model for real life messy scenarios.  We simplify because messy software is very expensive to write and maintain.  Ignoring unknown edge cases when architecting an application makes it possible to get something working quickly.  I don’t think you can do Agile without starting with a deliberately simple model.

I think the tension between Agile and Waterfall sums up the situation nicely.  They are yin and yang.  Agile works because the people involved have enough experience to identify a good model early on.  In other words, they already know the problem well enough to dispense with the rigourous process of Waterfall.  In practice I’ve found that we never know all of the problem enough so a good project is one that is balanced between up-front design and discovery through coding and user feedback.

Another common example of this is around performance.  Jeff Atwood generalizes that hardware is cheaper than programmers.  Lars totally disagrees.  This is hardly a new debate.  Anyone who has had to make a complex application go faster knows how much work and risk it involves.  Equally, anyone who has tried to get a complex application working (with incomplete and changing requirements) knows that optimizing too soon means you’ll never finish.  Both Jeff and Lars are trying to present a simplified view of the problem.  Both arguments are valid but lack the context of business needs.

The performance problem is often exacerbated by the lack of clear requirements.  “Make it fast” isn’t helpful.  “Ensure that account creation requests have a median response time less than 100ms and a 90th percentile less than 500ms” is better but

Good project management is about knowing when to use more hardware to buy time to build more perf.  It’s about defining performance goals in business terms that make it possible to prioritize the work.

Why I don’t own a Kindle

I would love to own a Kindle. I borrowed one for a week and even bought a book. It was easy enough to read and the convenience of having dozens of books in a single object. The battery life was good and although the UI wasn’t perfect it was good at staying out of the way.

The most obvious downside is the DRM. On so many levels I cannot countenance supporting any DRM regime. Experience has shown that sooner or later everyone will get screwed by DRM. Philosophically I cannot countenance supporting or encouraging any DRM regime although pragmatically it has been impossible to be entirely free of DRM.  I also understand that Amazon is walking a tight line here.  They want to establish the market and must appease the greedy and paranoid delusional publishers and author’s guild.  Roy Blount Jr. this means you.

The other major downside was the relative fragility of the physical device. A printed book is a remarkably robust thing. It can get somewhat wet, it can sustain intense shock, it never runs out of power.  The Kindle is pretty solid but it’s  no book.

In the week that I tried the Kindle I realized that I would probably trade the relative hardware fragility for the convenience.  After all, it’s only a few hundred and if a device is damaged one can always buy another and re-download all the content. Right? It turns that the answer is a DRM infested “not so much.”  How tragic that two weaknesses join forces to undermine the product entirely.  The utter lack of transparency by Amazon about download limits is at best an embarrassed omission and at worst a bald faced deception.  Even Apple did a better job of this.  I found out a little while ago that Apple let’s you reset your device count once per year.  In this case we’re talking about authorized computers but there’s no difference  That’s really smart.  It covers all kinds of legitimate scenarios but is nowhere near enough for any practical form of abuse.

Looking for wireframing and site map tools

I’ve been looking for tools to help with site maps and wireframing.  Visio is not the answer here.  I’m much more interested in the functional architecture of the page and site than pretty layouts.

In no particular order, here’s what I’ve found so far:

A few months ago I stumbled across something that looked great (and was open source) but now I can’t remember.  This is definitely an ongoing research project.  Please comment with other ideas.

Alternatives to DNS

While researching BigTable for the previous post about Hammers and nails and the rise of search engines over relational databases I came across mention of Google’s Chubby Lock Service.  This was inside an article about distributed lock managers and points out how they are in effect simple distributed file systems.  The article then goes on to suggest that Chubby has supplanted DNS inside Google.

I presume this to mean that instead of using the existing DNS protocol to distribute information they using Chubby (where the  value of the “lock” is the address of its name).  Do they have a custom DNS client stack?  I doubt it, more likely they have local DNS servers that act as proxies.

I wonder what actual gains they derive from this architecture.

Of hammers and nails

I’ve been a busy boy lately.  I’m in the early phases of two different projects, one of which is taking up all of my time and the other of which is taking up the rest of my time.  :)

One of my projects involved faceted search.  This is also known as search directed navigation.  What’s interesting is how this approach is becoming the new hotness.  Not only are multiple clients broaching the topic but as I become familiar with it I see faceted search as a solution to other projects as well.

Given how popular faceted search has become it’s surprising how few solutions are available.  The king of the hill these days appears to be Endeca.  I’ve been getting to know this quite a bit and while it is quite expensive it’s also really good.  Most people should be going this route.  If you’re building a platform then something like Solr is an option but if you plan to keep changing your search configuration, Endeca is hard to beat.  Do you really want to develop and maintain your own search system?  What business are you in?  Spread over a five to six year period, the cost of a commercially supported product like Endeca is a pretty easy sell.

I think we’re finally seeing the beginning of the end of the relational database era.  Relational databases will always have their place but developers are now able to apply other solutions.  Search based applications and BigTable like distributed stores are coming to the fore.  Even when you need transactional behaviour there are interesting locking models that are akin to row locking that cover many common scenarios.

When will the Rails of search based apps emerge?

Awkward photos

My wife has a well developed sense of the absurd. There are some things out there that just crack her up. Sometimes I just don’t get it but today she found a real gem.

I would caution everyone to check their parents’ photo albums before casting the first stone. I’m pretty sure we’ve all got a few awkward photo skeletons in the closet.


This is one of those songs that I’ll always just love.  Tonight, courtesy once again of Boing Boing, Courtney Woolsey with her own multi-track version of this classic.

There’s lots to like with the video and Courtney.  She has a nice voice, a great smile and the playful way she performs each of the tracks fits just so.  Solo performances of harmonies aren’t new but this is the first I’ve seen as a video.

It’s also interesting to reflect upon all the magic (good and bad) behind the scenes.

The video was made by a 19 year old girl at college on a personal computer.  Think about that.  That is why I love personal computers.

The video was made available to the entire world by a single person.  Think about that too.  That is why I love the Internet.  The Internet isn’t about ABC joining it’s about personal publishing and Courtney’s wonderful unique, personal voice.

The video was published on YouTube but it could have (and may have) been published on many different video sharing sites.  Although YouTube has a strong market position, the underlying technology has become commoditized to the point where different video sharing sites have a very hard time differentiating themselves.  But, they exist anyway and that is a good thing.  If YouTube were to become the only video portal then it’s corporate overlords would be unable to help themselves.  They’d get greedy and stupid and ruin it for everyone.  Unfortunately that kind of behaviour is all too common and takes decades to fix.

The odds are good that some automated music recognition system is trying to figure out what intellectual property was used and whether to issue a DMCA takedown notice or to send YouTube the bill.  It would be a crying shame if Courtney’s voice were to be stifled by fear of a new medium and greed.  We desperately need a new approach to intellectual property.

A roundup of Mac tools

Over time I am gradually mastering the finer aspects of Mac OS X.  As with any system a lot of value comes from third party utilities.

One recent discovery is something called sleepwatcher.  It will automatically run ~/.sleep and ~/.wakeup as your Mac goes to sleep and wakes up.  The first application of this is the following command:

hdiutil eject /Volumes/Encrypted

This unmounts my encrypted volume.  What’s the point of an encrypted volume if it’s always available?  The act of sleeping my Mac ensures that the encrypted data is locked tight.

Another useful tool is something called DropScript that wraps a file oriented shell command to a Mac app.  I’ve used this to create the Mac equivalent of the wonderful open command.  Why do I need this?  Because Firefox, Thunderbird and Postbox all insist on maintaining their own mappings between MIME types and applications to open them.  This almost makes sense but is quite annoying when all I really want to do is to defer the decision to the Mac default.  I currently do this quite naively with the following script:

open “$*”

Looking at the manual page it’s clear that there’s opportunity for refinement.  One minor benefit is that when I need to create a new association I’m always going to the same place (~/bin/

Finally there’s Tomboy.  Although I don’t use it all the time I like having it around.  It finally runs on the Mac although the port is somewhat simplistic so operations like copy and paste use Control-C and Control-V instead of Command-C and Command-V.

Got any cool tweaks for your Mac?