May 2009 Archives

Recently I got in a drunken conversation with another long time developer and consultant about what makes software projects succeed.

We both had many years of experience with many different projects, so it was particularly gratifying that we largely agreed with each other (or was that the alcohol?). Having seen many people write their prescriptions to project success, I thought I'd add my 2 cents in the form of my observations. I would consider project success to be that the software is delivered, the clients are satisfied with the result and there were no surprises on time or budget (time/budget could be higher than initially planned, but any slippage was identified and explained well in advance). As a technical person, I also include the subjective measure that the codebase would not cause embarrassment (no WTFs here).

Below are the some characteristics of successful projects I have seen, in no particular order. None of them is sufficient by themselves and I've seen projects succeed when some of these characteristics were glaringly absent. However, I can't remember seeing any project succeed without more than a few of these present.

• Good People. If not the most factor important for success, certainly the most important for enjoyment. I actually think that smart, motivated people who want to get things done can ameliorate many project difficulties. Thus forming a team of good developers should be the first goal of any project. Note here I haven't said "Smart People". While being smart certainly helps it's is not everything (I have a little disagreement with Google's "only the best" hiring practices here). On more than one occasion I have seen some very smart people get tied up in irrelevancies and doing things "right" resulting in nothing being done at all!
• Political Support or Cover. The senior management have to want the project to succeed (not always a given in some political firms!). Otherwise, the project will have problems getting the resources it needs. Similarly in larger organisations a project needs to have managerial support, both in terms of the problem being solved and the manner it will be solved. I was once at a place where the senior IT management were all old waterfall process COBOL coders and distrusted iterative development. There was so much friction and so many people just waiting to say "I told you so" (on both sides) that forward movement was difficult. At another place with similarly minded senior management, our immediate manager provided us with cover. To higher management he presented reports conforming to their waterfall style, but allowed us to run the project as we saw fit.
• Well Understood Problem. The more similar the problem being solved is to previous problems the team has solved the more effectively pitfalls will be avoided. This is one of the premises behind prototyping - solve the exact same problem twice and the second time should be much easier.
• Tight Iterations with End-user Interaction. I started commercial work just as iterative development was becoming popular (we were taught waterfall at university) and I'm a big believer. I have found iterations lasting just a week or so (or as short as possible while still getting some decent work in each iteration) and then deployed to an environment where an end-user can see it (and hopefully try it out) work best. The idea is to get regular and high quality feedback on whether the project is heading in the right direction as well as priorities for future work and changes required. Every project I have been on has significantly changed requirements from start to end. The chances for project success are greatly enhanced if change is accepted and handled. Indeed the best option is to proactively seek out the changes required and this is where short iterations with user feedback help immensely.
• Small Teams. I was once on a project with over 100 other technical people. The back and forth of meetings and coordination was inordinately complex. I think Brooks had this right in The Mythical Man Month, and teams should be kept no larger than around 9 people. That number seems to work well and if they are good people, a great deal can be achieved. On that 100 person project, I strongly believe that the 9 best people in a single team would have done much, much better than the other 91 as a team.
• Flexible Process. I wrote about this before, basically I think software projects work best where they are done by people who look to understand their environment and can adapt - not just retrying the same old thing that worked last time (for you or someone else).

I've been coding hard the last week or two on a secret squirrel project, thus this blog has been a little neglected. Fear not, I have a few posts planned once I have some spare time. For the meantime, here is an After School Special in honour of the UK inflation figures announced today.

When I was in primary school there was a strange little fad for a couple of months that illustrated some concepts in supply and demand. For some reason, I don't remember why, one student agreed to become another's servant. I think the "servant" was Jason, but the "master" was definitely Ben. In return for following Ben around and doing the occasional little task, Jason would receive a dollar a day. It wasn't a real dollar. Instead every morning Ben would present Jason with a piece of paper (about the size of an old Australian dollar note, or at least the best approximation thereof that could be managed by ripping up our workbooks) with "1 Dollar" written on it and Ben's signature. Jason didn't think this too bad a deal since Ben was the cool kid in class and getting to hang out with him was a blessing.

After a couple of days this agreement showed no sign of being forgotten. Instead, Ben even hired a second servant. At this point a number of other kids in the class decided they wanted servants too, and thus the problems began. About a third of the class wanted servants, but less than a third wanted to be a servant. A day's wage rocketed. Over consecutive days we started to see handmade $2, $5, $10 and $20 bills passed around. Some kids tried to make their notes more ornate and artistic to entice potential servants to prefer their money, which seemed to work for a bit.

By this stage some of the servants were building decent piles of worthless paper. There was a constant problem of what to use the money for beyond servants. Some was passed around as servants subcontracted or paid for small tasks. Occasionally someone might swap sides to get or spend cash. However, people were beginning to lose interest in the concept. There wasn't much of an economy until Jason (definitely him this time) starting making rings. By a ring I mean wire soldered into a small loop with a shard of glass at the join. With this innovation a whole range of backyard jewellery was produced, most of the class got involved as master, servant, jeweller or consumer. Some people from other classes joined in too. Prices continued their rise for another week or so but at a less frenzied pace as people were hitting the limits of being able make money by hand (strangely no one thought of producing a note with greater than $100 face value, the largest real Australian note available at the time). Then people began to print money.

Literally print money. I went round Ben's place one Sunday (he lived just round the corner) and he had obtained a toy printing press and paper guillotine. With each crank he produced six $100 notes and he was cranking hard! On Monday he instantly became the richest kid in class. By Wednesday the economy collapsed and we (the boys at least) went back to playing soccer and A-Team.

Update: The weather_report project and the weather archive is no more. The effort required to maintain it was too great, for more information see here.

Today I have released my first Ruby gem: weather_report, version 0.0.1. It connects to the BBC Backstage weather API and gets weather observations or forecasts for thousands of cities worldwide. Don't be fooled by the low version number, it is usable. However, it can only handle the BBC weather feed, requires pre-knowledge of BBC weather location ids and is metric only (all things I plan to fix).

require 'weather_report'
# 8 is the BBC Backstage weather code for London, UK
londonWeather = WeatherReport.new(8)

# to get the current temperature
londonWeather.observation.temperature 

# to get the tomorrow's max temperature
londonWeather.forecast.for_tomorrow.max_temperature

# and much more!

sudo gem install weather_report

There is also a website with documentation here.

I would like to say thanks to the authors of newgem and this rubyforge tutorial, both of which greatly aided in weather_report's creation.

A while ago I wrote a blog post asking is there dumb money on Intrade? The short answer was probably not once the implied interest rate was considered, unless a chain of nearly expired "easy money" trades could be constructed. When I heard that the Goggle app engine supported Java I was keen to try it out. So I ported my manual Intrade interest rate calculation to Java and had it read the Intrade XML feed to check for any contracts finishing soon with high implied interest rates. Unfortunately, the code fell foul of Google's time limit on reading data from other websites. Rather than see the code go to waste, I decided to clean it up and release it. It is a command line Java app with all the code available under the MIT license. It's probably best to just load it up in Eclipse and run the launch file to see it working. Enjoy!

You can download the whole eclipse project here.

On Saturday I played a round of golf at the Central London Golf Centre. The allure of this place is that it is the nearest golf course to me in South London. They also have a driving range, but I didn't use this. Parking is available and there were quite a few spaces empty on the weekend I went. Nearest train station is Earlsfield, about 15-20 minutes walk away. Booking is required for weekends, but tee time was fairly flexible. We arrived late but still started when ready. It cost £20 for 9 holes and club hire (cheaper on weekdays) - very reasonable. The hired clubs were only a random half-set, but between the two of us we had everything except a 3 wood, so we shared. If you need to hire and can't share clubs with your group, then check what you are given carefully.

It is a par 31, 9-hole course totalling 2,332 yards. Much of the detail I saw online spoke of an 18-hole course, but this involves playing each hole twice. There are four par 4's and the rest are par 3. It is a very straight course, the flag is clearly visible from the tee on all holes. Only on the 9th is there any impediment to a straight driver - a small lake. The holes are fairly tightly packed and we often encountered players on other holes playing across ours (and did the same a few times ourselves). It is a friendly course, good for beginners or a quick no stress round. If I lived nearer I'd definitely consider a membership, but my friend pointed out that it might get a little boring after regular play.

One caveat though, the food at the clubhouse is awful. Walk into Earlsfield if you want something to eat.

Previously I wrote about big companies not using Ruby, Scala or similar new languages because they are not mature enough, adding that this probably wouldn't apply in small companies. This implies an underlying assumption - that big and small companies are different. This might be obvious to most readers, but based on my experience it isn't to many people I see working in software. Much of the software development advice around is only useful for one type of organisation, although people get hung up on trying to apply it to everyone.

To me, as far as a software team is concerned, a big company is one where there is a sizeable layer of bureaucracy. This can manifest as a large number of political players, passive stakeholders who can greatly change requirements, seemingly arbitrary requirements, or just filling in forms and otherwise jumping through hoops. This is not to say that all of this is unnecessary - some degree of bureaucracy is required to stop developers going off on complete flights of fancy. Also government regulation often requires development to follow a set path; a developer doing their own thing could unknowingly expose the company to sanctions or lawsuits. There seems to be a correlation between the traditional measures of a big firm and bureaucracy (employees and revenues), but they don't always align. I have worked for a very profitable 50K+ employee firm and been allowed to do largely as I wished. While at a 50 person startup in a highly regulated (and litigious!) domain, I was drowned in paperwork.

Each system has its own advantages & disadvantages. Personally I prefer working in smaller companies, or at least in parts of large companies that have more of a small company feel. However, far more developers work under big firm conditions, and I have met good developers who prefer the system. So when I read about avoiding corner cases I think "yeah, lets get things done", but at the same time realise this might not work in an IT department of a large commercial bank. I can also empathise with Eoin Woods about how a Software Architect needs to be able to find and involve all the stakeholders of a project, especially the hidden, passive ones that can stop a project right before it goes into production - a lesson I learnt the hard way. Although, most small firms wouldn't even need a Software Architect. Obviously the context of the advice is crucial.

Some of the best advice I ever received in software development was while working at a small Rational Unified Process consultancy in the late 90's. The RUP is a large and detailed beast often favoured by large companies, although the consultancy successfully applied it to many clients large and small. This was done by never straying from the spirit of the process, but very regularly deviating from the written word of the process. I remember one of the owners there telling me the first step of the RUP on a project was to work out which bits of it you should throw out as unnecessary (the implication being that most of it would be bypassed).

Basically, there is no silver bullet. Advice is just that, advice. Think about how the advice fits your situation - what are the premises, and don't just blindly follow it. I think software projects work best where they are done by people who look to understand their environment and can adapt - not just retrying the same old thing that worked last time (for you or someone else).