August 2009 Archives

I recently listened to a Software Engineering Radio podcast on Sustainable Architecture. It was an interesting listen despite much of it being given over to discussion of well known and generally good software architecture practices, such as loosely coupled design. It also spent a little too much time on defining software architecture - everyone defines terms to suit their own needs.

However, the podcast did raise a couple of good points that anecdotally feel right and I hadn't heard before. It talked about an option space when designing software systems. That is, that there are a number of ways to design and implement a system. Many developers come up with one way to build a system based on the requirements; then stop designing and start building. This podcast advises looking at multiple ways to solve the problem, thus allowing a determination of which parts of the system will be hard to change or replace at a later date. How costly a decision is to change should be a major determinant of which direction to take. The podcast suggests the result will be easier to change in response to unknown future demands while at the same time not requiring possibly unnecessary effort in the present (to handle these unknown future changes). Interesting.

Recently I have been thinking hard about threading in a Java project of mine, and it definitely requires a great deal of hard thinking. When I started working with Java there weren't any native threads, so it was just easier to make everything single threaded as the performance gain was minimal. However, as time has passed more and more of my code both at work and home has needed to work multi-threaded. As I learn more about Java threading, I realise how wrong I was doing it before - a cycle repeated a few times in the last 12 years. It was only after reading the great book Java Concurrency In Practice a few years ago that I realised that in Java "locking is not just about mutual exclusion; it is also about memory visibility." I can only hope that this is the last such cycle, but realise it's probably not.

1. Talking to other developers there seems to be a standard progression in Java multi-threaded programming skill:
2. Just write single threaded programs
3. Write multi-threaded programs by marking everything synchronized
4. Use wait() and notifyAll() to build up "threadsafe" collection classes
5. Realise your "thread-safe" collection classes aren't particularly thread-safe and try using immutable domain objects with stateless controllers to work around the problem
6. Trust in the Java API concurrent utilities, with its collections, locks and executors
7. Realise that these classes don't solve all the problems and start thinking hard about all the issues in your code
8. Become very scared about Java threading

I think the next step may be using another language that doesn't allow shared state. Scala uses an Actor model to asynchronously pass messages between threads. There is no shared memory so no threading problems. This sounds like a good way to go, but I still work in Java for most of my programming (not many Scala or Erlang jobs around!).

This whole situation reminds me of memory management in C++. There were simple rules to follow to reduce memory leakage, but the problem was hard to completely solve (especially to a newbie programmer) and normally your program still leaked. I remember leaning heavily on Purify. Then Java came along with its automatic memory management and the problem largely disappeared. Developers could focus on the business logic of their code rather than housekeeping. Hopefully a similar thing will happen with Java threading.

Having read a few more chapters of my blender book, it is time to try another historical model (my first was a sculpture in Tozeur). Most Roman archaeological sites include a few amphora displayed behind glass. At Pompeii they just stack them up all over the place. This tutorial details the steps I used in modelling an amphora based on photos from Pompeii. Below is one of these photos next to the results of my modelling.

The continuation of this post is a set by step tutorial detailing my methodology.

For a recent project I had to connect to two different read-only web services. Both were from commercial vendors providing very similar information (which I aggregated). One API was REST based the other used SOAP. The REST interface took me a few hours to write, the SOAP one took days!

For the REST interface I had to send requests to a URL that somewhat indicated what I was attempting to do - eg http://example.site/contracts/123 to get the details of contract with id 123. An XML was returned and using a SAX parser constructed my internal objects. In total it took 340 lines of my code to handle the entire web service. Easy!

Using the SOAP interface required me to download their WSDL and then construct a model of their domain objects - I used Apache Axis2 (not Apache Axis though, that is an old and unmaintained project that does the same thing - a confusion that cost me some time). Unfortunately the WSDL was wrong and the model didn't generate properly. I had to dive into the 2000 line WSDL file to find and fix the error - and it was near unreadable! Try looking at a commercial WSDL file some time. After some trial and error the model was right and when surrounded by copious error handling code it could be sent and data received from the service. It took 810 lines of my code, plus hundreds more in the generated model. Urgh. At this point one might suggest it was just a problem caused by a dodgy service provider. However, I have since added a third read-only web service to the project. It is also SOAP based and just as difficult to get running properly - apart from having to modify the WSDL since theirs was correct from the start.

I also have experience from the other side, providing a web service. Using Ruby on Rails I added a REST interface to my existing website by just providing XML views of the data. While working at a bank a SOAP interface to an existing legacy system was required, it took a great deal of fiddling and trust in our IBM development tools to get it working after a couple of days.

The experience reminds me of an interview on The Register where Tim O'Reilly suggests that SOAP is a standards first specification created without reference to how it would be used in the real world (because at the time the spec was created there weren't any people using it). SOAP definitely feels like an overengineered solution. The kind of thing I would have loved earlier in my career when suffering from Second System Syndrome. Over time I am becoming increasingly appreciative of simplicity in code.

For me, there is now no comparison. SOAP just doesn't seem to work well in any place I've seen it tried. My first choice for web service API will have to be REST.

This week I met up with some old colleagues. While talking about what we were all doing now, one person who still worked at the place we worked together mentioned that there had been a change in software architecture. Thus all my old code would be rewritten. This news got me thinking. I've been writing software commercially for nearly 12 years and most of my code is no longer used. Three years of work is gone due to my employers going bust. Another 3 and a half years worth was (or will be) rewritten after changes in platform architecture. Two more years of work was retired after completing the purpose for which it was written. One employer from years ago still uses 2 years of coding work - just last month I crossed paths with one of the users who did acceptance testing (he remembered me after 5 years, although neither of us could remember the other's name) and he said they were still very happy with it. So about 70% of my work, measured by time, is gone and no longer of any use. Is that normal?

I'm not sure what is the appropriate comparison. Engineers and architects build structures that last for many years. I have known people who assisted building houses, power stations, and gas pumping stations. All of which still stand and should for decades to come. However, these are physical things, so are obviously physically harder to construct. Creating software is a thought driven logical process. Perhaps a better comparison is with other professionals dealing with abstract concepts like accountants, lawyers, analysts, etc. Much of their output is transactional in nature: calculating this year's corporate accounts; writing or checking a contract; or, producing a report. While such work may exist for some time, its utility drops greatly once the reason for its creation has passed. I suppose this is analogous to code still existing in sourcecode repositories long after it no longer runs. In which case still having 30% of my work useful is probably a good record.

However, I don't find the comparison to other abstract workers wholly compelling. Software developers seem to fall in between them and engineers. We think abstractly for a living, but produce something, even if it is just logic, that has utility beyond the point of its creation. Some computer programs have been running for decades - otherwise there couldn't have been such an issue around Y2K. Even in the faster moving consumer software world code can hang around. Just today I used a DOS command window - probably some very old code hanging around there.

Perhaps the best way to think about my profession is that it is a discipline of engineering which works with logic and thus allows the perception that its artefacts are easily replaced (even if in reality that often is not the case). Also it is a fairly new profession and as such it is developing and changing rapidly, which also encourages code to be replaced with updated, more modern, versions. As for my record, when I mentioned the figures to my old colleagues they didn't think it was strange or exceptional in any way. That is probably just modern commercial software development.