December 24, 2011

Bonds: Part 3a - The Technology

Tags: ,

Over my time working for EGB desks I was always a Software Developer, and software is the part of the industry I know best. Code written in banks is not especially different to other places, although there is a difference in attitude. This starts before a person even joins a bank.

In London at least, banks pay more than other industries for IT people. A senior developer in an investment bank can easily earn double or triple the pay of their colleagues outside finance. There is no significant local startup scene or culture to draw large numbers of students away. So the best students gravitate towards the highest salaries available. I’m told that the prospect of 3 months “training” (from the stories I’ve heard that definitely needs quotes) at the head office of New York based banks is also a strong draw. There can sometimes be a sense of arrogance that people from outside banking couldn’t handle banking IT. That to be successful, you must start in a bank’s graduate program. I will say that in my experience the general standard of developers inside banks is higher than in other industries. However, it is not high enough that good people from outside wouldn’t thrive, assuming they could handle the culture. I have certainly encountered a few well-paid duds in banks, and successful outsiders (including myself).

I started working in banks seven years after leaving university. Thus I have only second hand knowledge of the graduate program. However, I do have very good knowledge of the interview process, from both sides. Due to the high pay available, a lot of people from outside banking apply for banking IT jobs. Normally they don’t get interviewed. I was one of them. Over 3 years I applied for tens of banking jobs and only got interviews for 3. I only ever apply for jobs for which I’m qualified, and outside banking would be interviewed around 40% of the time. One investment bank offered me a job and in retrospect I think that luck and good timing were the largest components of my interview success.

Now I have worked in banks I see the greatest impediment to outsiders is the lack of “banking experience”. Often job descriptions will have this phrase in the list of requirements. Normally I don’t think it is vital, but nontechnical people in particular (HR and agencies) find it a handy shortcut when filtering large numbers of resumes. It is assumed that people with banking experience will among the best (since another bank has already vetted them), understand the business, be able to handle the “get it done now” culture, and have the required mathematical/IT skills. Interviews tend to check the same things.

All of these capabilities can be learnt on the job by a reasonably smart, experienced software developer. Largely because most of them are not a big deal – writing code and fixing bugs is still by far the most important skill a banking engineer requires. Not all the best graduates go into banking straight out of university (indeed not all the best coders even go to university, although I think it does help). The business knowledge is very specific to a particular area – most of my cash bonds knowledge would not transfer easily to derivatives, equities or foreign exchange. A general finance background can help, but most of the time it doesn’t matter. The little knowledge required to get the job done will be passed on as needed. Most of what I have learnt was picked up from general interest about how our desks worked. However, I don’t think I have ever been to an interview for a fixed income job that didn’t ask what a bond’s yield would do if the price dropped. Most will ask more – “how are bonds priced”, “how do you bootstrap a yield curve”, and “how is risk calculated” are all common. I would guess they are checking that I have picked up some knowledge while working on EGB desks, but these questions were asked even before I had that history.

Maths questions are also common at interviews. I don’t think the maths I have used in my job has ever gone far beyond the high school level. Confidence in your mathemathical skills is definitely a requisite (especially when a trader is testing you), but basic algebra is good enough for 90% of the job. The other 10% is numerical analysis, linear algebra and being able to understand calculus. Calculus equations are common outputs of the quants’ work, and I can certainly see the requirement for them to have exemplary maths skills. Although as a developer, the hardest it gets is understanding a quant’s work. Not as bad as it sounds as most of the time there is a reference implementation in Excel to follow.

Interviews also contain numerous programming questions and usually a programming test. I have seen these range from silly easy questions to the crazily hard. Although over time I have seen a pattern emerge – the same type of questions come up over and over again. Again I’m not sure they are relevant to the work we do. If you have a banking interview for a Java position, brush up on pre-1.5 multi-threading (despite everyone using the new java.util.concurrent classes nowadays – but the tests were written before they were added), garbage collection and collections. I remind myself of these things before interviews and they never fail to come up. Writing or explaining a blocking queue with wait/notify is a particular favourite. Also writing code on paper is very common, but the marking criteria can be obscure – try to discuss your answers with the interviewer if possible to explain yourself. One colleague marked a coding question wrong due to inefficiency because an applicant used String.reverse() rather than writing a bespoke reverse method – who ever does that on the job! Another non-banking applicant I liked got rejected because their amateur neural network home project was not up to the standard of one of the other interviewers (who did his Masters on the topic). I was impressed he had a home project. Sometimes I think the implicit belief that banking people are the best makes interviewers unnecessary dubious and harsh on non-banking people.

The biggest real problem a non-banking programmer will have moving into banking (as opposed to the many imagined problems) is the culture. Although this is usually only evident after starting work. Developers have to produce results quickly and accurately, usually on their own. The most important thing is to get a steady stream of functionality and bug fixes out for the traders. Management’s attitude is generally “you should know what you are doing, so just do it”. There is very little structure except what you create for yourself. Front office developers need to do a good job and know the best place to cut corners with minimal affect on code quality. The codebase soon becomes a path-dependent mess, so developers also need to be good at quickly diagnosing bugs and short-term hack fixes (if you’re lucky there will also be a long-term proper fix later).

If a problem occurs there is an expectation you will be available to fix it. People are regularly called on weekends, late at night or early mornings. Once a person was contacted to fix a bug in their code while they were on an overseas holiday. It was considered impressive he managed to fix it while still away. Early morning starts were very common. There was usually a rota for 7am starts to backup the support rota. Although if you released code to production it was expected you would be in early the next day just in case there was a problem. There wasn’t a proper hours culture – you could leave early or work from home if required with no repercussion, but it was normal to stay until the job was done. Once I stayed at work until 11pm to finish and release a system, then was back at my desk for 7am to support it (although that is nothing compared to some of the bankers). Another time I stayed back 4 hours just in case a person with a production bug needed help. These are not uncommon, at every bank where I worked most of my team members will have similar stories. For me it was a matter of loyalty to the team (I used to get very annoyed if I felt managers were taking advantage of this). Still I count myself extremely lucky I never had a Blackberry.

Every EGB team I was in had around 10 developers. There were normally about as many concurrent tasks requiring quick completion as there were team members. About 80% of the time people worked by themselves on a particular system. Occasionally a team of two or three developers was used on larger systems. There was always more to be done. Thus most of the time you were responsible for your delivery and had to be individually productive at cutting code. Generally there was no place to hide if you weren’t writing code. Similarly if you were writing buggy code, you would be spending time fixing bugs and not moving forward. Still at different banks there were differences in expected tempo. The workrate management expected of us in an American bank was greater than when I was at a European bank. Talking to coworkers suggested this wasn’t an aberration. Goldmans reputably requires the greatest effort from its developers.

The level of software development process followed depends largely on the team themselves. If they want it, but maintain the existing workrate, they can do it. However, more process tends to slow people down, but too little and an unmaintainable mess is quickly produced. People hung up on process didn’t do well, but neither did people with no process. The result is a sort of half agile process. Everyone iterates (usually 2-3 week iterations) and uses a continuous build server. No one pair programs. Unit tests are left up to the developer and coverage varies wildly. Most teams I have seen have tried daily standup meetings, but they don’t survive long. All the teams had testers, but they were always overloaded by the volume of work and most didn’t have a good understanding of the business area. Much of the time systems or fixes were deployed without being tested by anyone other than the developer who wrote it. Knowing that a bug could cost a great deal of money (even being out of the market has a large opportunity cost) promotes paying attention to little details. Thinking through as many failure states and edge cases is very necessary – if you don’t catch a potential problem in a system, it will probably be a trader who finds it (at the worst possible time). Often this means you don’t make the same mistake twice. For instance, the need to support bonds quoted by yield, as well as the more usual price quoted bonds is now burned into my memory.

Over many years of work, I have only once seen a written specification. Often the specification is just a quick description from a trader – if you’re lucky. Sometimes it is passed through a few levels of translation first. If elaboration is needed then often you will get a dismissive “just make it work and don’t lose me money”. IT people are generally considered as second class citizens by bankers and traders (although there are always exceptions). Once when asked how a system should look, a trader replied “make it look like Lehmans’ system”. This was years since Lehmans went bust and no one around had even worked there – we just did what seemed right. Another system I delivered was complained about by the traders. After listening to them I produced a bunch of fixes the next day. They complained again and sat me down for 10 minutes to explain what I had done wrong. The next day I reverted it back the the original system delivered two days previous. They said that version was ok and I should done that the first time – no joke, that really happened to me! It is a benefit to have your clients (the traders) so close and quick to specify when there is an issue. However, they are vociferous in their annoyance if something is not right and quiet in their compliments. If we heard nothing about a system we assumed it must be working well.

When something is wrong with their IT systems, traders will either complain to their support people or their front office developers. This is regardless of the source of the problem. As the team facing the traders, it is always us who have to diagnose the problem. On the many occasions the source of the problem is downstream from our systems, we liaise with the people responsible to fix it (a task that generally falls to whoever is around or was originally contacted). Other IT teams are normally helpful, although often don’t have the same sense of urgency as front office teams.

The work environment is always noisy. The closer one sits to the traders the higher the volume. People listening to music on headphones is very common. Interruptions are common. Support issues can be raised at any time and it expected that everyone will regularly check the support message channels. It is essentially impossible to concentrate for long stretches. Developers can’t be precious about getting into a “flow” state. I used to say “I am always interruptible” – might as well embrace what will occur anyway. Of course this affects the speed and quality of coding.

Most of the development work at banks is not much different to non-banks. Moving data around, transforming it and displaying it to users is the vast majority of the work. Quants may do more research or math based work, but not the “normal” developers. Adding extra functionality to existing systems is most common, followed by bugfixes (tactical or strategic). Least common is writing new systems, but that is what most coders want to do. There is some competition and lobbying when it comes to writing to new systems. A bit rewrite typically comes along every few years as technical debt increases and circumstances change (for instance if other banks are beginning to take advantage of a slowing pricing system in relative terms). Developers have a great deal of latitude in the design and implementation of new systems. It can be very rewarding. Many say that writing new systems (and the high pay) are the only reasons to work in banking IT. I don’t think that is far from the truth.

comments powered by Disqus