Brendel Consulting

Telecommuting for software developers

2009-09-15T10:26:00.010+12:00

I telecommute a lot, and it's a pretty extreme case of telecommuting as well: I'm based in New Zealand, while my clients usually are in North America, Europe or Asia. Occasionally I meet some of them in person, but often I never get a chance to shake their hands. In this article here I want to talk a little about how I tend to make telecommuting work for me and my clients.

Telecommuting and modern development processes: A conflict?

While telecommuting has become much more acceptable in the last few years, many prospective clients or employers still shy away from it. Their concerns are understandable: How can someone so far away be successfully integrated into a team? How can that person really participate in the development process, brain-storming or design sessions, code reviews or high-pressure debugging sessions?

In fact, there are two opposing trends at work here: On one hand we have more and better technology available for telecommuting. On the other hand, however, the more we talk about iterative and agile processes in software development, the more it is felt that in-office presence is required, since these processes rely on low-latency, high-bandwidth and frequent communication.

The question then for every telecommuter and their employer/client is this: How can we best adapt the way we work and utilize available technology to make telecommuting work even with the small-team, interactive and iterative, communication-heavy approach that is often preferred today?

Making telecommuting work requires some preparation and a little bit of flexibility from both sides. Undoubtedly, there are a lot of nuances, which you will miss out on, no matter how good the tele-presence technology at your disposal might be. I'm not saying that you can make telecommuting 100% equivalent to on-site presence. However, I do believe that telecommuting can be made to work very efficiently, and usually much more so than most people would guess.

In fact, I have telecommuted not only as a developer, but even as the engineering manager of teams, doing everything from architecture and design to the daily ins and outs of project management. It can work, and it can work well if both sides are willing to work together.

Here now a list of my favourite tools, methods and infrastructure components, which help a telecommuting software developer to integrate and work well with a team.

Almost like being there in person

Let me address some of the key concerns I hear a lot. For each of these concerns I try to suggest a solution.

Concern: "We have testing/development/documentation resources in our network. A telecommuter doesn't have all of this available and thus cannot be effective."

Access to important development resources, such as build-servers, test-databases, in-house wikis and other services can easily be provided through a VPN (Virtual Private Network). Good open-source VPN software is available, so aside from the initial setup work, cost should not be a concern. OpenVPN is popular and is available in the standard repos for most Linux distributions. It can be downloaded for Windows, or in its source form here.

In many cases, shared resourced like this can also be hosted 'in the cloud', or simply as a cheap VPS. Properly secured with SSH keys and logins for each developer it will do just fine. Wikis, test suites, databases, and so on can be hosted there.

Very important, of course, is the use of a version control system. There are some VCSs now specifically designed for distributed teams, for example Git. But I have also worked successfully for clients that used SVN, even though our team was distributed. If you are not dealing with projects the size of the Linux kernel and also have a relatively small team, you won't have to convert your project to a new VCS, the old one most likely will do just fine.

Concern: "In an office, people can just pop their head over the cubicle wall to ask a quick question. With telecommuting there is a lot of communication overhead and long turn-around time."

Fortunately, that's why we have instant messaging. A plethora of IM clients is available for free and most everyone these days has an account with at least one of them somewhere. Quickly typing a question is just as effective as 'popping their head over the cubicle wall', and possibly less distracting.

In addition there is always the phone and VoIP (see next paragraph for more on that).
Concern: "With the team in the same office they all see each other on a daily basis. Because of that they have a more personal, closer relationship and form a better team."

I can't say that presence in the same office always makes for good relationships, but I understand the concern. Luckily, for a few years already, we have a service available to us which solves this problem, again for free. Of course, I'm talking about Skype, which offers voice, chat and video conferencing. Personally, I would much prefer a capable open-source solution, but time and again Skype's painless setup and ease-of-use has proven just too simple and effective to argue with.

Whichever route you go: Seeing each others' faces and hearing their voices on a daily basis is not a problem anymore these days. The services are free, run on Windows, Linux and Macs and web-cams are cheap.
Concern: "We often have brain-storming sessions or our daily scrum standup meeting in the conference room..."

The easiest solution is to place a laptop with a web-cam in the conference room. Even something cheap like an Asus EEE PC will do. No, I'm serious! I have worked in teams where we did exactly that. We had a laptop in the conference room and a web-cam. The laptop was useful for occasional presentations anyway, and I was able to 'be there' during discussions and meetings.
Concern: "How can a code-review meeting be conducted? Don't tell me you can read the code that's projected on the wall when 'being present' via a web-cam from the other side of the room..."

There are some really nice tools available for code-reviews in distributed teams. For example CodeStriker, which is simple but very effective. It's open source, so again it's free. Sure, that's not a meeting where everyone piles into the same room, but having experienced both types of code review, I can attest to the fact that a code review via CodeStriker is much more pleasant than a sleep-inducing code review meeting with everyone in the same room.
Concern: "We do a lot of pair-programming / pair-debug-sessions. That's just not possible in a distributed team."

Of course it is! There might even be specialized tools available for this. However, the simplest one I have found is straight-forward screen sharing. VNC is the usual, free and open source standard. But there are also commercial screen-sharing services, such as GoToMeeting.com. You start a Skype session or phone call (voice is sufficient here) with each other and then one of the pair shares their screen with the other one. You can even set up VNC so that you can control the screen remotely, allowing you to 'grab the mouse' and point at something you are talking about.

A number of free and/or open-source VNC clients and servers are available or are already in the standard repos for most Linux distributions. In Ubuntu, for example, screen-sharing comes pre-installed already. You just chose 'Remote Desktop' from your preferences, or 'Remote Desktop Viewer' from your Internet applications. That's how simple it is these days.

Obviously, this is made easier when a VPN has been setup, since otherwise you need to figure out how to get the necessary network traffic through your firewall(s).

Even simpler than VNC is just a plain old 'screen' session in a terminal. That works great on any *nix server to which one or more team member can login to. The big advantage of a shared 'screen' session is that it has much lower bandwidth requirements than VNC and thus works better for any shared work that only requires textual information.
Concern: "We need to be able to collaborate in real time on notes/documents/drawings. In short: We need a whiteboard!"

There are several solutions I am using for this. One very effective and simple one is EtherPad, which is a straight-forward way for multiple people to simultaneously collaborate on the same document. You actually see the individual letters appear as the other person types, contributions from different people are color coded and you have a time-slider at the top to re-visit the evolution of the document. You can also export the documents as PDF or Word file.

For diagrams and drawings, Creately offers similar features. Just like EtherPad, you can get started for free. Of course, there are a number of similar offerings out there, EtherPad and Creately are not the only one.

In fact, remember what I said about pair-programming, using VNC? You can take the same approach for diagramming and sketches, allowing you to simultaneously work on the same drawing application (Visio or the drawing functionality in most presentation of word-processing packages).
Concern: "We use Post-It notes for our Scrum board..."

Online Scrum boards are offered by more than one provider. Scrumy is one of those services, giving you the 'look and feel' of Post-It notes. Many people still like to represent their Scrum board in spreadsheets or Wikis, though.
Concern: "All these technical solutions are fragile or take too long to set up when you need them."

The best advise I can give here is to prepare ahead of time. When you start your telecommuting workday, make sure nothing has 'fallen over' and you can still connect via the VPN. If you have a fickle sound system on your desktop then make a Skype test call (for example) to confirm that it all still works. Make sure to have a how-to on the company wiki that explains the use of VNC or whichever other screen-sharing service you use and occasionally go with your colleagues through a test session at least. Always send any visuals or supporting documentation out to all meeting attendees sufficiently ahead of the actual meeting.
Concern: "You are in a completely different time-zone, so even with all those communication tools, you still won't be able to interact with the team."

Addressing this concern requires a bit of flexibility by the telecommuting worker. Most work environments, especially for software development, won't be strictly 9 to 5. Instead, they may have 'core hours' (10 am to 3 pm, for example). It helps when you as the telecommuter can offer the client/employer that you will at least be at your computer during those core hours they have in their office, no matter which time-zone you are in.

Case in point: When I work for clients on the US west-coast, I tend to get up around 4:30 am, which is about the time the team there gets into the office in the morning. I also arranged my work week to go from Tuesdays through Saturdays, in order to compensate for the one day that New Zealand is ahead of the US.

As a telecommuter we ask for some flexibility from the client, therefore we can offer this much flexibility in return.
Concern: "When a team is housed in the same office you get valuable informal interactions, watercooler talk, which you just can't get when telecommuting.

Indeed, this is one of the toughest things to deal with. Short of roaming the corridors with one of these (a bit pricey), or these (somewhat cheaper), the best solution appears to be to (a) take full advantage of all the other means of communication, (b) encourage the team to involve the telecommuting worker(s) when the discussion turns to work-related issues and (c) to install a thin laptop/desktop in the kitchen or other common gathering area as well, start Skype and set it to auto-answer, which allows the telecommuter to 'visit' that area whenever they want to.

The cheapest and probably most effective solution, however, is to continually keep the team as well as the telecommuting worker in mind. It is easy to 'forget' about the other side at times. Staying in contact can take a little bit of effort if you are engrossed in a programming task, for example. Occasionally, it is good to make a point out of calling someone else, rather than 'just' sending an email or IM, efficient as those solutions may be.

So much for my thoughts on this. Hopefully this list can help you or your clients to consider telecommuting as a valid option. If you have any additional ideas or suggestions how to enable a telecommuting software developer, please let me know in the comments.

Oh, and by the way, you should follow me on twitter here.

Search engine comparison for software developers

2009-08-07T08:27:00.004+12:00

For today's software developer it is difficult to imagine that there once was a time where we did not have instant access to information on the Internet to help us do our work: Answers to questions, howto-guides, reference material, documentation... whatever we need, it is usually just a quick search away.

I can't count how many times I simply searched for something like "Django ... such and such ..." to instantly find the relevant documentation or reports of other people's experience while working on a Django project, for example. The amount of time this saved me is immeasurable. I have a friend who - as a software developer - is not allowed to access the Internet while at work! The mind boggles. How much does this cost his employer in lost productivity?

Anyway, clearly the access to search engines while developing software should be considered essential. If during a job interview I'd ask a candidate something like: "How do you define a C function pointer?" and the answer would be: "I've done it before, but can't recall the exact syntax now... I'd just google it!" ... well, I would consider that to be a perfectly valid answer, to be honest. One more reason why these types of questions are quite meaningless in interviews. But that's a different topic.

So, the question then is: What is the best search engine for a software developer to use? Many people default to Google, obviously, but personally, I don't like to let one company have all the information about me, so I like to mix it up a little. The difference in search results can be quite startling, though. With Bing as the new arrival on the scene, let's perform a completely subjective and un-scientific experiment and ask four of the major search engines the same set of questions and see what they tell us.

For this test, I have chosen Bing, Google, Yahoo and Ask. Sadly Yahoo search will soon stop to exist when they start using Bing as a search provider, but for sentimental reasons and while it is still available, I thought I include them. Especially since Yahoo was my default search engine for quite some time now.

I have chosen four questions, which obviously cover only a tiny fraction of software development related areas:

"Django IntegerField model": Here I would like to see links to the Django project documentation on how to define an IntegerField in a model. It should be noted that 'IntegerField' also exists as part of forms, so the search engine should not confuse these two.
"Define C function pointer": Our 'interview question' from above. The top results should lead to pages that very clearly explain how to define this thing.
"TCP checksum calculation": I would like to see links to pages that explain how the algorithm works that computes TCP checksums.
".Net class inheritance": I'm not an MS developer at all, so I'm not even quite sure whether this question makes much sense, but I thought I try this one, just on a hunch.

Ok, let's start...

The test

Question 1. "Django IntegerField model"

Here are the results from all four search engines, side by side. In all images here, you can see Bing in the first column, followed by Google, then Yahoo and finally Ask in the last column.

Bing: The first results are essentially useless, leading (ironically) to Google code pages, followed by some obscure changeset in the Django code itself. Sure, you can see an IntegerField used there. However, the link to the Django docs, which I was looking for, is only at the very bottom of the page.
Google: The results are highly relevant (only peeve might be that they are pointing to docs for the SVN version, not the latest stable release version, but that's really minor). A link to the older Django version's docs is in third place, which is useful. But as we can see, the ads are not quite as relevant.
Yahoo: The results are not bad. However, I didn't like the fact that it pointed me to docs for a much older version of Django in the top result. They could have done better here.
Ask: The top results are good. However, I am disturbed by those ads, which appear right under the top result. They could have shown them on the side. But on Ask the side space is taken up by "Related Searches", which in this particular case were completely irrelevant.

Ranking: Google wins hands down. Ask in second place, closely followed by Yahoo. Bing is dead last.

Question 2. "Define C function pointer"

Here are the results for this question.

Bing: Again disappointing. The top result is about a function pointer validation function... which would be relevant if obviously I didn't need to learn first how to define a function pointer. But since I am looking only for how to define a pointer these results don't help me much. The third one is even about how to call C functions from Forth!?
Google: This is exactly what I was looking for: A tutorial! Top link, good job Google!
Yahoo: Not bad, with the tutorial in second place. For some reason, the Wikipedia entry comes first. Both Yahoo and Google generally give quite some weight to Wikipedia pages, which I am normally fine with. In this particular case, the tutorials are really what I'm looking for, though.
Ask: Again a very good top result, just like Google, marred by irrelevant annoying ads, followed by other good results (for example how to define a function-pointer parameter for a function, which is very nice).

Ranking: Ask wins this one based on the result quality, even though their ads bother me. Google is a very close second, Yahoo in third, but still close. Bing is way off the mark again.

Question 3. "TCP checksum calculation"

The results for this one, side by side.

Bing: Very disappointing results at the top, leading to small discussion threads and postings here and there. Not at all what I was looking for.
Google: Excellent results, especially the third link ("TCP Checksum code"). That page contains a textual description of the algorithm as well as actual sample code. Perfect. The further links on the page remain relevant as well. The PDF in fourh place is actually quite good.
Yahoo: The top three results are the same as Google, so that's very good. The relevancy of the remaining links trails off quickly, though.
Ask: The top three results are the same as Google and Yahoo. The remaining links remain more relevant than Yahoo, but not quite as good as Google, though.

Ranking: Google in first, followed by Ask and Yahoo. Bing again in last place.

Question 4. ".Net class inheritance"

Here is a question for Microsoft's Bing to shine! It should know all about that, right? Well, here are the results...

Bing: Oh the humanity! What were they thinking? Again, the top link goes to some forum page, the second one is a blog. That's a recurring and unsuccessful theme with Bing. The third one contains some sample code, but is basically someone making the case for multiple-inheritance, which apparently is missing in .Net.
Google: Hits the mark again with the top link pointing straight to the relevant page of Microsoft's own online documentation. Why couldn't Bing come up with this? And the second link is great, too: A tutorial for class inheritance, which is always nice to see.
Yahoo: Good results in the top two spots, with a useful page about interfaces included, which Google didn't show. But the third result is disappointing: Apparently, Yahoo was confused by the domain name for that result, which ended in ".net".
Ask: The ads are disturbing as usual, but the top results are the same as Google's, which is good. Not visible in this screenshot, because too much valuable space was used by those ads, there is also a good tutorial link further down, which Google didn't show.

Ranking: Ask by a slim margin (if we manage to ignore those ads), Google in close second, followed by Yahoo a bit further back. Bing is again at the very end with a smattering of useless results.

I sure hope that Microsoft's own developers are free to use Google or Ask at work.

Conclusions

Let's start at the back.

Maybe they are still trying to sort out issues and will improve over time. However, at the moment Bing is just completely hopeless when it comes to these kinds of queries. I guess they are much more consumer oriented. As it stands, though, for questions software developers need to ask for their work, Bing quite plainly ... sucks!

Yahoo still tends to be good and delivers useful results. But while the top results are usually ok, the relevance of further results quickly drops. Of course, that will all end once they switch to Bing. At that moment they will get much, much worse. What a sad loss for the Internet to see Yahoo search go.

Ask and Google are actually ranking equally well here in this test. That may come as a surprise to many. The biggest annoyance about Ask is the placement of the ads right after the top result. And their "Related Searches" links usually don't provide any value at all for these types of searches. But the quality of the links is about as good as Google. The ads on both Ask and Google are close to irrelevant in both cases.

A message to Ask: Please change the placement of your ads or at least distinguish them visually a bit more!

So, take your pick: Google or Ask. I will start using Ask, because as I said earlier, Google is on enough pages already by means of their ad network and they don't need to know everything about me. And with Yahoo ranking behind Ask and soon completely fading away, Ask remains the only real alternative for search results that are relevant to software developers.

You should follow me on twitter here.

Can Agile work for a team of experts?

2009-07-29T06:26:00.008+12:00

My last article about considering Agile produced a lot of interesting feedback. Most of all, it prompted me to do a lot more research into Agile software development and specifically Scrum as a project management framework.

As has been mentioned before, many of these Agile ideas are just common sense. This is often pointed out by its critics, but is that really a bad thing? Since common sense is not as common as we would like it to be, Agile frameworks and methodologies codify their practices in rather specific rules on how to do things. Basically, these rules are designed to help (force?) a team to adopt a lot of common sense practices, which in traditional software engineering may have been forgotten. I have to say, I feel like I actually 'get' many of them and I might talk about some specifics here in future blog entries.

In the meantime, though, I'm still trying to wrap my head around one particular issue: Agile seems to work best - in fact it requires - that there are no 'silos' in the engineering organization: Particular knowledge should not be locked up in the head of one developer or a small group of them. Instead, knowledge should be shared across the team and some Agile practices, such as pair programming, for example, are specifically designed to address this. But in same cases those silos are unavoidable. Read on for an example of that. And in the moment you have knowledge silos in your team, many of the benefits of Agile seem to just evaporate.

Let me explain what I mean. For truly Agile development it must be possible to assign user stories and their related tasks to almost any team member. At least, there should be more than one team member who's able to take on a specific task.

If you have a team like this it allows a number of interesting things to happen:

Once a developer is done with a particular task they can merely grab the next highest priority item from the backlog. This makes it possible for the team to really deliver higher-priority items in order, thus producing most value to the product owner in each sprint.
You can use story point estimating, maybe utilizing surprising techniques such as planning poker, and it will actually make sense: Over the course of several sprints you get a feeling for the velocity of the team and even each individual developer. The nice thing about estimating in story points is that it is self-tuning and doesn't depend on the experience level of the individual developer. After a while you will know that John - a senior developer - can do about 20 story points per sprint, while Bob - fresh from university - may be able to do only 10. So, you can re-assign user stories without having to re-estimate them - well, you need to re-esitmate the associated tasks, but that is easier with these metrics - and still quickly get a good idea of how your team will be able to do in the sprint.
Every developer can just jump in and help others in the team. That is of course a given and is helpful in every kind of team, Agile or not. If a staff member goes on vacation or is sick the work can be picked up by whoever is available (see also point 1). Likewise, if a developer is held up with a previous task for longer than expected then the next task that he was supposed to tackle doesn't have to wait. If someone else is free or doesn't have a higher priority item to work on, that is.

Imagine you are in a team that develops web-applications, using Django or RoR or some such framework. There is a large number of user stories, all of them describing the various features that need to be implemented. All of your developers will have to work in pretty much the same environment and under the same constraints provided by the framework. In a situation like this, I can imagine Agile to work very well, since all developers essentially need to have the same abilities for the job anyway: Know the language, know the framework, know the infrastructure you use for developing and testing. That covers a lot of ground and establishes large areas of commonalities between developers. Everyone is much more likely to be able to take on any specific task during the sprint.

But now imagine a different team in a different company and suddenly we encounter trouble: Assume you are in a small startup, which doesn't just devlop a web-app, or work with a single framework or environment. Assume you are working on a project that hacks the Linux kernel for some innovative network monitoring solution, produces a lot of data that needs to be sliced and diced and requires a web-app to drive an interactive front end. You are a small startup with some angel investment, so you get yourself the smallest team you need to get the job done: An experienced networking/kernel hacker, a database expert, someone who knows Django or TurboGears (or a similar web-app framework), and a user-interface person taking care of the HTML/JS/CSS for the front-end.

So, that's your team. Now imagine a planning poker session for a specific feature: "Ok, on the count of three: What's your estimate for hacking the ipchains kernel module to add feature XYZ?" Guess who is going to put down a card with a number and who is going to put down a card with a question mark (meaning: "Don't know...")? Well, the kernel person may have a decent estimate, while everyone else - those who never hacked around in the kernel before - realistically will have no idea how complex that task is! You have just one person on the team who can do the job and can provide estimates. You may just as well estimate in hours then, and you will be back at much more traditional planning. You need to define exactly ahead of time who will do which tasks, since each person is so specialized that in effect they are the only ones being able to estimate and do it. Say hello to your Gantt charts.

I think this is an example where some of the best aspects of Agile seem to fall flat.

Maybe you will say: Well, you shouldn't have teams like this anyway, you should pair-program to spread the knowledge, you should assemble your team so that you don't have this formation of knowledge silos.

To those arguments I will have to say: VCs don't have a lot of patience and they don't hand you a lot of money to start with. You can only get a small team, and you can't turn an HTML/CSS/JS person into a Linux kernel expert just like that. Especially not when you deal with a very tight deadline and budget. You can try, but then who's doing the front end work, while you are sending that person through the Linux kernel 'apprenticeship'? No, you need subject matter experts that are as efficient as they can be and can crank out a piece of functionality with the least amount of fuss and delay. And all you can afford at that stage is one expert for each subject area.

If you have a team of experts, each expertly knowledgeabe in their own area with very little overlap: Does Agile still make sense?

As I said at the beginning, I can see a lot of value in many of the Agile practices and ideas. Yes, it's basically common sense, but I understand how by codifying it you can bring a lot of this common sense back into engineering organizations. But while I have learned to appreciate a lot about Agile since my [previous post], I have to stand by the assertion I made then: Agile works in some teams and environments, but there are also some cases where it simply makes much less sense. Some of its aspects might still be useful, but under some circumstances it loses a lot of value to the point where you have to ask yourself: Why bother with Agile at all then?

Agile seems to make sense only once your team can grow in size - or alternatively in the spread of knowledge across the team - to a point where most tasks could be effectively tackled by more than just one person on your team. If your work team consists of generalists, and your user stories and tasks are suitable to be tackled by those generalists. Note, with 'generalist' here I mean someone who is able to deal with all the aspects of your project. I'm not advocating that developers shouldn't have speciality areas.

This is just my impression. If you have experience working in Agile teams that just consist of a small group of specific and distinct subject area experts, please comment and let me know how you handled this problem. I would love to know.

In the meantime, I can imagine that non-homogeneous teams, with too many knowledge silos, may be the reason for many failed Agile projects: The team thinks it's agile, but when push comes to shove, they realize that actually they have been hand-cuffed by the insufficiently spread knowledge in the team. After a few sprints the situation may improve, but for starters?

Agile is about common sense, and there seems to be a very important common sense item that needs to be considered: Does Agile even make sense for my team and project? The answer is not always yes, no matter what Agile-enthusiasts may say. In reality (clue common sense) the answer shouldn't come as a surprise: It depends!

You should follow me on twitter here.

An entire web site in a single page

2009-07-16T09:21:00.005+12:00

Update: I received nice feedback from some readers, explaining
why this one-page design is not very good for SEO. Thank you
very much! I appreciate your feedback. As a consequence of this,
I have now moved my site to a traditional multi-page design.
If you are still interested in seeing the original single-page
site, you can go here.

When I first put up my web site, it was all just a single, static page. Basically a small blurb about me, the services I offer, and so on. It looked kind of neat, with a very simple design, which I like. And with a small enough font and big enough screen, it really fit all into your browser without scrolling.

But then a friend told me that it doesn't really look professional enough with just a single page. People expect to see at least several pages on a 'real' web-site. So, something had to be done.

Now, I'm not a professional web-designer who can photoshop together wonderful logos and backgrounds and also work a site based on the best user/computer interaction strategies. I mostly just wanted to retain the very simple and straight forward design of my site. And it should load fast. And it should be friendly to search engines. And it should still look good in text based browsers, or browsers that don't have JavaScript enabled.

When you look at my site now, you see that it does use multiple pages. But then... not really. Your browser will actually just load a single HTML page, which contains the entire text of the site. The navigation links merely trigger some changes in the CSS for various text sections, so that different content is displayed when those links are clicked.

Using the 'display' style property for hiding elements on a page is nothing new and is used on many sites. It's just that normally you see it used to show or hide specific elements within a page, while site navigation itself is normally left to the real thing: Actual separate pages.

So, there are probably other sites which do it this way, but it was the first time for me that I tried this approach with a site, and I like how it turned out. That's why I'm sharing this.

If you look at the site without JavaScript enabled, you see all the content in the page, one 'page' after the other. The navigation links on the left still work. I'm thinking I might add a link back to the top enclosed in <noscript> under each section at some point.

One of the most visible benefits is that site navigation is so fast, it feels instantaneous. Nothing needs to be loaded when a navigation link is clicked. But that is not the only advantage...

If you look at the page in a text-based browser such as Lynx, you see that all the relevant information is there, easily readable on a single page.

Best of all, the 'crap-to-content' ratio for the page is very good. I you look at the HTML source, you see that there is a lot of relevant textual content, and very little markup to distract from it. I hope Google gives me brownie points for that one...

In fact, I would be very interested to hear what those of you with more SEO experience have to say about the site. Does this approach actually help with search engines, or would it make it worse?

Ok, so there is one issue I can see with this approach: Tracking of visitors is made a bit more complex. Maybe I can use a small bit of JavaScript to re-load the image of the tracker that I use when a user clicks one of the navigation links? Again, if you have some feedback there, I would really appreciate it.

You should follow me on twitter here.

Considering Agile, deciding against it

2009-07-11T16:38:00.016+12:00

Some time ago, while managing a distributed software project, I spoke to a developer who was thinking about joining our team. This individual was a big proponent of Agile development. When he found out that in our particular project we were not using an Agile development model, he began to lose interest. He compared my position there to the position of the Tom Smykowski character from the movie Office Space. When I recently then came across this blog posting by Shannon JJ Behrens, I had to remember that particular incident...

In the movie Office Space, Tom is an older employee at Initech, whose job it is to “take requirement documents from the customers and bring them to the developers”. Not surprisingly, he is being characterized as 'useless' and let go. Similarly, in Agile development, direct communication between customers and developers is seen as the way to ensure that the deliverables align with customer requirements. No middle man is needed. Instead of extensive up-front design, requirements are captured in short and concise user stories.

I have to admit, I was somewhat taken aback by the comparison to the Smykowski caricature from the movie, but this episode certainly made me think about why we had chosen a non-Agile style for our project.

Let's try to answer this. Take a look at the graphic here (I don't remember where I saw it first, but it can be found on many different sites, click to see the larger version):

What do we not see anywhere? Design and architecture. The definition of the 'backlog' and the 'release plan' is probably the closest we get, but even that is feature focused. And in Agile, features always seem to be driven by customers.

For me it boils down to this: I believe that Agile is applicable in projects where not only a clear vision of the project has been communicated to the developers, but also where certain conventions and structures have already been established and where we are dealing mostly with explicit requirements, rather than implicit ones. Projects, where the 'how' is already known, and we are just dealing with the 'what'.

For example, to achieve a consistent UI you require some up-front planning, possibly a UI czar: One person - or at most a small number of people - who establish and downright dictate what the UI shall look and feel like. It may stifle the creativity of the developer, but it will ensure a much more consistent look for the UI. Imagine you have user stories describing the requirements for various dialogs in the application's interface. Different developers take on the implementation of those user stories. Unless there is some established, over-arching strategy and - dare I say it? - upfront design of the UI then the results are likely to appear disjointed or piecemeal.

Another example is error handling in the application. I remember one of my software engineering instructors a long time ago, who insisted that a project should have an error czar: Someone who establishes the standard by which errors should be handled within the code and who then diligently ensures that all new code complies with this standard. If this is not done and established ahead of time, how are the different developers supposed to produce consistent code that knows how to adhere to those standards and handles errors in a unified manner?

It appears to me that if you are developing an application for an existing framework then many of these up-front 'infrastructure' decisions have been made for you already. Consequently, you can focus more readily on explicit customer requirements and Agile is a much more worthwhile development strategy.

For example, if you develop a web-app in a framework such as Django or Rails then there are specific ways in which your database is going to be laid out. You also have specific conventions for how errors are handled, and so on and so forth. Thus, the design and architecture of much of your application is pre-determined by your choice of framework. You don't have to make those decisions any more. And later then, the more of the application has been written, the more such underlying decisions will have been made already. Agile works well in those situations. Since big-picture architectural decisions don't have to be made any more, we can focus on the addition of small features (or larger features broken down into many smaller ones), while all developers benefit from the established infrastructure of the project. The 'grab-bag' approach with focus on the individual user story becomes feasible.

However, if a vision for the project still has to be conjured up and the basic architecture has to be created and the UI look and feel needs to be designed and error handling or other standards have to be established... well, then Agile does not seem to be ideally suited for that. I have seen design and architecture by committee and design by chaos (everyone going off to implement things before anything was established). Neither one is a pretty sight. The best results are achieved if the architectural direction and basic design can be established by one or just a few, good individuals. Up-front. Well, somewhere after establishing the vision and collecting initial requirements, of course.

Realistically, I'm sure that most Agile projects will include some sort of up-front design and architecture, whether the participants call it that or not. Heck, even questions like: “Should we use Java or Python for this?” need to be answered ahead of time and are part of these implied up-front activities.

That particular project for which the developer from the beginning of this article interviewed was not at the stage yet where an Agile approach really would have worked. There were still too many unknowns at that time, which required up-front consideration. Thus, our project wasn't run as an Agile project. There is a time and place for everything, and our project was not at a time or place for Agile, yet.

However, projects can be transitioned from a more conventional approach to an Agile approach over time. The more we know, the more suitable Agile becomes. The less architecture and design decisions have to be made the better for Agile. But at the start of a brand new project, without anything established, some quite conventional up-front design and architecture is needed.

You should follow me on twitter here.

Google's Chrome OS is bad news for Linux

2009-07-09T08:06:00.003+12:00

Google's Chrome OS has been announced today and - like any announcement from Google - is already widely being discussed. People usually focus on the supposed threat that this 'new' OS represents to Microsoft's dominance on the desktop. The commentary focuses on this Google vs. Microsoft aspect of the story. However, I think the news is actually worse for Linux than it is for Microsoft.

How can I say this, considering that Chrome is be based on a Linux kernel? Let me explain, with a 'historical' analogy.

In the 80s and 90s we had the so-called Unix Wars: Prominent Unix vendors like Sun, Digital, IBM, AT&T and many others were battling for supremacy in the server market. Guess who emerged victorious from those Unix Wars?

Microsoft's Windows NT.

Customers were sufficiently fed-up with the fragmentation and confusion in the Unix server market and finally were receptive to a message of simplicity: Your desktops already run Microsoft software, so why not your servers? It will all play nicely together ... just deal with a single vendor ... yada yada.

Now fast forward 15 to 20 years. While Unix is holding on to various diminishing pockets in the server market and Windows server-products have a stronghold in many enterprises, Linux has emerged as a very viable and popular choice for the server. On the desktop, however, Microsoft rules supreme now as it did then.

Back in the Unix Wars, Microsoft used the fragmentation amongst the competitors to become a dominant force in the market. Today, it enjoys the advantage that the fragmentation of potential opposition in the desktop market is handing to it. Nowhere is the fragmentation of Linux more apparent than on the desktop.

Ubuntu managed to become a strong player in the Linux desktop market, and for once it appeared as if things were finally coalescing. Ubuntu's popularity started to feed itself: Any issues or problems? Chances are someone solved it for Ubuntu already, a quick search on the Internet reveals answers to most common issues. Therefore, while Ubuntu might not be perfect for everyone or might have a few annoyances, for the most part you just don't go wrong settling on Ubuntu: So many others are running it, momentum has been forming.

Now Google comes along with Chrome OS. We now have another major player entering the Linux desktop OS market. Linux needs this like we need another hole in the head. Microsoft will read this news with glee: Through the opponent's fragmentation it managed to win the Unix wars. Now Linux's continued fragmentation will ensure that there isn't even going to be a war for the desktop.

So, I wish for Google Chome OS to fail. We don't need it. It doesn't help Linux. As suggested here, if Google wants an improved non-Microsoft desktop OS then please throw your considerable resources behind improvements in the existing Linux desktop distros. We will all be better off for it.

You should follow me on twitter here.

Setting the initial value for Django's ModelChoiceField

2009-07-04T12:09:00.008+12:00

Recently, I worked with a Django form that utilized the ModelChoiceField. That is a convenient field that normally is rendered as an HTML select tag, which usually appears as a drop-down menu on a web-page.

A ModelChoiceField is specified as:

    class MyForm(forms.Form):
        my_field = forms.ModelChoiceField(queryset = MyModel.objects.all())

As you can see, this field is used to easily specify a drop-down for all items in a table (or whatever the queryset specifies). The model is represented as the output of its __unicode__() function. If you evaluate the form after it was posted, the value for the field is going to be an instance for the actual model whose unicode representation was selected.

The problems for me started when I tried to set an initial value for the field: In an 'edit' form, I wanted all fields to reflect whatever had been saved for a particular model instance, of course. As I said above, if you evaluate the form after it has been submitted you get an actual model instance. Naturally, you would think that initial values would be set in a symmetric manner, by specifying a model instance:

    form = MyForm(initial = { 'my_field' : some_model_instance })

Sadly, this doesn't work. And try as I might, I couldn't find an answer to this on the Internet either (more on that in a moment). So, after looking at the Django code, it finally dawned on me that you need to specify the ID of the model instance as the initial value:

    form = MyForm(initial = { 'my_field' : some_model_instance.id })

That works now. It's a bit unfortunate that the retrieval value (a model instance) and the initial value (the ID of a model instance) are of a different type. It's an inconsistency in the Django API, I think. But in the end, I probably should have at least tried that one a bit sooner.

The surprising thing is that I couldn't find any discussion of this anywhere on the Internet. I should mention here that I am using Yahoo as my default search engine. Shortly after I finally found the solution, it occured to me to try Google. And wouldn't you know it? Right there, third hit from the top, I had the answer.

Why did Yahoo not give me this result? Well, the answer was discussed on Google Groups. Is Yahoo not indexing those? Or is Google not letting them index it?

Either way, that small issue between Yahoo and Google cost me a few hours of frustration. So, I'm posting the solution here on a non-Google page, so that Yahoo users may also find the answer to that problem in the future.

You should follow me on twitter here.

Pitfalls of lazy evaluation in Django's ORM

2009-06-26T08:29:00.010+12:00

It's well documented that query sets in Django's ORM are lazily evaluated. The database is not accessed until you actually reference query set results. For example:

    my_set = MyModel.objects.all() # No database access
    print my_set[0]                # DB accessed and model instance created

The thing one needs to be aware of here is that any access to the result array will always cause a new database access and object creation for model instances:

    my_set = MyModel.objects.all() # No database access
    print my_set[0]                # DB access and model instantiation...
    print my_set[0]                # ... and the same again!

It is not uncommon to retrieve a set of results via a query set and then perform some operations on them, for example in a loop. It is tempting to view query sets just as if they were arrays - lists of in-memory objects - and use them in a similar manner. In-memory arrays tend to be quite efficient, query sets not so much: If you have to run over such a loop several times, you will end up retrieving the same object from the database multiple times as well.

If you need to iterate over the result set more than once - or just access the same element a few times - it is therefore more efficient to make copies of your individual results. That way, the query and the creation of model instance objects is done only once:

    my_set = [ m for m in MyModel.objects.all() ] # DB accessed for all
    print my_set[0]                               # No DB access!

In effect, the list you created in the last code snippet becomes a cache for your query results. You could also have just assigned a single result-set element to a variable for the same effect.

Fortunately, when passing a query set to a template it appears as if this caching is taken care of automatically for us. Making copies of query set results therefore is important when more complex operations are necessary on result sets within the view function.

You should follow me on twitter here.

Now on Twitter as well

2009-06-26T06:50:00.002+12:00

So, I thought I try my hand in this thing that everyone has been talking about and got myself a Twitter account. You can now follow me there, if you are so inclined.

I'm still undecided on whether this is going to be enlightening or merely another efficient way of wasting time, but we will see.

Read-optimize your source code

2009-06-14T20:00:00.012+12:00

When you design a software system, database or data structures, you take into consideration its most common use case. For example, you organize your data differently when it is read frequently, but only rarely written to. You read optimize your data.

I am convinced that the same applies to source code. In almost all cases, source code will be read much more often than it is written. Who writes the source code? You. A function is developed over some limited time, and once it is done it changes rarely, unless being refactored or modified to accommodate some changed requirements.

But who has to read your source code?

Well, for starters, your colleagues who have to integrate with or use your code.
New hires who join your team and try to find their way around.
The maintenance (or continued engineering) programmers.
Those who come after you, or inherit your code as part of their responsibility
You.

Your code is read around the time it is written and integrated and possibly for many, many years after you have long forgotten about your code.

We can see that over the life-time of some piece of code, it is very likely to be read much more often then written or modified. Consequently, source code needs to be read optimized just like we might read optimize a data structure or database if called for.

What does it take to read optimize source code? Here are the key points:

Meaningful variable and function names.

The compiler doesn't care whether the function name is 3 characters or 30 characters long. However, a colleague reading the code will be very grateful if the function is called get_daily_rainfall_average(), rather than dra(). Who cares that it takes two seconds more to type it? Many modern IDEs are going to do that for you anyway.

Thoughtful source code documentation.

Explaining in one sentence what some code does and in possibly many more sentences why it does it and why it's needed. The why is often much more important then the how: Explain the rational behind your design or implementation decisions. Everyone can see that a variable is increased in a for-loop, that doesn't have to be documented. But why the for-loop is needed in the first place is much more interesting and illuminating to someone who is new to the code.
Comments like this should be there for each module, class, function and the more complex code blocks within a function. Keep the comments close to the code they refer to. If they are all in the function or module header, they are often forgotten when the internals of code are changed, 15 pages further down.

A legible and consistent coding style.

We can endlessly argue about which style of parenthesis is the right one, but what's more important is that you remain consistent throughout your project.
If you are new to a team, use the coding style they use, even if it's not your favourite one.
Unless absolutely needed for performance reasons, don't try to optimize your doubly-nested loop into a fancy single line statement, exploiting even the most esoteric features of the language. Instead, break it up into a more easily understandable set of spelled out loops.
In general, don't optimize your code until you know you have to! Not only does it waste time if it turns out that the code really isn't performance critical, it is often also much more difficult to read and maintain.
If you can do what you have to do with simple and often used language features then use those.
Use white-space to make code less 'dense' and increase legibility. Many developers have pretty large screens these days, white space doesn't cost money.
Align code. For example, if you have several variable assignments, align the '=' operators directly underneath each other. It's astonishing how much more legible that block of code becomes. If you have to line-break a long function call, indent the arguments of the second line to align with the start of the arguments in the first line. That's just a little example, but there are many cases like this in most programs, where a bit of thoughtful alignment can make a difference.

It is often astonishing what amount of resistance some developers put up against even those few, simple rules. I have heard arguments, such as:

"I don't want to write comments, because then I will have to scroll more to see the code."
"Writing comments in the code takes time."
Formatting the code nicely takes time, especially when I need to change a few things, in which case I then have to re-format the code.
"Code changes, and before you know it the comments are obsolete."

I have absolutely no patience for the first three arguments. There might be some extremely rare situations where an emergency fix needs to be rushed out and short-cuts need to be taken. But for the most part, those arguments are bogus. The only one even remotely credible is the last one about comments getting obsolete, but this can be addressed in a straight forward manner: Keep the comments close to the code, and do code reviews where readability and correctness of the comments are stressed as well.

In fact, I claim that if you don't take those rules to heart in your own source code then you are either unprofessional, lazy, not a team-player, or all of the above. If you as a software developer take pride in your professionalism and quality of your work then you have to consider that it is not only the achieved functionality for which you are being paid: The code you produce in almost all cases becomes property of your employer. Therefore, the code itself also becomes a product you deliver, and actually your most important product.

How useful and usable your code is for the team who has to work with it is what really determines its value in the long run.

You should follow me on twitter here.

A Django ORM bug when using PostgreSQL

2009-06-12T15:54:00.006+12:00

Normally, an ORM such as the one used in Django is expected to 'do the right thing' with the database. I should be able to set up my models, and use the ORM without having to give any further thought to what it does behind the scenes. Switching between different databases should be as simple as changing two or three lines in my settings.py file: Use PostgreSQL instead of sqlite, for example.

At least in theory. It can be quite jarring to come across situations where this is not the case. And that is exactly what happened to me today.

Imagine a simple Django model:

    >>> class Foo(models.Model):
    >>>     date = models.DateField()

Now assign a somewhat unusual date to a model instance:

    >>> f = Foo()
    >>> f.date = datetime.date(1,2,3)
    >>> f.date.year
    1
    >>> f.date.month
    2
    >>> f.date.day
    3

You can see that we are talking about a date a long time ago. Nevertheless, a valid date as we can confirm by querying the individual elements of the date. Now save the instance:

    >>> f.save()

If you are using sqlite and do a select * from foo; in your database shell, you will see the following as the output:

       1-02-03

If you load a Django object from that table, you will find everything in order and as expected.

However, now try the same with PostgreSQL as database. After doing the very same thing in Python, take a look at what's written in the database table:

    2003-01-02

Oops!

Now that shouldn't happen!

The interesting thing is that as soon as you specify a year that has at least 3 digits, PostgreSQL stores the correct date. Do f.date = datetime.date(101,2,3) and after you save the model in the database it will have stored what we expect to see:

    0101-02-03

Also interesting, if you specify a year with two digits and a value more than 12, you get an exception. For example f.date = datetime.date(99,2,3):

    Traceback (most recent call last):
       .....
    DataError: date/time field value out of range: "  99-02-03"
    HINT:  Perhaps you need a different "datestyle" setting.

I checked the DATESTYLE setting for my database, and indeed, it was ISO, which defines a date as YYYY-MM-DD. In theory, therefore, that should work. But obviously, something is not quite lining up.

To summarize, while it all works fine with sqlite, in PostgreSQL I get:

A wrong date stored for years between 1 and 12
An exception for years between 13 and 99
Correct behaviour for years 100 or up

So, in case you need to work with really small dates in your Django app, you might want to keep this in mind.

You should follow me on twitter here.

How to do "count" with "group by" in Django

2009-06-01T13:05:00.006+12:00

Faced with the 'age old' problem of having to do a group by query in Django, I finally came across a solution.In short, you use the lower-level query interface directly:

    q = MyModel.objects.all()
    q.query.group_by = ['field_name']

When you then iterate over 'q', you get the results grouped by the value of whatever was in 'field_name'. Great! So far, so good.

Well, a word of caution at this point: Using the low-level query API is not really something Django wants you to do, apparently. In fact, try this with sqlite and it works. Try it with PostgreSQL and it does not (all sorts of error messages). So, your mileage may vary, depending on which database you are using. Ok, let's assume that your database is fine with this...

The only challenge for me now was that I had to do a count for this.
If you form your query using the usual methods of Django's ORM, you will be disappointed. For example, this here will not work:

    q = MyModel.objects.all()
    q.query.group_by = ['field_name']
    q.count()

It appears as if this returns only the count of the first group, not a list of counts, as you would expect.

The solution is to wade a bit deeper into the low-level query API. We can instruct the query to add a count-column. In fact, this results in merely a single column being returned, just like COUNT(*). It goes like this:

    q = MyModel.objects.all()
    q.query.group_by = ['field_name']
    q.query.add_count_column()

Since this returns counts, rather than complete objects, we now need to get the individual group counts as a list of values. We add one more line:

    q.values_list('id')

The values_list() function gives you not instantiated objects but instead the tuples of values for each object. The tuples contain only the fields you specify by name in the call to values_list(). Except, we have manually added the count column with the add_count_column() function. That count column is always returned first. So, what you get as result is something like this:

    [ (3,1), (19,8), ... ]

The first value of each tuple is the count, the second value is the value of the 'id' field of the last occurrence of the grouped model in the table. If you specify something other than 'id', you get the same thing: First the count and then the value of that other field.

But that's not what we want, right? We want a list of counts. We could manually extract the first element in each tuple, but Django offers us a shortcut:

    q.values_list('id', flat=True)

Setting flat=True tells the values_list() function to just return the first element of each tuple in a plain list (not a list of tuples). And since the first element now is the count column, we finally get what we want: A list of the counts for each group.

Note that we could have specified any other fields in the call to values_list(), not just 'id'. Because we specified flat those fields are ignored. It seems, though, that at least one field needs to be specified here, even though it won't be considered as part of the output.

You should follow me on twitter here.

Retrieving full objects with custom SQL in Django

2009-05-30T11:57:00.010+12:00

While working with Django, I recently had to retrieve some model objects via custom SQL. Looking for examples on how to do this, I noticed that most tutorials describe how to retrieve specific values via SQL, but not entire model objects. The solution is actually very simple. But since I couldn't find it described anywhere, I thought I share it here.

The usual approach goes something like this:

   from django.db import connection, models
   class CustomManager(models.Manager):
       ...
       query = "SELECT id FROM mymodel WHERE ..."
       cursor = connection.cursor()
       cursor.execute(query)
       return [i[0] for i in cursor.fetchall()]

This then returns the IDs of the objects that were selected. If you want to return actual objects, you can modify the last line into this:

    return self.filter(id in [i[0] for i in cursor.fetchall()])

However, the problem with this approach is that we are executing two queries now: The custom SQL query, followed by the self.filter() query.

Here then is a very simple way to get complete objects, with just a single, custom SQL query:

   class CustomManager(models.Manager):
       ...
       query = "SELECT * FROM mymodel WHERE ..."
       cursor = connection.cursor()
       cursor.execute(query)
       return [MyModel(*i) for i in cursor.fetchall()]

Rather than making a list of IDs, which is then used to query for the actual objects, we make a list of the complete objects. We can do that, because we changed the custom SQL: Instead of selecting just the ID column, we are now selecting all columns (SELECT * ...).

With custom SQL queries we get a list for each row. Fortunately, the order of columns in each row reflect the order of arguments to the model's __init__() function. As a result, we can use each row's list representation as positional arguments for __init__().

You should follow me on twitter here.

Welcome to the Brendel Consulting Blog

2009-05-30T06:30:00.004+12:00

Hello, and welcome to our blog. This is a first message, just to get us started. More is to come...

You should follow me on twitter here.