As NASA starts to wind down their Space Shuttle activity in the next three years, the space agency's effort to return to the Moon has been ramping up quietly in the background. With their new Orion/Ares space vehicle combination, crew automation will definitely be on the top of software priorities for NASA. But with a much smaller budget and shorter timescale than the last lunar attempt, would NASA and its contractors embrace new approaches and techniques so our tax dollars are better spent? Can Dan Goldin's "Faster, Better, Cheaper" approaches be finally achieved?

A couple of months ago I was fortunate enough to join ThoughtWorks, a company that advocates the use of Agile software development practices (Extreme Programming, Scrum, TDD, etc.) to bring business value to our customers. I have been using Agile practices on my previous project for over three years and it had proved to be highly successful. And ThoughtWorks' experience in this area proves that Agile can also be applied successfully on large enterprise software projects. But can Agile be used on a highly mission-critical software project such as the one for the Orion spacecraft?

Over ten years ago my first programming job was for small software company developing real-time, safety critical software for controlling railway trains. The work we did was the embodiment of the Waterfall model. The system requirements were collected and analyzed. The model was designed and validated. Then we mere programmers set out to write code to realize the model. Huge amount of unit tests and integration tests were created to make sure our code did what the model said it should do. All the while, the project manager kept track of our progress to ensure that, hopefully, we delivered the product on time and on budget.

At first glance, Agile sounds like a good fit with this type of project where requirements are generally very well defined and correctness are paramount. Short iteration and test-driven development will ensure features are delivered often and proved to be working by the unit tests. Continuous integration means there will be fewer surprises as multiple systems are joined up to work with together. The costs of requirement changes will be reduced and can be implemented quickly, rather than in the next version.

But would the world of safety /mission critical software development, dominated by engineers and scientists, be receptive to the less rigid world of Agile development? Would they feel that without the top down approach, its highly structured development process, and the tightly prescribed set of delivery artifacts, the project delivery cannot be guaranteed?

I would love to hear from people who have more recent experiences in this area of software development with regard to Agile. Is it being used, is it being used widely, and how effective it has been?

My wife (and the rest of my family in fact) has never comprehended what I do as a software developer. Throughout all the years we have been together she has seen me sat in front of the computer and typed code into the screen for hours on end. But still she does not know how ideas in my head are transformed into a software application like one that she uses everyday. She thinks it is all voodoo magic, really she does. Last week, I explained to her that software development is kind of like cooking. Not the follow the recipes in the cookbook type, rather the Michelin Star chief type where the dish is created out of thin air.

The image of 'programmers' and 'hackers' portrait by Hollywood does not help either. When I tell people that I write computer software for a living, I am pretty sure in their mind they see binary code (probably green) flowing down the black screen continuously in multiple overlapping windows. Just like in 24 or The Matrix, in fact. And coding involves typing a few lines of indecipherable command in one of those black windows, more code flows down, and Boom! Global warming is solved!

While this image works really well in a TV series or movie, unfortunately software development is not that dramatic or glamorous. The idea of someone (be it a genius or a mad scientist) working alone deep in the basement and conjuring a software application out of nowhere and that every single line of code is memorised is so deeply ingrained in the general population psyche that I truly believe this is affecting software development as a whole.

Of course, it is partly our own fault. We, the software developers, have worked so hard to make complex and powerful software easy to use for the users. We have worked so hard to improve our development process to decrease the turnaround time for each development cycle so new features and bug fixes are delivered to the users with increasingly shorter time-scale. This has raised the expectation of the users on our ability to deliver feature that looks deceptively simple on the surface but probably hugely complex behind the scene.

Is there a light at the end of this tunnel for software developers? Perhaps, but only if we work very hard on at least the following two areas. First better design concepts (object-oriented design, design patterns, refactoring), processes (Agile, TDD), and development tools (C# 3.0's LINQ, Ruby On Rails, etc.) will continue to be improved to let us deliver more and faster. These are already in place and many clever people are working hard to take us there. More importantly, as well as building kick-ass software; we also need to begin an education initiative.

We need to change the perception of our work in the users' mind from part voodoo magic, part art, part skills, and full nerds to a disciplined profession. Some may even want to call it 'software engineering', do you believe that?! Until the general population considers software development on the same level as lawyers, doctors, or engineers, recognises the immense complexity of software applications and the skills requires to build them out of thin air, our job as software developers would only get harder and harder.

* Dev Chair: The place where I plant my butt after a hard day of code bashing and muse about meta-issue. [Alex Hung is a co-developer of desktop blogware ecto and will be penning a regular series for DLS about software development.]