It seems like Agile software development methodologies (XP, Scrum) and their associated techniques like pair programming are all the rage these days. I’ve never worked for a truly successful Agile shop, nor am I actually a developer (any more) but I do want to address the delusions that some people, particularly managers, have around Agile. While I do believe Agile methodologies have their time and place, implementing these tactics do not guarantee that your project will deliver on time, on budget, or with happy developers. Here are a few of my observations about Agile and how it is misused.
Agile is not a substitute for the initial business requirements. If anything, it’s crucial that the business requirements for the product be negotiated and solidified by proper business analysts up front — before the developers even get a crack at coding. That way, you have a baseline set of requirements against which you can measure progress. While you may not end up delivering 100% of the stated requirements because of the iterative approach, at least you have a template against which to negotiate the scope of each iteration with the client.
Agile doesn’t mean you will necessarily emit code faster. I’ve seen this misconception espoused by unskilled managers. It’s impossible to predict whether one’s project will be completed faster or slower than a similar project implemented using traditional development methodologies. In some cases, an Agile approach will be slower, because the code may need to be done several times due to changing requirements. It’s a mistake to think that Agile is a process with which you can "beat" developers into working faster (I’ve heard that before from the mouths of disrespectful managers.)
Agile means that a great deal of control is given to the developers. It will be difficult, if not impossible, to implement Agile at an organization with a very top-down hierarchical functional management tree. Agile requires managers — and even senior managers and directors — to really trust the staff in the trenches implementing the product. If, on the other hand, developers are seen as being "the inmates in the asylum" and management is reluctant to let the inmates run the asylum, then Agile will probably not work for you.
Don’t implement Agile just because it’s the hottest buzzword on the block. What’s the problem that you’re trying to solve by choosing a different development methodology? If your existing methodology is not working — or if you don’t have a methodology at all — it’s worthwhile asking questions about why, before jumping onto another methodology "just to try it". For example, your products might be shipping late because of poor project management — and changing methodologies is not going to help you with that. Which brings me to:
Agile is not a substitute for project management. If anything, you need to have better project management when implementing Agile; who’s going to communicate expectations and outcomes to the client? Sure, you might not have a 6-12 month long WBS at the beginning of the project, but you’re still going to need a stellar project manager — if for no other reason than, if the developers are just developing and charting their own territory, then the PM is going to have to do the heavy lifting to take care of everything else.
Those are my observations, having worked in a couple shops where Agile was poorly implemented or its purported benefits were wrongly sold as a panacea to solve endemic problems in the organization. I think the lesson to be learned is that one should never change methodologies to solve poor business processes and disorganization; fix those problems first, and then you can contemplate whether Agile makes sense for your organization.
Footnote: There’s an interesting book by XP’s own creators entitled Extreme Programming Refactored that also might be worth reading; although I have not read it, it seems to be a counterpoint to their own literature promoting XP. It’s great to hear both sides of the story from the same individuals.
As you said, Agile is really hyped these days. It got a huge boost last month when Toyota said it's going to switch to Agile.
Whether or not it's merely a hype is something that only time will tell, but I have to concur that there are some misconceptions about Agile. I have published an article about the limitations of agile (can also be agile requirements), but the article explains in details the drawbacks of Agile. I hope you'll have time to read it and give me your comments!