When Illusion beats Reality

We often help our customers update their software development environments to more modern practices.  Sometimes this involves languages and frameworks. Other times it involves updating a classic waterfall process with many of the practices from the lean and agile communities. At still other times, it involves an investment in the user experience.

This particular post is about introducing modern software development techniques, including scrum, into a large organization. I’ve been noticing how much the concerns and the values change as what we do becomes more and move visible to larger and larger parts of the organization.

Our first projects were small (for an enterprise organization). The stakeholders were key members of the user community, and maybe first or second level managers. Our scrum process worked quite well, and we had good success. The stakeholders understood the core values of our project: Let’s build useful software in a timely fashion. Everything else supported that goal. We wrote only enough documentation to support building software. Most (almost all) of the testing was automated. The automated tests were the test plan.

The projects were successful. Our stakeholders showed their management that our projects all achieved great ROI, and we showed a much better on time track record than their traditional process.

That was great, until it wasn’t.

See, we were successful. But, we weren’t ‘following the corporate standard.’ As more and more new managers and new departments were involved, the priorities changed. That core value of “building useful software in a timely fashion” was replaced by a core value of “following the process”.

I have to admit, following a waterfall process is seductive. It 'looks like you can accurately predict the future. You spec things, you build schedules, you ‘follow the plan’. Unfortunately, software projects (and real life) are not like that. No plan survives contact with reality.  But, until those fallacies are exposed, the illusion is powerful. “These other teams have a plan. They’re organized. They’re following the process.”

My theory on why this happens is this: The farther removed a manager in a large enterprise actually is from a project, the more their goal shifts from the project actually succeeding to not being responsible. The enterprise process models, in my opinion, make that easier. The team no longer ‘owns’ the success or failure of the project. The team ‘owns’ adherence to the process.

I hope that data, such as this provided by Mike Cohn, can help. Will the same people swayed by “THE PROCESS” by swayed by “THE DATA”?  (Read the comments on Mike’s post, he does discuss some of the industry’s concern with Standish reports.) 

I hate writing just ‘downer’ posts, so I must close with some wisdom from Mary & Tom Poppendieck. Whenever we find ourselves in one of these situations, we look at Principle 7 from “Implementing Lean Software Development: From Concept to Cash”:  Optimize the Whole. Process has some place. and some communication is valuable. However, too much process means not optimizing the whole, and not delivering great software. We keep reminding our stakeholders that the point isn’t to have a pretty plan. It’s to have working software sooner. Some get it. They hopefully keep pulling others along.

Created: 2/14/2012 7:35:44 PM

Current Projects

I create content for .NET Core. My work appears in the .NET Core documentation site. I'm primarily responsible for the section that will help you learn C#.

All of these projects are Open Source (using the Creative Commons license for content, and the MIT license for code). If you would like to contribute, visit our GitHub Repository. Or, if you have questions, comments, or ideas for improvement, please create an issue for us.

I'm also the president of Humanitarian Toolbox. We build Open Source software that supports Humanitarian Disaster Relief efforts. We'd appreciate any help you can give to our projects. Look at our GitHub home page to see a list of our current projects. See what interests you, and dive in.

Or, if you have a group of volunteers, talk to us about hosting a codeathon event.