I think I know a fair bit about software project failures, at least small ones. My professional passion is to prevent them, and I’ve been mostly successful.
I learned about this latest one, “What Went Wrong? US Airforce blows $1B on failed enterprise software,” because of a lively discussion on the IIBA® (International Institute of Business Analysis®) LinkedIn group. At last count, there were 60 comments by 20 or so different people, most (including me) writing from general experiences and knowledge. The protagonists—Oracle, Computer Sciences Corporation (CSC), and the Project Management Office (PMO) for the Expeditionary Combat Support System (ECSS)—are being predictably guarded, but here are the facts, as near as I can tell:
- The just-canceled Air Force ECSS was one of nine ERP projects underway in the Department of Defense,
- Six of the nine are behind schedule by 2 to 12 years,
- Five of the nine are over budget by $530M to $2.4B,
- The canceled Air Force ECSS project was launched in 2005, with a total life-cycle budget for 2004-2027 of $3B, and projected savings of $13B. It was expected to replace 240(!) existing systems,
- Oracle won the “firm-fixed price” software bid at $89M,
- CSC was selected as the system integrator; they were given a stop-work order in September 2011, and their contract canceled in May 2012, after being paid $527M by the Air Force,
- The General Accounting Office (GAO, re-branded as the Government Accountability Office in 2004) had already reviewed four of the nine and requested tighter controls, mostly in the areas of cost and schedule estimation and planning, and
- At the time it was shut down, pilot deployments had been made in two of four functional areas, but with “known deficiencies and work-arounds.” The total expected lifetime cost had risen to $5.2B, and the completion date moved to 2016 (or 2020, depending on the source).
To put things in perspective, the USAF’s 2012 budget is about $165B. For a company with a $10M operating and capital budget, an ECSS-sized goof would pour about $60K down the drain. Smugness? Time to lose it. I’ve been involved in one of those, and that in the years since I understood software project failure.
You know what’s really sad though? The story of ECSS and its cancellation didn’t surprise me. As the saying goes, “Dog bites man? That’s not news. Man bites dog? Now that’s news!”
It doesn’t seem to have surprised very many other people, either. There’s little suggestion in the various articles about ECSS that the process is more than a tad broken, much less that there’s a scandal brewing. A year ago, IT project failure expert Michael Krigsman was philosophical. “The size and scope of government IT projects is staggering and failure rates are high. Then you take massive projects combined with the intense politics and information silos in the government…” He also mentioned the politics of dealing with various enterprise vendors during the procurement phase. But recently, Krigsman’s resignation had turned to frustrated anger. “Why did it take the [Air Force] $1 billion and almost 10 years to realize this project is a disaster? What kind of planning process accepts a billion dollars of waste?”
“The planning process for DOD ERP projects,” as it turns out. I just found a 2011 report, “Assessment of DOD ERP Systems,” by the Institute for Defense Analysis which, at 145 pages, I’ve no time to digest before press time. Plus, I have client work to do.
Now, neither you nor I will probably ever be responsible for a project anywhere near this big. But I’ve decided to devote a newsletter and blog post series to it, anyway.
- Even little software projects still fail too often.
- I think I understand software project failure, which means it’s time to start over with a clean sheet of paper so I can learn something.
- We learn deep lessons best from our own pain, and second best from outsized stories like ECSS (think “Excess”), so I think you’ll learn something, too.
I’ll do the hard work of wading through the 145 page report from the Institute for Defense Analysis and that GAO report (did you click on that link earlier? I dare you!) so you don’t have to. Plus, you’ve already paid your $3 tuition.
If you need a weekly email reminder, subscribe at the upper left.
London Transport passengers are exhorted to “Mind the Gap” between the platform and the train. The gap’s dangerous.
Likewise, I exhort business analysts to “Mind the Gap” between the Stated Requirements and the Implied Requirements. If you don’t, you’ve doomed your team to scope creep (more like scope avalanche) and probably a lot of budget and schedule pressure, and it won’t be the sponsor’s fault.
Minding the Gap—getting the implied requirements close enough—is the difference between a requirements order-taker and a Business Analyst.
So, how do you come up with requirements that no one’s stating?
I got the idea from Doug Rosenberg’s & Kendall Scott’s “Robustness Analysis” and Ellen Gottesdiener’s “Multi-Modeling.” I invented the Requirements Merry-Go-Round© as a way to explain it to others. I’ve since found it useful for the other things I’ve been writing about.
The original Merry-Go-Round© had only two horses—Function and Data.
You start with Function—say Use Cases or User Stories. Imagine some (very simplified) software to help pilots with flight planning and navigation. Functional requirement elicitation turns up:
- As a Pilot, I can select my departure, arrival, and alternate airports when preparing my flight plan.
- As a Pilot, I can enter the departure runway when I’m given my departure clearance.
- As a Pilot, I can enter the arrival approach to the arrival airport, and the assigned runway for landing at the arrival airport, when I’m assigned them by the air traffic controller.
- As a Pilot, I can select Voice Mode and the software will read me the waypoints, altitudes, and speeds for approach and landing.
Now, you head off to your lonely analyst’s garret (if you’re just learning) or let your Subject-Matter Experts (SMEs) take a break (if you’re a rock star) and head around to Data.
Everyone who makes software learns to look at the functional requirements for nouns: Airport, Flight Plan, Departure Runway, Arrival Approach, and Arrival Runway (I left out a few to keep it simpler). Then, sketch a logical data model. Then, re-convene your SMEs for some more elicitation.
The crafty BA will have noticed the sloppy use of sometimes “enter,” sometimes “select,” and found out that there are lists of Airports, and each Airport has a list of Runways, and each Runway has multiple Approaches.
Then, for each entity on the data model, ask three simple questions:
- Where do these data come from?
- What are all the possible uses for these data, and by whom?
- When and how are these data disposed of, and by whom?
You’re checking for CRUD—Create, Read, Update, and Delete. Data have to come from somewhere, and be read by someone.
The Pilot enters the Flight Plan. That’s one of the stated requirements. And s/he selects Airports, Runways, and Approaches. This we know.
But where do the lists of Airports, Runways, and Approaches come from?
“Oh, I guess we’ll need screens to enter and edit them.” Inspired by Ruby on Rails, modern application frameworks in many programming languages, and modern programmer tools like Microsoft’s Visual Studio now have “scaffolding;” when you create a data entity, it just creates plain-vanilla versions of these screens for you.
“We get them in a file from the government.” I’ll leave it to you to imagine what you would need to find out next!
“That’s a good question. I was hoping you could tell me. I’m just the marketing guy—I’ve been going around to air shows, talking to pilots about what they would like our software to do.” Someone, maybe you, now has a research project that could invalidate the feasibility of the whole project. Disaster averted.
To summarize, here’s how to Mind The Gap and find the requirements that no one’s mentioned:
- Elicit the functional requirements—the ones that are driving the project
- Infer a logical data model from the nouns in the functional requirements
- Infer the implied functional requirements from the entities in the logical data model using the CRUD check (Create, Read, Update, Delete
You’re not done, though. Your implied functional requirements are just a first draft for another round of elicitation. Involve additional subject-matter experts (SMEs) and expect a flow of additional information. If the implied requirements touch another part of the organization, you may have a new senior stakeholder. Get the sponsor’s help in developing that relationship and working out a potential change in the business requirements as well.
You still might not get a bonus though. Disaster averted is disaster disbelieved.
Until I started writing this series, the Requirements Merry-Go-Round© I drew on white boards had only User Interface, Function, and Data.
I added Business Rules because I know they’re important. But I’ve been putting off writing about them because I don’t know as much about the tools for eliciting, analyzing, and describing them as I feel I should. But that doesn’t mean I know nothing, so here are some simple tools that have worked well for me as I’ve developed requirements for smaller systems.
The Sponsor and primary Subject-Matter Experts (SMEs) are not likely to board the Merry-Go-Round at Business Rules, unless the system itself is about some kind of compliance. So you, the business analyst, have to ask the right questions to elicit them (draw them out). They’re often found near Data and Function, so ask these questions when working with those perspectives.
What exactly do people here mean when they say … ?
Read more of this article »