What’s a Failure? (and is Poor Business Analysis the culprit)

Posted by Robert Merrill on June 15, 2013 under Putting the B in Business Analysis | 2 Comments to Read

Poor Business Analysis-The Culprit of Project Failure.” That’s the topic of a lively thread, started by Rezaul Karim Majumder, on the IIBA (International Institute of Business Analysis) LinkedIn group.

It’s been healthy dialogue and I encourage you to read it—a lot about roles and the very relevance (or lack thereof) of the “culprit” role in software projects and systems thinking. But the 19th comment, by Kirby McInnis, asked a question I’d been waiting for.

“What constitutes success (or failure)?”

The traditional, project-manager definition is, “on time, on budget, as defined.”

We, as an industry, have a poor track record of success by that standard, hence the discussions about who’s to blame.

I think it’s an unnecessarily small target that distracts us from the overall goal of making money[1], so my definition of project success is,

“Worth more than it cost.”

By this measure, it’s possible to hit scope, schedule, and budget and still fail. More importantly, it’s possible to end up a long way from the original scope, schedule, and budget and have succeeded beyond one’s wildest dreams at the outset.

Here’s Kirby’s example for discussing success and failure.

“If a project is originally projected at $2M, and at $500K a decision is made to kill it because it’s now known that the real cost will be $4M, is that a success or a failure?”

Two things leap out. First, there’s no mention of missed requirements, only a missed cost estimate. Second, it’s in terms of money.

Many who claim the “Business Analyst” title shrink back from quantifying cost and value, and we shouldn’t. If we want to be more than IS-Requirements people, or even Business-Process people, we need to engage in cost-benefit analysis involving money, before there are even projects!

Here’s why we should get involved.

  1. Someone’s going to come up with cost-benefit numbers anyway, to justify this project instead of that one. If the numbers are tied to what the potential project will deliver—requirements—shouldn’t we be in the middle of that? And if they’re not tied to what the potential project will deliver, they’re at best guesses and at worst wishes.
  2. Once management funds a project, those cost-benefit numbers, wherever they came from, harden into commitments. If they’re unrealistic, they trigger negative feedback loops in productivity, quality, and waste, which drive projects towards failure.

So, my answer to Rezaul’s question is,

“Poor Business Analysis is indeed a culprit of project failure, but often not the way you think.”

By “Poor,” I don’t mean “low quality,” as in incomplete or incorrect requirements, although that does increase the risk of failure. I mean, “Too late and too narrow.” Business Analysis not done in time to support business decisions about IS-enabled process changes, and Business Analysis that doesn’t connect to money, is indeed “poor,” as opposed to “rich.” Such requirements-focused Business Analysis is not negligent, given the profession’s roots. But such Business Analysis is so much less than it could be. Think about it. Isn’t “Worth more than it cost” the ultimate non-functional requirement?

So, back to the LinkedIn discussion. Is Kirby’s canceled project a “success” or a “failure?”

At the project level, it’s a failure. We spent $500K and got nothing.

But thinking of the whole enterprise, we just made a second decision—to cancel a project. (The first decision was to start it in the first place, probably based in part on a $2M cost). Is canceling a wise decision, or did we just fail again?

Here is how I would approach it. Keep in mind that in reality I would know more about:

  • How the $2M and now $4M estimates were made
  • What’s actually been accomplished for our $500K
  • What we’ve learned about the team, technology, and requirements since we started, and
  • The nature of the $500K (cash outlay, or salaries we would have paid anyway)

but I want to keep the discussion simple without being simplistic.

First, only the $500K is a real number. The $4M and $2M are still estimates, the $4M produced by the same culture and probably the same people as the $2M. All estimates are really probabilities-they’re just seldom expressed that way. Because I don’t have actual project history I turn to Steve McConnell’s trusty Cone of Uncertainty from Software Estimation: Demystifying the Black Art.

Graph of expected errors in software project effort estimates, starting with a 16x range at Initial Concept and decreasing to a 1.6x range at User Interface Design Complete.

The “Cone of Uncertainty” in software project estimates, depending upon project phase.”

We’ve spent $500K, or 12.5%, of what is now believed to be a $4M budget, and we’ve moved from left to right along the cone. But where are we? The Cone’s scaled by phase, not effort; most of the effort is in the right half. Based on numbers from Capers Jones’ Estimating Software Costs and a little Kentucky windage, we’re somewhere between “Requirements Complete” and the “User Interface Design Complete,” so there’s still a pretty big range around that $4M. Multiplying $4M by 1.4 gives me $5.6M, an upper “bound.” (It’s still only probability, there’s still perhaps a 10% chance that we’ll exceed it). Dividing by 1.4 gives me $2.9M, with a 10% chance that we’ll finish for less than that. So our most-likely outcome is $4M, with an 80% chance that we’ll be between $2.9M and $5.6M total cost.

In reality, I would also regard the $4M figure with some skepticism. The first estimate was optimistic, and there’s probably been pressure on the second one. It might not be unwise to shift the whole range upward. But in the interest of simplicity, let’s proceed with the $2.9M to $5.6M range, or $2.4M to $5.1M cost-to-complete from this point.

Now it’s time to ask a crucial question.

What are the present estimated benefits of completing the project as currently defined?

In my experience, there’s even more hand-waving and wishful thinking on this side of the equation than on the cost side, and even less attempt at verifying what really happened (admittedly, it’s harder to do). But in nearly all firms, there is an initiation document out there that says why the project is being done:

  1. Improved regulatory compliance
  2. Direct cost reduction, through decreased order handle times, decreased error rates, etc.
  3. Direct revenue increases, through larger orders, more orders, new customers, less churn, etc.
  4. Indirect benefits, through better information available to support subsequent decisions
  5. Consistency with overall strategy

(I’m using “order” as a place-holder for the unit of value that goes through some kind of process that we’re working to introduce or alter).

The first three are easier to estimate with more confidence, so I start there, and consider the last two if the cost-benefit comparison isn’t clear.

With the help of Subject-Matter Experts (SMEs) in Legal, Operations, Marketing & Sales, and senior management, an estimation facilitation technique I call The Stretch will get you to estimates, with ranges, for all of these things.

If our expected benefits are much less than $3.5M expected cost-to-complete, it’s probably an easy decision to cancel. There’s probably a better use for the remaining funds and people’s time.

If the expected benefits are still $5M or more, it’s probably an easy decision to keep going. It’s not the bargain we thought it was (or made it out to be) when we started, but unless there’s a really compelling use for the resources (people and money) that would be freed up by a cancellation, finish it.

So is Kirby’s example decision to cancel the project a success or a failure? I have to say,

“I don’t know. Let’s talk about expected benefits.”


[1] Thanks to Eli Goldratt for re-illuminating The Goal. People in non-profits may read this as, “Increasing capacity to serve clients.” Socially responsible business people might add, “While serving all our stakeholders.” Regardless, if the numbers don’t add up, you stop doing the remaining good.