Another thread on the IIBA® LinkedIn forum, this one called, “Disturbed by seeing a raft of costly projects with no projected ROI – thoughts?” (you might have to be a group member to see it) prompted me to write a reply that I didn’t want to bury in the LinkedIn forums.
The author, Lance Latham, reported that of 32 projects, only 6 had any cost-benefit analysis at all, and 4 of those only looked at the costs of doing the project in different ways. I’m not surprised. Here are some suggested reasons why this is, and a thought on why it’s not a complete disaster.
Fear of being wrong
If I’ll be punished for being wrong, and I know that early estimates of cost and benefit are likely to be wrong, I won’t create them, or I won’t publish them. If I think the project is a good idea, I’ll create just enough supporting documentation to get it funded.
Ranges and probabilities instead of hard numbers
If I’ve never learned, or am not comfortable, working with numbers that I know are imprecise but not meaningless, I will struggle to come up with them, especially if the scope is vaguely defined. And the scope is always vaguely defined at first.
The very process of refining the scope requires decisions about whether a particular outcome is in scope or not, and that involves some kind of cost-benefit analysis of the thing in question. Or it at least means a guess at the cost of the thing, and a comparison of the uncertain total cost of all the things in my scope basket with a vaguely known budget. (The budget itself is subject to negotiation.) And at this point, the outcome or feature in question is itself only vaguely defined. It’s like a nested Russian doll of chicken-and-egg problems. How do I deal with that?
Organizational natural selection
People who are good at working with nested chicken-and-egg problems using uncertain numbers don’t seem to climb the corporate ladder to the height where they’re approving major projects. Or they lack other traits which are needed to keep from falling (or being pushed) off the ladder. Or they don’t want to climb the ladder at all.
But maybe it’s not the end of the world
If we believe Jonah Lehrer (“How We Decide”), maybe we shouldn’t be surprised. When analyzing complex decisions, we reach the point of diminishing returns relatively quickly.
Of Latham’s 32 projects, how many were turning out to be not worth the cost (the Ultimate Non-Functional Requirement)? How many projects that were shelved in favor of the 32 would have been winners? How much would additional analysis have helped?
I sometimes advise, but I’ve never ascended to the heights where I approve major projects. But I’ve watched (and been downwind). Here are my impressions.
- We do two levels of initial cost-benefit analysis: a) none that I can see, or b) way more than the uncertainties in the numbers justify.
- We’re not quick enough to radically re-scope or even cancel projects based on their actual progress.
“Poor Business Analysis-The Culprit of Project Failure.” That’s the topic of a lively thread, started by Rezaul Karim Majumder, on the IIBA (International Institute of Business Analysis) LinkedIn group.
It’s been healthy dialogue and I encourage you to read it—a lot about roles and the very relevance (or lack thereof) of the “culprit” role in software projects and systems thinking. But the 19th comment, by Kirby McInnis, asked a question I’d been waiting for.
“What constitutes success (or failure)?”
The traditional, project-manager definition is, “on time, on budget, as defined.”
We, as an industry, have a poor track record of success by that standard, hence the discussions about who’s to blame.
I think it’s an unnecessarily small target that distracts us from the overall goal of making money, so my definition of project success is,
“Worth more than it cost.”
By this measure, it’s possible to hit scope, schedule, and budget and still fail. More importantly, it’s possible to end up a long way from the original scope, schedule, and budget and have succeeded beyond one’s wildest dreams at the outset.
Here’s Kirby’s example for discussing success and failure.
“If a project is originally projected at $2M, and at $500K a decision is made to kill it because it’s now known that the real cost will be $4M, is that a success or a failure?”
Two things leap out. First, there’s no mention of missed requirements, only a missed cost estimate. Second, it’s in terms of money.
Many who claim the “Business Analyst” title shrink back from quantifying cost and value, and we shouldn’t. If we want to be more than IS-Requirements people, or even Business-Process people, we need to engage in cost-benefit analysis involving money, before there are even projects!
Here’s why we should get involved.
- Someone’s going to come up with cost-benefit numbers anyway, to justify this project instead of that one. If the numbers are tied to what the potential project will deliver—requirements—shouldn’t we be in the middle of that? And if they’re not tied to what the potential project will deliver, they’re at best guesses and at worst wishes.
- Once management funds a project, those cost-benefit numbers, wherever they came from, harden into commitments. If they’re unrealistic, they trigger negative feedback loops in productivity, quality, and waste, which drive projects towards failure.
So, my answer to Rezaul’s question is,
“Poor Business Analysis is indeed a culprit of project failure, but often not the way you think.”
By “Poor,” I don’t mean “low quality,” as in incomplete or incorrect requirements, although that does increase the risk of failure. I mean, “Too late and too narrow.” Business Analysis not done in time to support business decisions about IS-enabled process changes, and Business Analysis that doesn’t connect to money, is indeed “poor,” as opposed to “rich.” Such requirements-focused Business Analysis is not negligent, given the profession’s roots. But such Business Analysis is so much less than it could be. Think about it. Isn’t “Worth more than it cost” the ultimate non-functional requirement?
So, back to the LinkedIn discussion. Is Kirby’s canceled project a “success” or a “failure?”
At the project level, it’s a failure. We spent $500K and got nothing.
But thinking of the whole enterprise, we just made a second decision—to cancel a project. (The first decision was to start it in the first place, probably based in part on a $2M cost). Is canceling a wise decision, or did we just fail again?
Here is how I would approach it. Keep in mind that in reality I would know more about:
- How the $2M and now $4M estimates were made
- What’s actually been accomplished for our $500K
- What we’ve learned about the team, technology, and requirements since we started, and
- The nature of the $500K (cash outlay, or salaries we would have paid anyway)
but I want to keep the discussion simple without being simplistic.
First, only the $500K is a real number. The $4M and $2M are still estimates, the $4M produced by the same culture and probably the same people as the $2M. All estimates are really probabilities-they’re just seldom expressed that way. Because I don’t have actual project history I turn to Steve McConnell’s trusty Cone of Uncertainty from Software Estimation: Demystifying the Black Art.
The “Cone of Uncertainty” in software project estimates, depending upon project phase.”
We’ve spent $500K, or 12.5%, of what is now believed to be a $4M budget, and we’ve moved from left to right along the cone. But where are we? The Cone’s scaled by phase, not effort; most of the effort is in the right half. Based on numbers from Capers Jones’ Estimating Software Costs and a little Kentucky windage, we’re somewhere between “Requirements Complete” and the “User Interface Design Complete,” so there’s still a pretty big range around that $4M. Multiplying $4M by 1.4 gives me $5.6M, an upper “bound.” (It’s still only probability, there’s still perhaps a 10% chance that we’ll exceed it). Dividing by 1.4 gives me $2.9M, with a 10% chance that we’ll finish for less than that. So our most-likely outcome is $4M, with an 80% chance that we’ll be between $2.9M and $5.6M total cost.
In reality, I would also regard the $4M figure with some skepticism. The first estimate was optimistic, and there’s probably been pressure on the second one. It might not be unwise to shift the whole range upward. But in the interest of simplicity, let’s proceed with the $2.9M to $5.6M range, or $2.4M to $5.1M cost-to-complete from this point.
Now it’s time to ask a crucial question.
What are the present estimated benefits of completing the project as currently defined?
In my experience, there’s even more hand-waving and wishful thinking on this side of the equation than on the cost side, and even less attempt at verifying what really happened (admittedly, it’s harder to do). But in nearly all firms, there is an initiation document out there that says why the project is being done:
- Improved regulatory compliance
- Direct cost reduction, through decreased order handle times, decreased error rates, etc.
- Direct revenue increases, through larger orders, more orders, new customers, less churn, etc.
- Indirect benefits, through better information available to support subsequent decisions
- Consistency with overall strategy
(I’m using “order” as a place-holder for the unit of value that goes through some kind of process that we’re working to introduce or alter).
The first three are easier to estimate with more confidence, so I start there, and consider the last two if the cost-benefit comparison isn’t clear.
With the help of Subject-Matter Experts (SMEs) in Legal, Operations, Marketing & Sales, and senior management, an estimation facilitation technique I call The Stretch will get you to estimates, with ranges, for all of these things.
If our expected benefits are much less than $3.5M expected cost-to-complete, it’s probably an easy decision to cancel. There’s probably a better use for the remaining funds and people’s time.
If the expected benefits are still $5M or more, it’s probably an easy decision to keep going. It’s not the bargain we thought it was (or made it out to be) when we started, but unless there’s a really compelling use for the resources (people and money) that would be freed up by a cancellation, finish it.
So is Kirby’s example decision to cancel the project a success or a failure? I have to say,
“I don’t know. Let’s talk about expected benefits.”
 Thanks to Eli Goldratt for re-illuminating The Goal. People in non-profits may read this as, “Increasing capacity to serve clients.” Socially responsible business people might add, “While serving all our stakeholders.” Regardless, if the numbers don’t add up, you stop doing the remaining good.
A couple of days ago, I learned about Adam Grant’s book, “Give and Take,” through LinkedIn or Twitter or some other digital fire hose that I’ve turned on myself. Being very interested in the role of altrusim, collaboration, and chains of trust in business, I followed the link, read about the book, and saw that authors I like were endorsing it. I was impressed, and was planning to order a copy.
Like a lot of people, I’m a sucker for lists and surveys, so I took the survey at www.giveandtake.com. The questions were interesting, although it was sometimes frustrating because the answer I would have given wasn’t one of the choices. That happens. Good surveys are incredibly difficult to write; I’m sure Adam Grant (Ph.D. in Org Psych) knows that. Read more of this article »
Accounting is a business that I don’t know much about, so I’m going to this one-day conference in St. Louis in July (plus I can visit my brother).
The Accounting Technology Conference topics are buried in the middle of the 2020 Group’s Seminars page. So here they are, all by themselves, verbatim.
Accountants Technology Conference (Course Code: ATC) 9:00 AM-5:00 PM
This information packed 1-day seminar will showcase the latest and best technology that cutting edge firms are using to drive productivity, encourage efficiency, market effectively and attract (and retain) the best talent. This program will help you think strategically about your IT. (CPE credits: 8)
- Moving to less paper
- Why the Cloud?
- Work flow and doc management
- Top technology initiatives
- Setting an IT budget
- Web-based accounting
- Portals and security
- Apps, gadgets and other tools
If you’re an accountant and would like me to be on the lookout for something, let me know. If you would like to go too, you can register here. (They ask you what conference and venue you want on a later page).