A Note on Programmer Productivity

Posted by Robert Merrill on November 22, 2008 under Estimation | Read the First Comment

In Recession: A Quiet Opportunity for your Software-Intensive Business, I suggested that the productivity gains from improving your team far outweigh the cost savings possible in a buyer’s market for talent.

Just how variable is programmer productivity, anyhow?

Person-to-person and same-person variability

If you give a group of people the same programming project, how long does it take them? How much does an individual’s productivity vary from task to task?

As Sherwood (2006) says clearly, “I have found it surprisingly hard to find good academic literature that measures productivity.” She quotes the following ranges:

• 28:1 best to worst (Sherwood 2006a, citing Sackman et al. 1968)
• 22:1 to 8:1 best to worst for two problems, depending on how you treat the outliers (Sherwood 2006a, citing Curtis 1981)
• 3:1 to 2:1 best to median for two problems (Sherwood 2006a, based on Curtis 1981 data)
• 4:1 to 2:1 best to median (Sherwood 2007, citing DeMarco and Lister 1985. The latter also found that productivity was correlated with the organization that programmers came from, even though they worked independently during the study).
• 2:1 best to worst (Sherwood 2006b, citing Ko et al. 1996)

Spolsky (2006) quotes another range:

10:1 to 5:1 best to worst (Spolsky 2000, based on several years of student project data from a computer science professor at Yale)

Project to project variability

My own project metrics over a period of about five years show:

• 20:1 best to worst project productivity (largely the same people, in different teams, using different technologies to implement various web application projects. Some were all-custom, and others were customization of commercial or open source software).
• 3:1 difference between productivity on two projects completed by the same developer using the same technology

The COCOMO II estimation model (Chulani et al. 1997) has adjustment factors for total project effort based on personnel attributes by role:

• 2.25:1 best to worst analyst
• 1.85:1 best to worst programmer
• 1.5:1 most to least familiarity with the application domain
• 1.5:1 most to least familiarity with the technology in use

Interpretation

There is a lot of variability. No wonder estimation is difficult. Here are some principles about productivity from the above.

• Productivity probably varies within a given organization by at least 2:1
• Individual productivity varies from task to task, probably by an equal amount of 2:1
• A team is more than the sum of its parts. Analysts have the highest leverage; analyst capability swings productivity by over a factor of two, even though they directly contribute maybe one-fourth the effort (my rule of thumb).
• Project productivity, based on individual capability, team capability, familiarity with the domain and the technology, and the technology in use, easily varies by 10:1.

References

Chulani, S., B. Clark, and B. Boehm 1997: Calibration results of COCOMO II.1997. 22nd Annual Software Engineering Workshop, NASA Goddard Space Flight Center.

Curtis, B 1981: Substantiating programmer variability. Proceedings of the IEEE, 69, 846. ISSN 0018-9219.

DeMarco, Tom and Tim Lister 1985: Programmer performance and the effects of the workplace. International Conference on Software Engineering, 268-272. ISBN 0-8186-0620-7.

Ko, A., Myers, B. A, Coblenz, M. J. and H. H Aung 2006: An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks. IEEE Transactions on Software Engineering, 32, 971-987.

McConnell, Steve 2006: Software Estimation: Demystifying the Black Art. Microsoft Press, 308 pp. ISBN 0-7356-0535-0.

Sackman, H., W. J. Erickson and E. E. Grant 1968: Exploratory experimental studies comparing online and offline programming performance. Communications of the ACM, 11, 3-11. ISSN 0001-0782.

Sherwood, Kaitlin Duck 2006a: Don’t get stuck, a review of Sackman and Curtis. http://www.webfoot.com/blog/2006/12/07/programmer-productivity/.

Sherwood, Kaitlin Duck 2006b: Review of Ko’s “An exploratory study of how developers seek, relate, and collect relevant information during software maintenance tasks.” http://www.webfoot.com/blog/2006/12/15/programmer-productivity-part-2/.

Sherwood, Kaitlin Duck 2007: Review of “DeMarco and Lister’s Programmer performance and the effect of the workplace.” http://www.webfoot.com/blog/2007/02/16/demarco-and-lister/.

Spolsky, Joel 2000: Hitting the High Notes. http://www.joelonsoftware.com/articles/HighNotes.html

We cannot win, but does it matter?

Posted by Robert Merrill on under Estimation | Be the First to Comment

In Software Sanity: Accurate Estimates and Other Myths, Robert explains why we can’t solve the project estimation problem by getting better at estimation. But that doesn’t mean there’s no solution.

Stupid Estimation Tricks 1: The Squeeze

Posted by Robert Merrill on October 2, 2008 under Estimation | Be the First to Comment

Use this to help someone (or yourself) to come up with an estimate when “you have no idea.”

Imagine you just met with a sponsor for a half hour and heard a project vision. She ended the meeting with, “The steering committee meets this Friday, and I need to know about how long this will take. You don’t need to be exact–just a ballpark will do.”

Famous last words.

Being a firm believer in the principle that “estimates should come from people who will do the work,” you convene the three-person team that seems right for this project.

Six deer eyes, staring at the headlights. Here’s where I use a facilitation technique I call “the squeeze.”

“Could you do this in a month?” I ask. I’m met with gales of laughter. I open my notebook on the table in front of them and draw a graph. “OK, one month, no way.”

Initial graph, with zero probability at one month

“How about a year?” I ask. “No problem,” responds the chorus. “They’ll never give us that long.” Cautious Chuck then adds, “And don’t forget all of the data conversion that they say they’ll do, but we’ll end up doing ourselves. It’s not a sure thing.” So I plot another point–12 months, 90 percent. “OK with that?” “Sure,” says Chuck.

Graph with four points

“How about three months?” Everyone shifts nervously in their chairs. “Well, if we can code it in Rails, we can reuse most of Waldorf,  and if they’ll drop some features if we get in trouble, we could have something in three months,” says Optimistic Olivia.

“What are the odds?” I ask. “50/50?” “No way,” says the chorus. “One in ten?” “Yeah, that sounds about right.” So I plot another point, and draw a smooth curve.

Graph with initial range

“Does 50/50 in eight months, and almost-sure-thing in thirteen months sound about right?”

“Sally [the sponsor] is usually pretty reasonable. Working with her, we could have something done in a year, even if we run into data problems,” says Pragmatic Paul. So I adjust the curve a little.

Graph with range adjusted on the high end

“OK, I’ll tell Sally 8 months 50/50, and that we can commit to 12 months based on what we know now. She understands ranges and won’t sign us up for trouble.”

The Squeeze is useful a number of reasons:

• By starting with a very low number and going to a very high number, it gives the estimators permission to say NO and YES with confidence.
• It also reflects the reality that the estimate is a range, and not a single number, and produces a range as a result.
• It can be applied very early in the lifecycle, even “gleam-in-the-eye” stage, as here.

If you’re supporting a knowledgeable decision-maker (more on that later), you can give them a range directly from The Squeeze. If you know your number will turn into a not-to-exceed, you can use the top end, and then have the 50/50 for your initial project plan.

You can combine The Squeeze with Wideband Delphi if you’re worried about groupthink setting in.