In Ways to Implement Agile Without Breaking The Bank, James Sullivan explains how Agile practices can be applied, piecemeal, to yield benefits even on big projects, where one of the Big Agile framework like SaFE or DAD might be called for, but not affordable. Though I’ve catalyzed several Agile adoptions, I’m not yet that familiar with these Big Agile techniques. So I read the article with great interest, to understand how people like Mr. Sullivan are thinking. I believe I have a pretty firm grasp on why Agile works, and therefore, what its Load-Bearing Walls are. So I read, checklist in mind, looking for what’s not there.
- Iterative development, with regular reviews of running software. Check.
- Control of technical debt through automated testing, integration, and deployment. Check.
- Frequent evaluation and adaptation of the process through structured retrospectives by the people doing the work. Silence, but maybe that’s just an oversight. A practitioner could add that later without rocking the boat too much.
- Variable overall scope, based on observed progress. UH-OH.
Read more of this article »
Retrospectives don’t get the respect they deserve, compared to the other elements of agile work. By comparison, they’re right-brained and touchy-feely. And even though I consider them a load-bearing wall of agile, weak retros aren’t felt nearly as quickly as failures in commitments, quality & technical debt, or user feedback. But they are essential to long-term team health, and maintaining and growing your way of working even in a changing context.
A Retrospective-Technique Library
Today I was coaching some ScrumMasters about the retrospectives they’re about to lead, and I ran across an excellent article by Ben Linders on InfoQ, “Why Do Teams Find It Difficult to Do Retrospectives?” I encourage you to read it, and follow the included links. There are a lot of good ideas to help your retrospectives fulfill the last principle in the agile manifesto:
At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Another excellent source of retrospective techniques is Agile Retrospectives: Making Good Teams Great by Esther Derby and Diana Larsen. They offer a five-step retrospective plan that keeps facts, insights, and actions from running together, and activities to help the team with each step.
The Basic Formula, And Adjustments
But sometimes it’s good to get back to basics. We spent our prep time today on the minimalist retrospective formula:
- What went well?
- What went badly?
- What shall we change?
Here are some ways we came up with to fine-tune it.
Encouraging a forward, action-oriented perspective
If your team is prone to dwell on the past rather than improve for the future, rephrase the questions:
- What should we keep doing (or do more)?
- What should we stop doing (or do less)?
- What should we start doing?
We thought of several other alternate wordings—you can do the same.
Hearing all voices
If you aren’t sure you’re hearing from everyone because of more and less dominant team members, especially on a bigger team, start the process with, “I’d like everyone to take the next 15 minutes to write down three items for each of the three questions.” That will let the introverts work the way they work best, by gathering their thoughts first. Then go around the room and let each person read out their list in turn, keeping a merged list on the board.
Making it safe
If you think that people might withhold information out of fear, add some anonymity. For a small team, collect the lists and write them out on the board yourself. For larger teams, have them work in triplets to merge their individual lists, and then have one person from each triplet read out.
If there’s a particular subject that you believe deserves special emphasis, shine a light on it. Sometimes a flashlight is enough. “As you’re writing down your items, spend at least a little time thinking about how well we handled the Definition of Done in the user stories and the conversations with the product owner and subject-matter experts she recommended.”
If the particular subject needs a searchlight, separate it. “Make a separate list of ‘keep doing, stop doing, and start doing’ when it comes to the build environment. That seems like it’s been a source of frustration.”
Helping your team follow through
Finally, to keep retrospectives from reverting from Lessons Learned to Lessons Observed (and repeated), write the action items—not too many!—on special cards and display them where you hold your daily stand-ups. Keeping them in sight is a good way to help people keep them in mind.
Agile has reached the early majority, and is running into trouble. At best, it doesn’t deliver what it’s capable of when the shackles are off. At worst, its failure feeds the political beast and vaccinates the organization against future change. “We tried that Agile nonsense, and it didn’t work! I Told You So™!”
Usually, Agile fails because organizations leave out an essential feedback loop.
- A governance process sets scope, schedule, and budget before initiating the Agile project, disabling the feedback of observed productivity (velocity) into a variable scope (the prioritized backlog); the team devolves to Cowboy Coding under the schedule pressure, and the product remains locked into a definition created when everyone was the most ignorant they will ever be,
- The team doesn’t do automated testing because they don’t know how or aren’t given time, disabling the feedback loop that controls technical debt; refactoring—as-needed redesign—is too risky, and productivity drops because the design is becoming incoherent,
- A defined process, outside the control of the Agile team, inhibits meaningful change based on retrospectives, which become glorified lessons-observed post-mortems instead of the lessons-applied process feedback loop they were meant to be, or
- Functional boundaries inhibit the frequent, regular evaluation of finished software by production SMEs and stakeholders in production-like environments, disabling the right-thing-done-right feedback loop.
Since early 2009 (four years at this writing), I’ve been using the metaphor of a house’s load-bearing walls to explain this. You can remodel Agile to suit your environment, but you can’t cut through a load-bearing wall, or it will fall down. In a match between personal will or political “realities” against the laws of physics, the smart money’s on the laws of physics, every time.
But I just read an article called Your HR Department can be hurting your company that caused me to say, “Maybe I’ve been missing a wall!” It’s not the gist of his article, but Don Herrmann makes this fascinating statement:
The real focus should be on leadership and for that a leadership expert is better suited. Notice I said leadership, a method of self directed empowerment that has sustainable accountability. I did not say leader, an others directed process relying on personal style that is not sustainable. Focusing on leadership ensures the long term strategic direction of the company is properly defined, navigated and accomplished. It survives the natural departure of leaders by ensuring a replicable and consistent conductor culture for the company.
This lack of what I’ll try calling “Peer Leadership” in established organizations explains some things I’ve experienced.
I frequently use Facilitated Requirements Workshops to elicit and analyze software requirements. The tensest moment for me is what I call “marker allergy.” I’ve just asked them to start writing user stories on 4×6 cards, using the markers in the middle of the table, and they’re just sitting there. And I’m wondering, “Are they allergic to markers or something?” They’ve always started writing, eventually, but meanwhile you could cut the tension in the room with a knife.
Or, we’ve just come out of a cycle start meeting with the sponsor/product owner with a set of user stories we’ve committed to, and we need to organize the work. They’re usually OK with putting together a WBS (Work Breakdown Structure), but when it comes to “Who Does What by When,” they all look at me. I’m the expert. I’m the “leader.” So I’m supposed to issue orders, and go down with the ship if need be, and I’m instead impersonating a mime. Eventually they get over it, but meanwhile you could cut the tension in the room with a knife.
Agile proponents talk a lot about self-organizing teams. I’ve spent all of my programming life in cultures like academic research and then start-up businesses where this was a given. The people were there in part because they couldn’t stand being told what to do! So I never really thought about self-organizing teams because that’s all I knew, until I started consulting in organizations with a management culture, and a “but I followed instructions” response when things go awry. It’s not their fault. It’s what they’ve been conditioned to do. They expect it, and there’s safety in it. Like Wallace, I can tell the sheep to “get yourselves organized down there,” but the results (after about 1:30 in the video) might not be as spectacular.
Perhaps Self-Organization or Peer Leadership is a fifth load-bearing wall, but one that needs to be installed, not merely protected like the others. Maybe team members used to big-M Management need explicit help with Peer Leadership. As a first draft, I’d include training and then coaching on how to:
- Size up and divide work,
- Estimate work, and help others do the same,
- Suggest first-draft ideas without being afraid of being wrong (or “getting a bad grade”–there’s a lot of PTSD left over from school. Just ask about the dreams),
- Be mindful of the “vibe” of the team and the work they’re producing,
- Identify strengths and weaknesses in self, others, and team, and change your mind when the evidence suggests (tell Malcolm Gladwell of Blink to go to blazes),
- Give feedback without becoming snarky, and
- Receive feedback (even snarky feedback) without becoming defensive, or, as Alan Godwin puts it, how to have Good Conflict, and
- How to prepare and give a simple briefing to a project sponsor without freaking out.
I’d then prep the sponsor on how to respond when things go awry (and they will). Keep the “Who’s in charge here?“ inside, and instead say, “What do you all plan to do differently now?” The experienced sponsor should use that experience to ask questions that help the team strengthen their plan, and resist the urge to pull an Al “I’m in charge here” Haig.
When people start to learn that they can’t hide behind a manager, but that they can take responsibility for their successes and failures, choose how to respond to the latter, and not get shamed and whacked by those with Authority, I’ll know that Peer Leadership is taking hold, and that our load-bearing walls are complete:
- Frequent inspection of running software in a (near-)production environment, by production people (functionality and usability feedback),
- Commitments based on actual velocity (productivity feedback),
- Automated testing (quality feedback),
- Regular retrospectives, resulting in action (process feedback), and, welcome to our new load-bearing wall,
- Peer leadership (team cohesion feedback).
Let’s try it out. Let me know how it goes!
One frustrating evening years ago, when I was a professional scientist and a self-taught programmer, I blurted, “Programming takes longer when I’m in a hurry than it does when I’m taking my time!”
I had stumbled upon what is sometimes called the Law of Software Quality.
Whoever said “Fast, Cheap, or Good—pick two!” started a dangerous lie. Unless it’s extremely simple software, to be used by its creator, once, and then deleted from the computer, improving quality makes the software less costly, not more.
OK, this isn’t true if you’re writing software where people could die if there’s a single bug (mistake) in it. That level of quality costs extra.
But for typical business software, there’s a sweet spot of quality, and most organizations aren’t there. Unless you have intentional quality measures in place, you can probably double your productivity by doing a little extra work of the proper sort at the right steps in the process.
So, why are effective quality practices not universal?
- To uninformed outsiders, everything but typing looks like Extra Work. Budgets with tasks (as opposed to features) as line items invite cutting quality measures to save time or money (but it doesn’t work in the end). Feature-based budgets are better because they drive conversations about feature value and scope when “cutting to fit.” But if your organization has separate groups that do quality-related tasks, task-based budgets are hard to avoid. Many big shops are organized this way.
- The most cost-effective quality measure, formal Inspection, isn’t as fun as programming. Lesser forms of requirements, design, and code review, while not as anal-retentive as Inspection, are less effective and still not as fun as programming. “The only problem we have around here is not enough meetings,” said no programmer, ever. Startups begin with just programmers (and maybe UI designers) and it’s easy to establish an inefficient code-and-fix culture and lay down a foundation of low-quality code that makes it hard for the business to pivot or scale up.
The first “quality measure” most firms introduce is Testing, in order to keep External Quality at an acceptable level. (External Quality is essentially, “does it work?”) The problem is that testing can only detect mistakes, not prevent or fix them, and then it can only detect certain kinds of mistake. Software can have high External Quality but still be made of Magic Spaghetti—poor Internal Quality.
Improving Internal Quality is where the payoff really happens. The reason why the quality curve looks the way it does is that tracking down mistakes in software is the most time-consuming form of Extra Work there is. It’s also hard to predict. Quality measures other than manual post-coding testing pay off because they detect mistakes relatively soon after they’re made, when they’re still easy to find.
Although I believe it’s easier for most teams to create high-quality software using an Agile method (I’m assuming this means automated testing, one of the Load-Bearing Walls of Agile), quality isn’t the main reason I prefer Agile. Regardless of methodology, introducing the right quality measures is a sure win.
Waterfall shops tend to use more formal requirements, design and code reviews, while Agile shops tend to use refactoring, supported by automated testing. Coding standards, especially naming conventions, are a good idea anywhere, provided they don’t trigger a religious war. Most languages have one or more de facto standards. Pick one and use it. Pair programming is the Agile form of code review. It catches a lot of mistakes but drives cost accountants crazy, unless they believe the quality graph. The most common review mistakes are reviewing too late and reviewing too much at once.
Another term for poor Internal Quality is Technical Debt. Like financial debt, it bears “interest,” in the form of extra time spent changing, enhancing, repairing, and upgrading a system that’s full of Magic Spaghetti and therefore hard to understand and change without making another mistake.
So whenever I have to help a firm Make Software Better, I first help them get a grip on their commitments. That frees up some time for them to introduce some quality measures. Those take time to gain traction. The longer the cycle time (time from gleam in the eye to feature in use), the longer it takes for the low-quality work in progress to clear the development pipeline and—gasp!—enter production, where it will continue to drive higher-than-necessary costs until remediated.
So reads a section heading from “The Best-Kept Management Secret on the Planet: Agile,” by Steve Denning, a contributor to Forbes.com.
Denning’s premise is that Agile was invented by the Wrong People, namely geeks—software developers—“the most problematic of a big organization’s employees,” and not B-School professors and management gurus. He compares it to the invention of the chronometer by a carpenter, the foundation of modern genetics by a monk, and the identification of the cause of stomach ulcers by a practicing pathologist (all true stories).
According to Denning, Agile is not just a software-development breakthrough, it’s a management breakthrough that allows disciplined execution and continuous innovation to occur at the same time. When I saw his article, my first thought was, “Here’s more evidence that Agile’s going mainstream!”
But then I got this sinking feeling. Read more of this article »
“Dog bites man? That’s not news. Man bites dog! Now THAT’S news!”
A mega-project that comes in near budget and that users actually like is NEWS…Man Bites Dog. But the April 2, 2012 Information Week’s “First Look at the FBI’s New Sentinel System,” mostly talks about user acceptance testing (300 FBI agents gave it an 8.5 out of 10), a re-platforming on Oracle Exadata servers to boost performance, and a description of its Microsoft-Outlook-like user interface. Almost in passing, the article says, “For the past 18 months, the FBI has been using agile development to push the long-delayed project across the finish line.”
Reading on, I learned that EMC (of VMWare fame) just bought Pivotal Labs, the maker of Agile project management tool Pivotal Tracker. The headline reads, “EMC Brings Social Networking to Big Data.” Pat Gelsinger, EMC President, describes it’s Greenplum Chorus as “Facebook for Data Scientists,” and the article goes on to say, “EMC’s Greenplum division and Pivotal Labs’ agile methodology experts developed the Chorus big data platform together,” before the Pivotal acquisition.
Big Government Projects and Big Data…that’s news. Agile? More and more, that’s just how you get software done. Dog bites man.
Those who wonder how to mesh Agile with a portfolio management or gating process will find some help in Agile Idea Qualification and Project Initiation, an engaging, 50-minute webinar by Sally Elatta.
Sally highlights the similarities between Agile and Lean. I wholeheartedly agree with her that the jelled, multi-functional team (BA, DBA, UI/UX, programmer, QA), not the individual “resource,” is the real unit of value creation in the software-intensive business. Rather than assembling teams around projects, flow work to teams.
In the early days, Agile started with, “in the beginning, there was the Project.” Sally backs up a couple of steps, to the Idea Qualification stage, which proceeds even Project Initiation. Tom Koulopolous teaches a lot of the same concepts. There are many more ideas than there is capacity to implement them, and that’s a good thing. Ideas are the gist of the innovation mill. There must be a central clearing house for idea evaluation, and it must be transparent and respectable. Though the norm, WSTL is just another form of the curse of local optimization.
Idea Qualification can, and must, be quick, both low effort and short cycle time. Sally recommends qualifying ideas by asking:
- What is its basic value proposition (make money, save money, improve customer satisfaction, improve competitiveness, stay in compliance, or KTLO—Keep The Lights On)? How can its success in delivering that value be evaluated? Qualitatively is better.
- Is it aligned with strategic direction? [Or, Iwould add, “Is it such a breakthrough idea that it represents a change in strategic direction?” This should be uncommon.]
- Does it have an empowered, engaged, knowledgable sponsor who is willing to see it through to Done and then take responsibility for the value created, 3-6 months after Done?
- Is it Feasible? (Do we have the time, money, know-how, and access to the necessary information and technology)?
Sally wisely recommends the use of a Kanban wall (visual representation of tasks and queues, with a limit on WIP—Work In Process—for every step) to manage Idea Qualification. Her explanation of a Kanban wall starting at about 28:45 in the webinar is very helpful. In addition to being Qualified, Ideas that are forwarded on to the next step, Project Initiation, must be prioritized. The Lean/Agile principle of minimal cycle time/WIP processes executed by standing teams off of a prioritized backlog, is used everywhere.
During Initiation, a portion of the implementation team, and the prospective sponsor, revisit the vision, measurable value, and feasibility (cost, skills, and technology) from Qualification, and add high-level requirements and costs, alternative solutions, a roadmap, and a bit more detail about how and when we will measure value. Projects that are approved to move on must also be prioritized. As Sally cleverly illustrates with the “Name Game,” starting at about 8:30 in the webinar, starting more projects than you have capacity just leads to multitasking which slows everything down.
Sally’s webinar provides a lot of help for business-side sponsors of software projects, which delights me because I think this is the hardest role and also the one with the greatest ability to affect significant change for the better. Starting at about 46:15 in the webinar, she provides a six-step process for evaluating value.
- Confirm (it should already have been identified) the value driver (revenue, cost, etc.)
- Identify the measure
- Gather as-is data (using the Lean principle of Go See)
- Define target measures—To Be—a valuable insight is to make this multiple thresholds, with a stretch goal and a target as well as a failure threshold; she also introduces probabilities or percentages, which is something I harp on when it comes to cost estimation
- Plan for and commit to measurement of actual outcomes
The higher the overall trust level in your organization, the better this will work. But even in a low-trust organization, there will be benefits, if you have the courage. One of the first gifts of Agile is transparency. Following this process of idea intake, qualification, initiation, and closing the loop with post-project measures, will reveal sources and sinks of trust.
I was a bad basketball player—pathetically bad. I was too bad even for scrub-team intramural basketball.
But I played a lot of “Horse” on the hoop on the garage.
(If you know Horse, skip this paragraph). In Horse, the first player calls a shot. If doesn’t make it, the next player calls a shot, and so on, until someone calls a shot and makes it. Then, everybody else in turn has to make the same shot. If they don’t, they get a letter, starting with H. When you’ve spelled HORSE, you’re out. The game continues until there’s only one player left.
The trick to Horse is calling shots that you can make and others can’t. Sometimes the shot-calling gets elaborate. Read more of this article »
I’m speaking at the High-Tech Happy Hour’s Pecha Kucha this Thursday the 18th sometime between 3:00 PM and 4:30 PM.
“A Tale of Two Processes” shows what happens to a hypothetical twelve-feature, one-year project when run following first the waterfall SDLC and second an agile method, assuming that the estimate/commitment is too low by one-third. This is a highly visual presentation of an article I published on the Wisconsin Technology Network back in February 2009.
What makes a Pecha Kucha fun for presenters and attendees alike is the run-and-gun format—20 slides, 20 seconds each (they’re on timed advance), no exceptions. Pecha Kucha enforces Franklin Delano Roosevelt’s advice to public speakers, “Be sincere, be brief, be seated.”