Too long/did not read version:

• The people-people took over the label Agile, as they do.
• The Sound Craftspeople, as always, have work to do.
• It’s time for us to get back to work.

Details follow.

About the Agile-PMI Partnership

After waking up to this announcement, that the Agile Alliance joined the project management institute, i’ve been processing for a day. What follows is some background that might be helpful, and a few thoughts.

Matt’s Background on The Agile Manifesto

The late 1990’s were a different time for software. The internet didn’t really exist in it’s current form. There were not a lot of opportunities for people to share practices that were actually working. The loudest voices in the software room were talking about the three c’s: Complete, Consistent, and Correct requirements. “Scope Creep” was a bad thing, not an emergent property. In graduate school, we studied “Change Control Boards“, whose job it was to contain scope creep. The Waterfall Model de facto ruled the roost, back before you could download it as a PDF and read for yourself that Royce said it did not work for complex projects … in 1976. There were, of course, exceptions. DeMarco and Lister wrote PeopleWare which suggested treating people as … well … people … on software efforts. Fred Brooks published The Mythical Man-Month, which included the idea of “Plan to throw one away; you will, anyway”, which would inspire rapid prototyping. Eli Goldratt had written his book on Critical Chain project management. Perhaps most importantly, the book “Wicked Problems, Righteous Solutions” laid the way for Scrum, back in 1990. Yet, Sadly, as Demarco/Lister pointed out in their first edition of PeopleWare, the typical programmer had one or zero books on the shelf, and if they had a book, it was a book about a programming language. 

Many technical staff had lost the confidence of leadership, at least partially because of the inherent problems in the Waterfall[1]. The technical people just could not keep their promises, they could not hit deadlines. (I was one of those technical people, and it was true. Now where, exactly, those deadlines came from, that I never really figured out.) This loss of confidence spawned an entire group of people to act as go-betweens for the people building the software and the customer. These were the managers, the analysts, the Project Managers, the Liaison Officers, the customer service representatives, as well as an entire new layer of process people and people-people who were going to make software development “stable, predictable, and repeatable.” To a great extent, these folks lacked deep expertise in software, but that was okay, because the programmers also lacked deep expertise. By the 1990s In the developed world, it seemed that every promotion was away from programming. People wanted to become architects, managers, directors and vice presidents. “Coding” was left for those who were left, as an entry-level thing for beginners, and perhaps also for those not bright enough to be promoted elsewhere. As you expect, the programmers would be given poorly-thought-through requirements by people who had never themselves written code, along with due dates based on wishful thinking.

To borrow a line from Thomas Moore’s Utopia: First we made bad programmers, then we punished them.

Then the conference at snowbird happened

Snowbird is a ski resort in Utah. The “Snowbird conference” was a weekend when seventeen software experts got together to write up something about their shared values, which at the time were loosely grouped around “Lightweight Methods.” Somebody called it Agile, the name stuck, they wrote the Manifesto quickly and hit the slopes to go skiing.

Brian Marick, one of the signatories of the Manifesto, once explained to me that you could think of the Manifesto as a conversation between those  who actually understood how to develop software and the people-people. It might go something like this:”Please get out of the way. Let us talk to the customer directly, and trust us to build the software. We know how to do it.” The two main supportive methodologies at Snowbird were Scrum and XP. Both contained the idea of frequent, shippable-software, visible progress. This allows the customer to see the software coming together and steer. That is a different approach than the dominant paradigm of the time (around 2001), which was stage-gates and delivery-date “trip wires.” 

The other interesting trick in the Agile Manifesto was the marketing bit. It was specifically written to be directly counter to the loudest voices in the SoftwareDevelopmentIntellectualOSphere, the ones focused on comprehensive process documentation and contract negotiation. It was actually worded so those companies (one was big and blue, another was headquartered at Carnegie Mellon University, a third was, well, the PMI) could not even try to claim to be Agile, as it would contradicting their own marketing literature.

And, indeed, a few years later, when I put a poster up that showed the Agile Manifesto, one of the PMI-PMPs in the building walked by and said out loud “This offends me.” 

That was 2006, three years after I signed the Manifesto in 2003, and four since I read my first XP book, XP:Installed.

A small, scrappy band of rebels had figured out how to do this software thing through iterating and feedback. The internet was young, and people were talking in Yahoo Groups with hundreds (thousands?) of posts daily about how to do XP and Scrum. Bookstores (remember those?) were reserving yards of shelf space to a seemingly never-ending series of Extreme Programming Books by Addison-Wesley. XP and Scrum programmers were this strange group of people challenging the status quo.

Of course, I put the title “Matt’s Background” above because it is my background. You are free to have your own experiences, but those were formative for me.

The good news is, whatever the background, the stuff actually worked[2], at least on small projects. Part of actually working was acknowledging reality. For example: “If we have a deadline but do not know what we will build, then the best we can do is to define a set of features, build from the top down, let you periodically review working software and adjust the plan. When the time runs out, we have built whatever we built.” This acknowledges the fiction of the “complete, consistent, correct specification defined after the schedule is defined” as what it is, a work of fiction.

I remember one employer who told me “We are going to let you do your thing on this project. Just don’t tell anyone what you are doing.” This struck me as an implicit admission that the techniques worked – but the organization could not recognize that fact, even if individuals in management could. I left a little while later.

Then, slowly, something happened.

Agile became the status quo.

Crossing the Chasm 

Geoffrey Moore wrote a book about when a technology moves from “early adopter” to mainstream acceptance called Crossing the Chasm. Malcolm Gladwell wrote Tipping Point about that point when a technology becomes so popular suddenly everybody needs it. For example, when suddenly everyone has a Linkedin Account but you, and you need to create one. That means the people who do not have one yet are even further behind, and they need to create one, and so on. Thus the technology goes from irrelevant to a necessity, seemingly overnight. (Before linkedin, the same thing happened with facebook and email). On the negative side, in the television show “Happy Days”, the character of the Fonz Jumped the Shark, which came to mean something so amazing the show could only go downhill from there.

Companies started hiring Scrum Masters. Certification and coaching businesses took off. To the extent they offered real education, I wish them every success. I even arranged a Scrum Master course taught by a signatory of the Agile Manifesto, partially so I could attend myself. I earned my Certified Scrum Professional badge, indicating I had actually worked on Scrum Projects for a few years and continued my education. The impact was not bad at all.

Suddenly Agile was popular. Executives were bragging about it.

But wait, you ask. If Agile is suddenly so popular, how can that be bad?

 

Jumping the Shark

Something else happens when intellectual ideas become popular. In his book Secrets of Consulting, Jerry Weinberg sums it up as his “Law of Raspberry Jam.”[3] That is, “The Wider you spread it, the thinner it gets.” Extreme Programming required a bunch of difficult technical practices. In many cases, a Scrum “conversion” could just organize the existing teams around a board and changing an analyst’s title to Product Owner. By and large, Scrum was easier to adopt. To be fair, it also did include a good number of innovations on the management side. 

Another thing happens when an idea becomes popular: A different group of people come in, the second-order consultants. These people did not pick Agile because it was a great way to develop software. They picked it because it was easy to sell, or easy to get them a job, or the path of least resistance.  In a few cases, the very same people who I found arguing with me a few years prior that Agile or XP could never work, would not work here, or was “a fad”, were now active proponents of the same words they had decried. When I asked them to explain what changed their mind, or how they reconciled the two, I was never really able to get an answer.  If that sounds critical, that is because I mean to be critical.

And it got worse.

Because now Agile was popular. 

Remember those executives who wanted to make up deadlines that were firm? That wanted to define everything for the project (and change their minds as often as they wanted) and hit that deadline made up before we even decided what to build? Those executives had, at first, complained that Agile didn’t Scale.

The new crop of consultants came in and said:

• “Scaling? Yeah, sure, we can do that.”
• “Hard deadlines? Sure, we got that. The organization just needs to make and keep commitments.”
• “Project planning? Sure, we got that. We got velocity.”
• “Continuous flow got you down? We can do hardening sprints. In fact, we can do some design sprints, some code sprints – better add some requirement sprints into our Methodology.”
• “Architecture? Of course. A good scaling effort needs an architecture team, along with enterprise governance. Along with (name spurious idea). It’s all here in our Big Box O’ Agile (™) product. You can buy it off the shelf and install it, I mean, um … hire us to help lead your Enterprise Agile Transformation at Scale (™).”

Ward Cunningham’s C2 Wiki got a “Big Agile Up Front” anti-pattern added to its list.

Someone explained to me that there was a market for a “big box of agile” that allowed one to have the Agiles and Eat waterfall too, and that if there was demand, someone would come to sell it. And so, the law of raspberry jam applied.

We wanted to transform the world of work, and instead we got Sprints, Standups, Stories, and Certifications.

Then we lost the technical people

DevOps and Code-as-Craft began as technical offshoots from Agile Software, and over the years, the Agile Conference had less and less technical material. This was, in some ways, a never-ending doom loop. The conference had less technical material, so less technical people attended. The organizers noticed the number of technical people had decreased, so they adjusted the track sizes based on attendance, so the conference was less attractive to technical people … and on it went. I had a long, extended conversation with Paul Hammond about this after I had been a technical track chair/co-chair for several years and he was incoming to lead the program for the conference. To his credit, he listened and invested a great deal of time in the conversation. His logic trapped him in the doom loop, but I could see how he got there.

It wasn’t just the Agile Conference. People with subject matter expertise would call themselves programmer, or graphic designer, or tester, or database admin, perhaps. On the other hand, “Agile Coaches” were often expected to Coach anyone in any role. This made “Coach” a meta-role. Given that only a handful of people had the expertise to Coach all the technical roles, and even then, only the ones as an employee of one specific company, Coaching became a career and skill that was away from the code.

Thus, to make more money in the Agiles, one gets away from the technical work, and becomes a Scum Master, or a Coach, or a Manager.

We’ve seen this pattern before, haven’t we?

We may have come full circle.

 

The Agile Alliance Merges with the PMI

Then, this week, the big news. The Agile Alliance “joined” the PMI, something between being acquired, merging, and partnering. The legal structure isn’t quite clear. The press release is full of statements like this:

The future of project management—centered on project success driven by delivered value—cannot rely on rigid distinctions between delivery approaches. The integration of Agile Alliance within PMI further solidifies PMI’s leadership in transforming the project profession. PMI Agile Alliance will benefit from PMI’s global reach and resources, while PMI members will gain expanded access to Agile thought leadership, tools, and resources, enhancing their professional development and enabling greater project success.

I have no idea what that means.

At least, I don’t know what it means for AA members. Do they become members of the PMI? … Maybe? It seems most likely that the PMI would get a list of names to add to invitations, to try to get to join local chapters, to invite to conferences, and vice versa. 

The one thing we do know, though, is the very organization the manifesto was created to critique is now the stewarded by the organization it meant to reform.

There is a possible explanation for this, something called a “Hegelian Synthesis.” According to Hegel, after you have idea one (“thesis”), then an opposing ideas (“antithesis”), it is possible to blend the two to create something new. This is a fancy way of saying the reformers did their job, pulling the PMI out of stable/predictable/repeatable land and into the realm of continuous refinement and collaboration – but the melding of the two made something even better.  Indeed, when I was at Agile2018, I asked the program manager of Carnegie Mellon’s Software Engineering Institute (SEI) why they were there, as the Capability Maturity Model Integrated (CMMI) seems so oppositional in values to the Agile Manifesto. He replied exactly that – that the Hegelian Synthesis of the two ideas created a third option that was better.

I have one problem with his statement.

It’s crap.

He is wrong.

The Agile Manifesto was opposed to something. Something real. The people-people who did not understand software had taken over software long ago, in the 1980’s and 1990’s. The manifesto took ground from them.

For some reason, the institution dedicated to advancing that mission seems to have given the ground back.

Let’s look at why.

 

The Why of the Merger

The rumormill on linkedin seems to be that the Agile Alliance has funding problems, with a goal to fund 18 major initiatives, including Ethics for Agile Coaches, Academic Support for Agile in the university, sustainability and business agility. Despite this, when I dig into the financials, I see that the conference is becoming more expensive and bringing in less revenue. Here’s the expense breakdown for 2021-April of 2024:

And the income for the same period:

That’s the problem. Expenses exceed income. Mostly conference and payroll have gone up.

One could argue that to have a world-class conference, one has to compensate speakers, that speakers are expensive, and there is some upper-limit to how much to charge for the conference, thus the Agile-Conference-Business is not sustainable. But if that were the case, wouldn’t all conferences as a businesses be at risk? And indeed, thanks to the 2020 global pandemic, many conference companies are struggling — yet some, such as Trendig, which runs Agile Testing Days, seem to be making it work.

There are solutions to this, most notably zero-based budgeting. Overhaul the events, from top to bottom. Run basic IT services – a website, membership, access to some web based databases. Have the board meet over zoom. Run the conference profitably.

That last one may be the rub. It is possible the conference has multi-year contracts to run in locations where it cannot run profitably. That seems the most logical cause of the problem. Perhaps the conference planned for growth that did not emerge, thus reserving expensive space at the largest conference venues in the world for years in advance, creating a liability problem. Thus, getting a big brother to bail out of future liability – perhaps even convert the conferences to profitable by expanding the base, becomes arguably a matter of survival.

If that were true, it sounds like a heck of a case study in the risks of Big Design Up Front, and a wonderful conversation for us to have as a community. There were other options, such as negotiating with the venues and having much smaller, more elite events, or even bankruptcy and recovery.

Instead of having the tough conversation, the Agile Alliance joined the PMI. 

It was fun while it lasted, friends. We tried. We had some fun.

The people-people took over.

Again, perhaps, they might argue, by reaching mainstream acceptance to join the PMI, we have in fact moved the needle a little. I’ll give the tiniest bit of credit there. I have to respect that the people at the “XP conference” and the “Agile Universe” were a ragtag band of ruffians that weren’t having much impact, while the AA did become a force to be taken seriously.

Sadly, the “Stories, Standups, and Sprints” don’t help much in a traditional project management paradigm.

When I was in graduate school at the turn of the century, we had one lecture when the president of the university visited our project management class. Someone asked why try to build something great when inevitably, everything we build will crash back into the sand. He replied: “Build it anyway.”

I’m glad we tried. I was a part of it. If you were too, well, we’re not done yet.

The opening line of the agile manifesto is: “We are uncovering better ways of developing software by doing it and helping others do it.”

In C.S. Lewis’s speech “the inner ring“, he describes all I had ever hoped to be as an engineer:

If in your working hours you make the work your end, you will presently find yourself all unawares inside the only circle in your profession that really matters. You will be one of the sound craftsmen, and other sound craftsmen will know it. This group of craftsmen will by no means coincide with the Inner Ring or the Important People or the People in the Know. It will not shape that professional policy or work up that professional influence which fights for the profession as a whole against the public: nor will it lead to those periodic scandals and crises which the Inner Ring produces. But it will do those things which that profession exists to do and will in the long run be responsible for all the respect which that profession in fact enjoys and which the speeches and advertisements cannot maintain.

It’s time to get back to work[4].

Footnotes

[1] “Beat up the waterfall” is an old and tired game, but if you need a quick summary, I think it is fair to say that the Waterfall model focuses on getting everything right up front, including the schedule. Often, the schedule is done before you’ve figured out what it is you are going to build! Then you assume everything will work right the first time (what Demacro and Lister call the “nano percent date“) and try to lock everything down. This also means you define the software at the beginning before you have explored what is possible and figured out what won’t work. That is the point of wicked problem – that you often don’t really understand the problem until after the first attempt to solve it.

[2] Why and what actually worked is beyond the scope of this article, but if I had to give it a nutshell, I would say iterative development, collaborating with customers, anticipating change, good adherence to technical practice and skill, including test driven development/refactoring/craft, holistic and emergent approaches to testing.

[3] I should probably put Weinberg’s book Rethinking System Analysis and Design in the list of books that led up to Agile, along with the Harvard Business Review article The New New Product Development Game. The main blog post was long enough, and, honestly, I don’t see The Toyota Production System as a precursor to Agile. After my exhaustive literature review, I find it more likely than not that TPS was discovered by most people after-the-fact as explaining some of the meta-physics for why Agile/XP works. That is different than people discovering TPS and applying it to software to derive Agile and XP. 

[4] Sadly, this article had to pre-empt a three-part series about AI in software. That series ends on a similar line: The work of developing software is noble. Lets, you know, do that.