May 25, 2023 16 min read Podcast

ProdFund 1.2: Intro to Iteration

Iterative & Incremental Development got its start in classic early 20th century manufacturing and evolved through pioneering NASA projects into an important precursor of how we work today.

Product Fundamentals is a podcast dedicated to spreading the core knowledge software product people need in order to succeed. Season 1 is structured as a history of how we make software.

Hot on the heels of 1.1: Chasing Waterfalls, here is season 1, episode 2 of the Product Fundamentals podcast. Our history of software development reaches back a bit deeper into the past, tracking how ideas about continuous improvement worked their way from industrial manufacturing through NASA and on to the early modern software industry.

The audio is embedded below, and the episode transcript follows.

You can also find this episode of Product Fundamentals on the show website, on YouTube, and through all the usual podcast services.

Transcript

Hello friends, and welcome back to the Product Fundamentals podcast, episode 2: Intro to Iteration.

In this season, we are tracking the evolution of how we came to make software in the weird way we do, from the earliest origins of our methods, through to today.

Intro

Last time, we discussed the Waterfall methodology, including its lasting articulation by aerospace executive Winston Royce in a 1970 paper. But that process-heavy, top-down, one-shot approach to making software wasn’t the only game in town, even before 1970.

In this episode, we’ll uncover the early days of incremental and iterative development – which was really the first precursor of the Agile development patterns that so many organizations espouse today. We’ll also talk through how the “cross-functional teams” that are so ubiquitous in software companies came to be.

This path will take us from statisticians studying randomness, through the earliest days of NASA, and on to nuclear submarines and the space shuttle, with a quick stop by Max Weber along the way. Let’s get into it.

Quality analysis

Iterative and incremental development, or I.I.D., has its roots in the pre-software world of statistical analysis and industrial product improvement.

Shewhart

In the 1910’s and 1920s, an American physicist and statistician at Western Electric named Walter Shewhart began work to improve the quality of manufactured phone system components (source). When Western Electric partnered with AT&T to found one of the first modern industrial research parks, Walter Shewart would become one of the original engineers of Bell Telephone Laboratories.

Before Shewhart, manufacturers might test their finished products before delivery to a customer, but those inspectors made only a binary evaluation of the final product as acceptable or not. Shewhart recognized that random variation would creep into any production process, resulting in a distribution of nearly any characteristic of a manufactured good: size, weight, electrical conductivity, and so on would all vary from unit to unit. As the complexity of manufactured products increased, and the number of each item produced increased, the number of dimensions that any unit of a product could vary on compared to the ideal would also increase. All this opportunity for unwanted variation meant a growing risk of the final product being unacceptable for the customer.

Shewhart drew two linked conclusions: first, companies should set specific tolerance thresholds for quality, and use rigorous statistical methods to sample and test the variation in their manufacturing output – no more eye-balling it or using subjective judgment to evaluate quality.

Second, to reduce defects, manufacturers should implement quality control at every step of the manufacturing process. If, under the old model, Western Electric would only have evaluated a finished telephone, with Shewhart’s approach, they would now set quality standards for each component. They would measure the wire, the diaphragms and magnets in the receiver, the size of the phone body, and so on, at each stage of manufacturing. And at each stage, they’d use statistical methods to know how many units to sample and what level of variation between units was acceptable to ensure the overall batch was high enough quality to send to a customer.

Deming

Shewhart published two books, as well as other academic papers in the field of statistics, which drew the attention of William Edwards Deming, another American statistician and engineer. Deming built on one obvious consequence of Shewhart’s work: by measuring at each step of the production process, it became much cheaper to identify and fix problem areas. Deming packaged Shewhart’s approach into a four-step cycle of product improvement that he called Plan, Do, Study, Act.

In this scheme, one plans a change to a process, does the change, studies the effect of the change, and acts by either adopting the change or rejecting it. Each of the changes advanced in this PDSA cycle were meant to be small iterations very limited in scope. For manufacturers, implementing small improvements to an existing production line proved much lower-risk and much more cost-efficient than building an entirely new production line in order to improve product quality.

Shewhart and Deming made the case that issues in the quality of goods produced should be treated as a cost. Even if the factory was able to meet its quota, the opportunity cost of producing sub-standard parts was a waste, and should be treated as such. By giving manufacturers the methodological tools to identify and improve processes cheaply, they made problems quantifiable, and the improvements to processes quantifiable, opening up a much more incremental and iterative process to improving manufacturing.

Deming was recruited by Douglas MacArthur to come to US-occupied Japan in 1947 to use his statistical skills to help evaluate and rebuild the shattered Japanese economy. Deming apparently took quite a liking to Japan, returning to the country at least four times in the 1950s to act as a lecturer and consultant for business managers on continual process improvement. His methodology, along with similar work from engineer and consultant Joseph Juran, would go on to have enormous impact on Japan, accelerating its post-war economic miracle, and planting seeds that will definitely come up again when we get to Lean manufacturing later.

So, Shewhart and Deming have laid foundations for rapid cycle iterations to generate small improvements in industrial processes. How does this get to software?

IID at NASA

In the 1950s, NASA and US Air Force engineers picked up these threads while working on an experimental hypersonic rocket-powered airplane, the X-15. This was a small single-seat craft that would attach under the wing of a B-52 bomber, and would be taken by the bomber up to its cruising altitude. Then the X-15 would drop from the bomber, and its own rocket engines would take over, propelling the small rocket plane up to the edge of space. Its development was a landmark in using incremental and iterative development processes on a major high tech project.

In some ways, a military rocket plane is a surprising place to find a more flexible development approach. But seen from another angle, it makes total sense that this kind of NASA project would embrace rapid iteration: this is an experimental aircraft! These engineers are trying to do things that have never been done before, with new materials and new science, so there is no playbook to follow. They can’t write out exhaustive documentation for all the work ahead of time, because no one alive yet knows how to achieve what they’re trying to achieve. And they’re doing this work between 1955 and 1968, the most dangerous years of the Cold War and the peak of the Space Race. There’s a strong motivator to move fast before all the pieces are totally understood.

So, the engineers on the X-15 project adopt an approach of constant iteration across many components. They build three airplanes, each of which is significantly different, and each of which evolves over time with parts getting switched out in order to test new approaches and technologies. Fun fact: the X-15 set a world record for the fastest winged, powered, crewed aircraft in 1967, when pilot William Knight flew it at 7,274 kilometers per hour. That’s more than 2 kilometers per second, and that record still stands today.

While the X-15 didn’t have software to speak of, some of the engineers on the project were transferred to work on Project Mercury, the United States’ first manned space missions. The Mercury project, which ran from 1959 to 1962, did have significant software, running on multiple mainframes on the ground. There would not be computers on the spacecraft until the Project Gemini flights of 1964 to 1966.

Fun fact: The lead contractor to supply Mercury’s ground-based control and tracking systems was Western Electric, where Walter Shewhart had worked, while the lead computer systems contractor was IBM, whose early lead in mainframe computing we discussed last episode.

Rather than investing a large amount of time and effort into designing and documenting an ideal system, and then implementing that grand plan in one big push, the Mercury team adapted their more experimental approach to hardware and applied it to software development.

They used very tight working sessions to come up with new versions of software, time-boxing each iteration to as little as half a day of work. Given the very basic tooling of the time, these iterations were thus very small in terms of new functionality delivered. The Mercury programmers also used test-driven development, meaning they wrote tests that their code should satisfy before they wrote the software itself. While I was surprised to learn the technique existed as early as 1959, given the life-or-death stakes for the astronauts and the very public nature of any failures, perhaps this shouldn't be surprising at all. In any case, using iterations measured in hours, which start with a test and end with working code, is a radically different approach than the single- or double-pass documentation-driven way of building code that Winston Royce would describe as the industry standard in 1970.

This work, informed as it was by earlier innovations in industrial quality improvement and recent experience cutting-edge hardware experimentation, made the Mercury project’s software the first ever major iterative and incremental development software project.

Matrix Management and the cross-functional team

The scope and complexity of the US’s technical challenges during the Space Race led to another important innovation for how we make software today: it planted the seeds for the modern cross-functional team.

Before the 1960s, almost all large companies followed German anthropologist Max Weber’s model of an ideal bureaucracy: all power resided at the top of the organization, and responsibilities were then delegated down strict reporting lines through layers of management until tasks reached individual workers. Everyone had clear job responsibilities, everyone had exactly one boss, every layer of the organization had a clear scope, and so on. The company was a highly-regimented, cleanly structured machine.

In this period, the “functional” organizational model predominated. This means that workers and managers are organized into functional groupings, like Marketing, Engineering, and Sales. Engineers report to engineering managers who report to engineering vice presidents, marketing associates report to marketing managers who report to marketing vice presidents, and so on.

In a company organized around functions, there’s little reason for functions to cooperate at the lower ranks. Instead, projects would rely on executive alignment and executive sponsorship from leaders of the different functions, who would then push work on a project down to the lower ranks when that function’s contribution was needed.

To this day, many companies are primarily organized around function. After all, there’s an intuitive rationality to each person being managed and evaluated by someone else with the same functional skill. However, the system is also prone to move very slowly, and no large project can succeed without support from multiple executives across the various functions.

One alternative to the functional model is a project structure, often called a team structure. This is the structure of all small businesses, many franchises, and so on. In structure, workers are organized around the project to be done. There is no “marketing department” to speak of, but there is someone on the team who does the marketing work. This also has an intuitive appeal: it puts the emphasis on getting the project done. But as an organization grows, it’s not obvious how to scale project organizations.

Polaris

A promising example of reconciling these models emerged in the 1960s in the form of the US Navy’s Fleet Ballistic Missile program and its eventual product, the Polaris submarine-launched missile. In this project, which began in 1956 and was propelled to top priority by the 1957 launch of the Soviet Sputnik satellite, the Navy took a gamble on a new organizational structure. It spun up a new temporary organization outside of the normal chain of command, called the Special Projects Office.

In contrast to the usual practice, the Special Projects Office organized its thousands of contributors to each have two bosses to satisfy. One boss was the technical expert, who demanded technical excellence from each contributor. The other boss was the project manager, who was responsible for ensuring that work was delivered on time, and fit the “big picture” needs of the overall project. As a result, each worker had two groups of relevant colleagues – their functional team, such as other engineers, and their project team, which included experts from many disciplines.

This approach had several apparent benefits. One was that it helped ensure that all of the work delivered was both technically excellent, and well-aligned with the concrete needs of the project. The other benefit was that it made specialists more portable between projects. An engineer could be assigned to the overall missile program for many years, keeping the same technical leader for years, while smoothly transitioning between different project managers over time as individual projects were completed. In a world of expensive government contracting with large overheads, this ability to keep a smaller number of experts engaged for a longer period of time generated significant cost savings.

The Polaris missile project was an enormously important undertaking for the US military, as the country understood itself to be at a significant nuclear disadvantage to the Soviet Union. There was a perceived “missile gap” between the superpowers, which John F. Kennedy emphasized in his presidential campaign to paint Eisenhower as weak in defense. In reality, there was no missile gap, but perception was everything, and when the first Polaris missile test occurred in 1960, on budget and two years ahead of schedule, the project was seen as a triumph.

As a result, the project management techniques used by the Special Projects Office were rocketed into high popularity, including the “Program evaluation and review technique,” or PERT, which is one of the foundational concepts of modern project management – though we’ll save that for another time.

This two-boss model spread throughout the aerospace industry in the 1950s and 1960s, and was used heavily by NASA throughout the Space Race. Given the tight linkages between aerospace and the nascent software industry, it was inevitable that it would work its way into software companies as well.

DEC

The technology company most associated with this model was the Digital Equipment Corporation, or DEC, which also went simply by “Digital.”

Though now defunct, DEC was a major producer of minicomputers, which were computers smaller and cheaper than mainframes, but more expensive than the “microcomputers” that would evolve into our modern sense of personal computers. These minicomputers were bought by companies to perform tasks like computer-aided design and seismic data analysis for oil exploration.

DEC implemented the two-boss style of management, which came to be called “Matrix management,” though DEC leaders apparently did not like the term. This structure was called “matrix management” because when visualized, each worker could be seen as a cell in a table. They belong to a column which indicates their function (such as being an engineer or a salesperson), and a row, which represents the project or product to which they are assigned. That table is a matrix. Each function has a function leader, and each project has a project leader.

During the 1970s and 1980s, matrix management spread rapidly through corporate America, with the financial success of DEC and the general sense of novelty and innovation that matrix management seemed to exude drawing attention.

By the late 1980s, though, the limitations of this matrix model in corporate settings were showing. First, most companies did not have resources at the scale of the military, where the structure had first succeeded. They couldn’t afford the overhead of having two parallel management tracks. Indeed, even at the Navy, the Polaris project was understood to be a massive drain, setting back other projects as it pulled money and talent to the shiny new project. Its missile had been delivered on-budget, but that was in part because the project’s budget was practically limitless.

Companies also found matrix management tended to be slow and conflictual, as workers balanced directives and expectations from two different bosses, and those bosses competed for influence and ownership. While functional organizations suffered bureaucratic challenges from getting different functional leaders aligned, matrix management also had to contend with getting project leaders aligned as well.

The dream of matrix management was for project managers to pull in high-skill experts from various functions as needed to form a skilled team, and then release those experts back to the pool of talent living in the functional orgs when they were no longer needed. Few existing companies were able to adopt their existing workforce and bureaucratic systems to this new model, though; even matrix management’s advocates describe it as requiring more investment in soft skills and internal diplomacy than other models.

DEC itself stagnated in the late 1980s, missing the personal computer wave and limping along as a shell of its former self until a final acquisition by Compaq in 1998. By this time, the management style that it had been so tightly associated with was largely discredited in the wider economy.

Still, there was clearly an attractive intuitive kernel to cross-functional teams. Parts of it just made too much sense to be totally discarded.

As companies grew in scale and complexity, the new management structure that rose to prominence was the Division. If you work at a large company today, you almost certainly work in a division: Consumer or Enterprise, APAC or EMEA, New Ventures, and so on.

Divisions have supplanted functions as the dominant pattern; functional orgs are often now subordinate to the division, so there’s an engineering leader for Enterprise and an engineering leader for Consumer. While these divisions are broader in scope than the projects of a matrix organization, they do put the business need ahead of the function. Instead of making each workers the servant of two masters at the same time, most organizations today put one master clearly before the other.

The benefits of real cross-functional teams would not be lost entirely, though. We’ll see this pattern return when we get to the Agile Manifesto in an upcoming episode.

IBM and IID

So, government work on missile programs gave us organizational structures that we still work with today. But the incremental and iterative way of building software would also make the move from government aerospace projects and into software.

After the Mercury program, with its test-driven development and half-day iteration cycles, some of NASA’s software engineering went to IBM and became leaders of IBM’s Federal Systems Division, which would go on to be a major supplier of software to the US government. This IBM division became one of the biggest corporate practitioners of an iterative software development process throughout the 1970s.

The first major IBM project to apply iterative and incremental development was a serious one: in 1972, IBM won the contract to write the command-and-control system for the Ohio-class nuclear submarines, which carries intercontinental ballistic missiles. The IBM team broke the project into four time-boxed iterations of six months each, and eventually delivered a million lines of code on time.

IBM’s FSD team was also responsible for writing the avionics system software for the space shuttle. That’s the main software for flying the shuttle, responsible for all of the shuttle’s flight systems. Between 1977 and 1980, IBM engineers went through 17 iterations, deliberately structuring the project to be ready for changing requirements as the incredibly complicated shuttle project evolved.

IID and modern development

Compared to the Waterfall methodology we described before, this is a much different approach. But it’s important to note that iterations in this period are still a far cry from today’s two-week sprint, Mercury notwithstanding. Most large software projects were still enormous undertakings, often done on contract for an external client, with a well-defined and documented final state to work toward. Teams are still mostly working on terminals. The iterations created checkpoints in a big project that could be used to derisk bets and set branch points to fall back to if a project took a wrong path.

These iterations weren’t meant to just chunk time. Each iteration was expected to deliver some concrete value: ideally that value would satisfy some specific customer requirements, but if not, it should either provide a measurable benefit to the user, or at least generate some specific new learning.

But despite its presence since the early days of software, it’s important to recognize how different the incremental and iterative development of the 1960’s and 70’s was from its descendants today. Iterations may have targeted distinct blocks of value, but that value generally couldn’t be actually delivered to the customer. Software engineers were still building monolithic software that was tightly coupled with hardware. These iterations were not being sent out to consumers over the Internet through a continuous deployment process. The iterations were generally milestones toward a single package delivered to the enterprise- or government-scale customer once, at the end of a long contract.

IID vs Waterfall

Still, incremental development had significant advantages over Waterfall that its advocates pointed to.

The first was avoiding wasted time and effort, most often because requirements in large software projects often change.
Another was that iterations created opportunities for customer or user feedback, which could identify issues early in the project.
The most interesting one to me, though, is the issue of complexity. Software systems are simply too complicated for humans to handle. We can’t realistically design them with firm plans from the outset. Harlan Mills, a research fellow at IBM, wrote in 1976 that,

“The danger in the sequence [waterfall approach] is that the project moves from being grand to being grandiose, and exceeds our human intellectual capabilities for management and control.”

I draw your attention back to Moore’s law. With computing power doubling every year, the complexity of what is just-barely-possible rises at an exponential rate. But there are also countervailing forces of a sort against that complexity: abstractions and utilities, as well as higher-level programming languages, are helpfully hiding some of that complexity, providing shortcuts around it. This is why a human being with the same basic intelligence can write software to handle far more complicated problems in much less time than our forebears could, despite the complexity creep brought on by having tens of millions of times more transistors on a chip in 2023 than in 1970.

There’s a peaceful arms race of sorts between the hardware developers, delivering breakthroughs that exponentiate the possible complexity of the system, and the abstraction developers, working to tame complexity so that human minds can still make sense of it and leverage the new raw capabilities of the hardware.

Before the rise of personal computing, when most software was still written as part of large monolithic projects, IID’s big advantage was that it positioned teams to be more successful in a world of human limitations. Customers can’t really articulate their needs; designers can’t really juggle all the pieces in their minds at once; no one can predict how technology will change. IID was the first conscious attempt to find a way to work that accommodated the inevitability of those shortcomings, rather than trying to brute force its way through them.

Despite some increasing formalization of iterative and incremental development into a variety of project management schemes during the early 1980s, though, IID remained the significant underdog to Waterfall from the 1950s through at least the 1980s. The Brute Force approach to hard software problems was still on top.

In our next episode, we’ll take a step back from the Waterfall vs. IID divide in order to dive into how we plan and measure success at work, tracing the path that took us from the Industrial Revolution through to the European emigres who fled Nazis and communists and went on to give the world… OKRs.

As always, your comments and feedback on this episode are very welcome. You can find an episode transcript and links to sources at prodfund.com. And if you like this series, and want to hear more, support this project by sharing it with someone you think would enjoy it too.

Thank you very much for listening.