I've been thinking a lot lately about Intentional Emergence in the defense IT enterprise. I advocate for emergent approaches like open source because I don't think buckets of money can continue to make up for central planning's failures, because it's frustrating to work in an environment where good ideas languish, and, if I'm honest, because I think it would just be a lot more fun if the enterprise felt more like the web. Lately however, I've become convinced that facilitating emergence with policy and practice isn't just nice to have, it's absolutely necessary. In this post I'll try to explain why. The argument goes like this:
People cooperate to make society function. In capitalist societies we do this by combining our efforts into companies and other organizations that are centrally planned and centrally controlled. However, beyond a certain scale central planning falls down. It gets unwieldy. So, instead of having one giant company that plans and controls everything (or a socialist government standing in for it), these independently planned entities interact in a market that isn't centrally controlled, it's emergent. The market has some simple rules and out of it emerge both prices and patterns of behavior.
Long before Complexity Theory, Social Physics, Chaos and all its about talk of basins of attraction and stuff like that, or before any of the modern offspring of Cybernetics came along, Friedrich Hayak wrote The Road to Serfdom. In it he made the argument that centrally planned economies would always result in authoritarianism and the failure of democracy (or more generally, freedom). The planned economies of Nazi Germany and the Soviet Union, though starting at opposite ends of the political spectrum, kindly reinforced his point by sliding into near mirror-image fascist states. The planner's intentions made no difference and the Gulag filled up with well-intentioned old Bolsheviks.
So, what does this have to do with Defense Enterprise IT?
The Defense Enterprise is huge. Whether in terms of number of participants, dollars spent, assets, or any other measure, it's massive. In fact, it has never really been a single enterprise if the definition "enterprise" includes the span of control of a single controlling entity. It has always been broken up into near-independent organizations with different missions, cultures, financials, and etc. to make each sub-enterprise span of control more reasonable. It's scope and activities make it a lot like an economy filled with at least partially independent organizations.
Inside that super enterprise, individual IT systems (and system acquisitions) proceeded in near isolation. They had external requirements and policy to follow, but those changed slowly and all of the money, authority, and accountability for a given effort intersected at a single program office. The program office might have a devilishly complex task to accomplish, but at least it (mostly) controlled its own destiny.
Of course, I'm not describing a perfect world. Many of these programs were themselves very large by the standards of commercial enterprise. As program size and the necessary controls that go with size increased, a greater portion of their overall effort was consumed with controls instead of code. As a result acquisition cycle times and overhead have increased dramatically in the last thirty years, and so have failures. Also, given the lengths of time involved, by the time a system was delivered it often turned out that no one really cared about it anymore.
This has been true from the very beginning of the DoD's IT enterprise. In fact the precedent was set by one of the earliest defense IT systems, the Semi Automatic Ground Environment (SAGE). It was a massive undertaking that required the simultaneous invention of the first multi-user digital computer (whirlwind) and a continent-wide radar system to go with it. Its completion was lauded as a testament to the newly devised methods of systems engineering, processes we are still living with today. What's talked about less is that it took so long to deliver that it never served its intended purpose. By the time it was operational, strategic bombers had been supplanted by ICBM's and the need to vector fighter defense on a continent-wide scale had gone away. SAGE became a useful but very expensive air traffic control system until the early 80's when it was finally retired.
So, for many years we've had programs of such massive size that they have been pressing up against (and frequently exceeding) the limits of our ability to centrally plan and manage them. With that size has come a decline in the ratio of value achieved for the money spent while failure rates have increased. But, we could live with that because we had the richest economy on the planet and now and again, through sheer force of will, all of that systems engineering managed to spit out a useful system. Systems that, if their time had passed, could at least be used for some other related and useful purpose.
More recently however, the DoD has started a migration to NetCentricity. NetCentricity is a revolution in the use of information in warfare. If its proponents are right it will mark a discontinuity in warfare as big as the advent of armor or air warfare. Strictly speaking NetCentricity isn't about technology, it is about the power of networked organizations, rapid information flow, and self-synchronization to serve as force multipliers and enhance the operational art of maneuver. Technology is a necessary enabler though and the important key to this discussion is the fact that NetCentricity requires systems all across the DoD super-enterprise to share information. For people to have shared awareness and self synchronization in the cognitive domain, their information systems must be connected.
The Software Engineering Institute at Carnegie Mellon University recently published a paper on ultra large systems (pdf) that goes into great detail about the challenges that come with this evolution - from thousands of independent large (or very large) systems into a single ultra large system fabric. I think one of its most important points is that Ultra Large Systems won't be managed as a single effort, but instead will consist of operationally independent but connected nodes (see page 11). What this is saying without saying it, is that the era of pretending that the DoD IT Enterprise is a single enterprise is over. An enterprise is by definition the people, systems and processes that are under a single span of control. This statement is an acknowledgement that in a NetCentric ULS-operating DoD this can no longer be the case. So, if it's not an enterprise, what is it then?
Let's ask the question another way. What happens when hundreds of programs that were already so large that they were teetering at the limits of central planning suddenly get connected to each other? When huge programs that at least had nice clean edge conditions suddenly find themselves having to coordinate across boundaries that were once impermeable and fixed? Do you get an Ultra Large System that is too big to plan? Or do you still have hundreds of only Really Big systems that just tipped over?
Either way it's a communication storm where suddenly every program manager is taxed with external communications about rapidly evolving boundary conditions (e.g. protocols, taxonomies, semantics, technologies, etc.). The speed at which those boundary conditions change, combined with the effort inherent in all that additional communication is the tipping point into a cybernetic stall. The DoD has become the proud owner of an ultra large system that is simply too large to survive and thrive as a centrally planned entity, but they are still trying to plan it.
Today the Army struggles to respond to this problem (within the still-limited scope of the Army enterprise) with its Software Blocking process. This is an effort to align all of the Army's related systems into a single release schedule (i.e. blocks). Essentially it attempts to abstract the planning process so that it can span many large programs. But the reality is, what was once a hard problem is becoming an intractable one and while software blocking can help improve interoperability across the Army's systems, it does it at the expense of speed. To stay in synch, all of the systems in a block have to keep pace with the slowest runner. A better plan would probably be to adopt emergence at the scale of each of those systems like Amazon has, with a combination of architecture, organization, and incentives.
In a more general sense, Software Blocking is part of a broader organizational reaction to the stall. It's a reaction that seeks to exert more and more control in the face of uncertainty. Acquisition processes add steps to wring risk from the process, SEI issues updates to CMM that require more process documentation and evaluation, and old timers wax nostalgically about the days "when people would just take orders and do what they were told." What those old timers are really nostalgic for are the days of clear lines of control in the hierarchy. As the hierarchy is replaced by a network both the clarity of accountability and the control that goes with it are lost.
In the military's culture of order giving and taking, pushing for even more control in the face of failing programs is only natural, but it isn't going to fix the problem. It's like trying to sidestep Heisenburg's uncertainty principle by squinting really hard when you look at an electron. It's a fools errand, a Paradox of Control. Because it turns out that once you're in a cybernetic stall, trying to de-risk with more planning and control will just make things worse. By the very nature of software, the slower you build it, and the more complete you try to make your plans before you start, the more real risk you create. Does anyone believe that the JCIDS process is enhancing warfighter effectiveness?
The web, unlike the expanding DoD super enterprise, has always been emergent. Since it was never really under anyone's control it demonstrated emergence from the beginning - It didn't need anyone to let go of control for it to get that way. As a result it has experienced a great deal of innovation - innovations that usually start small and in large quantities and then culled as they grow into a power law curve. The exact technologies of the web, or even the exact attributes that comprise its emergent eco-system, might not be the right ones to make the DoD emergent. After all, the DoD has different operational and environmental conditions. However, we would be dumb not to look at the web for ideas for the defense IT enterprise.
This is a conversation with consequences. Despite how it sounds, we're not just bantering about esoteric theory. Let's continue for just a moment with the analogy between the defense IT enterprise and the broader economic activity of the state. From The Road to Serfdom:
“It is no exaggeration to say that if we had had to rely on conscious central planning for the growth of our industrial system, it would never have reached the degree of differentiation, complexity, and flexibility it has attained. Compared with this method of solving the economic problem by means of de-centralization plus automatic coordination, the more obvious method of central direction is incredibly clumsy, primitive, and limited in scope. That the division of labor has reached the extent which makes modern civilization possible we owe to the fact that it did not have to be consciously created but that man tumbled on a method by which the division of labor could be extended far beyond the limits within which it could have been planned. Any further growth of its complexity, therefore, far from making central direction more necessary, makes it more important than ever that we should use a technique which does not depend on conscious control.” (Emphasis added by me).
China had its Deng Xiaoping and Russia had Beria. Both of them saw (at different times) that central planning was leading to economic disaster. Xiaoping started China's economy on the road to growth by permitting markets and encouraging the emergence that goes with them. Beria on the other hand didn't survive the post-Stalin power shake out and as a result the Soviet Union just kept five year planning itself into oblivion.
The Defense IT Enterprise faces a similar crossroads. The scale of the super enterprise (and the Ultra Large System landscape inside it) exceeds the limits of central planning, yet the DoD is still trying to plan itself to success. The result is failing programs, a focus only on the big stuff, and obvious (and growing) digital serfdom for our troops on the ground. In general they are less connected and more constrained in their use and development of IT than their third world adversaries. And to add insult to injury, when they show up at disaster relief sites, they often find that the NGO's have better situational awareness on the ground. Meanwhile the systems engineering machine keeps grinding out yesterwar's systems.
This begs the question, who is the DoD's Deng Xiaoping? The one that will break it out of the Paradox of Control and recognize that along with the planned "enterprise-like" components, there must be vast spaces of facilitated emergence? Most people doubt Mao meant it when he said to let a hundred flowers blossom, but the Defense IT establishment needs to embrace the notion in its policy and create an ecosystem that supports emergent behaviors, or continue to watch large scale system development falter, money get wasted, and our troops fight in digital squalor.
