Limn This

The Cybernetic Stall - Emergence isn't Optional

I've been thinking a lot lately about Intentional Emergence in the defense IT enterprise. I advocate for emergent approaches like open source because I don't think buckets of money can continue to make up for central planning's failures, because it's frustrating to work in an environment where good ideas languish, and, if I'm honest, because I think it would just be a lot more fun if the enterprise felt more like the web. Lately however, I've become convinced that facilitating emergence with policy and practice isn't just nice to have, it's absolutely necessary. In this post I'll try to explain why. The argument goes like this:

People cooperate to make society function. In capitalist societies we do this by combining our efforts into companies and other organizations that are centrally planned and centrally controlled. However, beyond a certain scale central planning falls down. It gets unwieldy. So, instead of having one giant company that plans and controls everything (or a socialist government standing in for it), these independently planned entities interact in a market that isn't centrally controlled, it's emergent. The market has some simple rules and out of it emerge both prices and patterns of behavior.

Long before Complexity Theory, Social Physics, Chaos and all its about talk of basins of attraction and stuff like that, or before any of the modern offspring of Cybernetics came along, Friedrich Hayak wrote The Road to Serfdom. In it he made the argument that centrally planned economies would always result in authoritarianism and the failure of democracy (or more generally, freedom). The planned economies of Nazi Germany and the Soviet Union, though starting at opposite ends of the political spectrum, kindly reinforced his point by sliding into near mirror-image fascist states. The planner's intentions made no difference and the Gulag filled up with well-intentioned old Bolsheviks.

So, what does this have to do with Defense Enterprise IT?

The Defense Enterprise is huge. Whether in terms of number of participants, dollars spent, assets, or any other measure, it's massive. In fact, it has never really been a single enterprise if the definition "enterprise" includes the span of control of a single controlling entity. It has always been broken up into near-independent organizations with different missions, cultures, financials, and etc. to make each sub-enterprise span of control more reasonable. It's scope and activities make it a lot like an economy filled with at least partially independent organizations.

Inside that super enterprise, individual IT systems (and system acquisitions) proceeded in near isolation. They had external requirements and policy to follow, but those changed slowly and all of the money, authority, and accountability for a given effort intersected at a single program office. The program office might have a devilishly complex task to accomplish, but at least it (mostly) controlled its own destiny.

Of course, I'm not describing a perfect world. Many of these programs were themselves very large by the standards of commercial enterprise. As program size and the necessary controls that go with size increased, a greater portion of their overall effort was consumed with controls instead of code. As a result acquisition cycle times and overhead have increased dramatically in the last thirty years, and so have failures. Also, given the lengths of time involved, by the time a system was delivered it often turned out that no one really cared about it anymore.

This has been true from the very beginning of the DoD's IT enterprise. In fact the precedent was set by one of the earliest defense IT systems, the Semi Automatic Ground Environment (SAGE). It was a massive undertaking that required the simultaneous invention of the first multi-user digital computer (whirlwind) and a continent-wide radar system to go with it. Its completion was lauded as a testament to the newly devised methods of systems engineering, processes we are still living with today. What's talked about less is that it took so long to deliver that it never served its intended purpose. By the time it was operational, strategic bombers had been supplanted by ICBM's and the need to vector fighter defense on a continent-wide scale had gone away. SAGE became a useful but very expensive air traffic control system until the early 80's when it was finally retired.

So, for many years we've had programs of such massive size that they have been pressing up against (and frequently exceeding) the limits of our ability to centrally plan and manage them. With that size has come a decline in the ratio of value achieved for the money spent while failure rates have increased. But, we could live with that because we had the richest economy on the planet and now and again, through sheer force of will, all of that systems engineering managed to spit out a useful system. Systems that, if their time had passed, could at least be used for some other related and useful purpose.

More recently however, the DoD has started a migration to NetCentricity. NetCentricity is a revolution in the use of information in warfare. If its proponents are right it will mark a discontinuity in warfare as big as the advent of armor or air warfare. Strictly speaking NetCentricity isn't about technology, it is about the power of networked organizations, rapid information flow, and self-synchronization to serve as force multipliers and enhance the operational art of maneuver. Technology is a necessary enabler though and the important key to this discussion is the fact that NetCentricity requires systems all across the DoD super-enterprise to share information. For people to have shared awareness and self synchronization in the cognitive domain, their information systems must be connected.

The Software Engineering Institute at Carnegie Mellon University recently published a paper on ultra large systems (pdf) that goes into great detail about the challenges that come with this evolution - from thousands of independent large (or very large) systems into a single ultra large system fabric. I think one of its most important points is that Ultra Large Systems won't be managed as a single effort, but instead will consist of operationally independent but connected nodes (see page 11). What this is saying without saying it, is that the era of pretending that the DoD IT Enterprise is a single enterprise is over. An enterprise is by definition the people, systems and processes that are under a single span of control. This statement is an acknowledgement that in a NetCentric ULS-operating DoD this can no longer be the case. So, if it's not an enterprise, what is it then?

Let's ask the question another way. What happens when hundreds of programs that were already so large that they were teetering at the limits of central planning suddenly get connected to each other? When huge programs that at least had nice clean edge conditions suddenly find themselves having to coordinate across boundaries that were once impermeable and fixed? Do you get an Ultra Large System that is too big to plan? Or do you still have hundreds of only Really Big systems that just tipped over?

Either way it's a communication storm where suddenly every program manager is taxed with external communications about rapidly evolving boundary conditions (e.g. protocols, taxonomies, semantics, technologies, etc.). The speed at which those boundary conditions change, combined with the effort inherent in all that additional communication is the tipping point into a cybernetic stall. The DoD has become the proud owner of an ultra large system that is simply too large to survive and thrive as a centrally planned entity, but they are still trying to plan it.

Today the Army struggles to respond to this problem (within the still-limited scope of the Army enterprise) with its Software Blocking process. This is an effort to align all of the Army's related systems into a single release schedule (i.e. blocks). Essentially it attempts to abstract the planning process so that it can span many large programs. But the reality is, what was once a hard problem is becoming an intractable one and while software blocking can help improve interoperability across the Army's systems, it does it at the expense of speed. To stay in synch, all of the systems in a block have to keep pace with the slowest runner. A better plan would probably be to adopt emergence at the scale of each of those systems like Amazon has, with a combination of architecture, organization, and incentives.

In a more general sense, Software Blocking is part of a broader organizational reaction to the stall. It's a reaction that seeks to exert more and more control in the face of uncertainty. Acquisition processes add steps to wring risk from the process, SEI issues updates to CMM that require more process documentation and evaluation, and old timers wax nostalgically about the days "when people would just take orders and do what they were told." What those old timers are really nostalgic for are the days of clear lines of control in the hierarchy. As the hierarchy is replaced by a network both the clarity of accountability and the control that goes with it are lost.

In the military's culture of order giving and taking, pushing for even more control in the face of failing programs is only natural, but it isn't going to fix the problem. It's like trying to sidestep Heisenburg's uncertainty principle by squinting really hard when you look at an electron. It's a fools errand, a Paradox of Control. Because it turns out that once you're in a cybernetic stall, trying to de-risk with more planning and control will just make things worse. By the very nature of software, the slower you build it, and the more complete you try to make your plans before you start, the more real risk you create. Does anyone believe that the JCIDS process is enhancing warfighter effectiveness?

The web, unlike the expanding DoD super enterprise, has always been emergent. Since it was never really under anyone's control it demonstrated emergence from the beginning - It didn't need anyone to let go of control for it to get that way. As a result it has experienced a great deal of innovation - innovations that usually start small and in large quantities and then culled as they grow into a power law curve. The exact technologies of the web, or even the exact attributes that comprise its emergent eco-system, might not be the right ones to make the DoD emergent. After all, the DoD has different operational and environmental conditions. However, we would be dumb not to look at the web for ideas for the defense IT enterprise.

This is a conversation with consequences. Despite how it sounds, we're not just bantering about esoteric theory. Let's continue for just a moment with the analogy between the defense IT enterprise and the broader economic activity of the state. From The Road to Serfdom:

“It is no exaggeration to say that if we had had to rely on conscious central planning for the growth of our industrial system, it would never have reached the degree of differentiation, complexity, and flexibility it has attained. Compared with this method of solving the economic problem by means of de-centralization plus automatic coordination, the more obvious method of central direction is incredibly clumsy, primitive, and limited in scope. That the division of labor has reached the extent which makes modern civilization possible we owe to the fact that it did not have to be consciously created but that man tumbled on a method by which the division of labor could be extended far beyond the limits within which it could have been planned. Any further growth of its complexity, therefore, far from making central direction more necessary, makes it more important than ever that we should use a technique which does not depend on conscious control.” (Emphasis added by me).

China had its Deng Xiaoping and Russia had Beria. Both of them saw (at different times) that central planning was leading to economic disaster. Xiaoping started China's economy on the road to growth by permitting markets and encouraging the emergence that goes with them. Beria on the other hand didn't survive the post-Stalin power shake out and as a result the Soviet Union just kept five year planning itself into oblivion.

The Defense IT Enterprise faces a similar crossroads. The scale of the super enterprise (and the Ultra Large System landscape inside it) exceeds the limits of central planning, yet the DoD is still trying to plan itself to success. The result is failing programs, a focus only on the big stuff, and obvious (and growing) digital serfdom for our troops on the ground. In general they are less connected and more constrained in their use and development of IT than their third world adversaries. And to add insult to injury, when they show up at disaster relief sites, they often find that the NGO's have better situational awareness on the ground. Meanwhile the systems engineering machine keeps grinding out yesterwar's systems.

This begs the question, who is the DoD's Deng Xiaoping? The one that will break it out of the Paradox of Control and recognize that along with the planned "enterprise-like" components, there must be vast spaces of facilitated emergence? Most people doubt Mao meant it when he said to let a hundred flowers blossom, but the Defense IT establishment needs to embrace the notion in its policy and create an ecosystem that supports emergent behaviors, or continue to watch large scale system development falter, money get wasted, and our troops fight in digital squalor.

The Army, the Web, and the Case for Intentional Emergence

Lt. Gen. Sorenson gave a Higher Order Bit talk at the Web 2.0 Summit in San Francisco back in November. I didn't make it to the Summit this year but I'm glad I got to see the video. 

I'm glad General Sorenson is thinking about how the Army's systems and methods can be improved with Web 2.0 ideas and technologies but I wish the Army would really go after the really fundamental benefit of the Web, the fact that it is a platform that supports emergence. It's not just about the specific technologies, it's about the ecosystem of technology, economics, policy, and culture that supports rapid innovation on a generative platform. 

I think the Army can unleash a wave of innovation at the edge by replicating the web's generativity on the battlefield and a couple of California National Guard guys I met have proven it. They managed to get a single Linux box authorized for the SIPRNET in theater and quickly used it to build a collection of web applications called Combat Operations Information Network that scratched a bunch of itches for their unit.

As simple as it was (a single underpowered Linux machine on the network), once COIN was on the network it was a generative node and people lined up to get other problems solved and is now widely used across the theater.

I tell the rest of the story about my reaction to General Sorenson's talk and how the Army's Battle Command System can support innovations like COIN here at Radar. I'll just link to it rather than cross posting the rest of it.

Technorati Tags: Army, Defense, Enterprise2.0, web2.0

Want to Know What NetCentricity Looks Like?

Hey DoD, if you want to know what NetCentricity looks like, here you go. Look here and especially here.

Open Technology Conference Wrap Up - Where the Geeks At?

Yesterday I sat on the panel that I referred to here. I thought I'd follow up with a brief post about one topic of our panel conversation.

To start the panel we were asked "what's bugging us?" This started an interesting conversation about some specific open source roadblocks in defense. In particular, Bdale Garbee made the point that open source projects rely heavily on personal reputation. Even when major corporations participate in open source community, it's the reputation of the individual that determines whether and how contributions make it into the project repository. People get commit rights, not companies. This can be problematic in the defense space.

I added that many key contributors to open source projects have self-selected to participate. The ability to self select is important to the ability of a project to find people with high levels of commitment and expertise. Look at the list of contributors on the Apache web server project for example. While there are certainly participants that represent major corporations, I would estimate from looking at the list that at least a third self selected. And that third is important as it is often the source of key (and difficult to find) skills. In fact, even many of the company sponsored contributors self selected and were later hired because of their participation.

Unfortunately, for a variety of reasons, within the DoD it can be much more difficult to self-select to participate. In defense work every hour is accounted for and must match a specific project plan line item. Community participation often requires a contributor to assist with things that don't have an exact corresponding work breakdown structure element from the program that is paying them. In defense work, if you don't have a charge code, you don't work. There's simply less wiggle room for participation that doesn't directly relate to the program that is funding you at that moment.

We also touched on a bunch of other issues that impact the ability to participate in or contribute to open source projects. Things like export controls, copyright, culture, etc.

These specific issues that impact defense contribution and participation have broad implications if defense is to be able to effectively leverage the work going on in open source communities. One of the things that makes open source community tick is the right to fork. Knowing that you can fork the source if the project direction deviates from your own direction is important to alleviating risk. The antidote to forking is community participation and the development of trust. The more you participate, and the more you develop trust, the more you or your organization can influence the direction of a project or at least make sure that your specific needs can be met. With all of the rules that currently make meaningful participation difficult, it is very difficult for defense contractors to participate in the upstream software value chain. The result is perpetual forking.

It will work like this. A defense contractor does a trade space analysis and decides that they can save a lot of money for the government by using a particular open source project, so they include it in their bid. They win and they build the system using the open source component, however, they realize that they have to modify it in a few critical ways to satisfy some specific requirements. They can't participate in the community so their changes never get offered back, and never make it into the trunk. A few years later, under a follow up maintenance and sustainment contract, they do an upgrade of the system and, because their changes never made it into the core project, they have to repeat the work again on the newest version of the open source project.

In the not too distant future there will probably be whole classes of software infrastructure that are effectively only available as open source. It simply won't be economic for a proprietary software firm to compete in areas that have been completely commoditized. Therefore, it's imperative that the Department figures out how to resolve the issues that are preventing their own people or their contractors from participating meaningfully in the communities that they will be forced to rely on.

That's probably enough on that. There was one other thing I wanted to touch on in this post. This was the fourth year of this conference and, maybe I'm just an impatient person, but I'm getting really bored of the same old remedial conversations with a bunch of suits (full disclosure, I was in a suit too). Or as John Scott put it to me during a break, "Where the geeks at??" Too much of the conversation is still about whether or not Linux qualifies as CoTS in the FARS and that sort of thing. Where are the breakout groups on open geo tools? Where's the presentation from the guys using XMPP as a cheap messaging stack in some major program? Where are the non-DoD geeks who are attending because they are participating in an open source community that was started in defense but is now being widely used to solve all kinds of other problems? Where are people trying to build an open service bus that will deal with intermittent service end points that you find on a battlefield? Where are the SOSCOE developers talking about how they used JXTA's service advertising mechanisms? Etc...

It's time to move from the basics into the advance course kinds of stuff; the stuff you talk about when you are actually doing it. It's time for DoD policy makers and decision makers in key programs to really start to push; push for expertise, program outcomes, and key policy initiatives that will alleviate the kinds of road blocks we discussed (again) in our panel. In short, it's time to stop talking about open source in defense and start using it at such a meaningful scale that next year the room won't be full of suits, but will be full of geeks and practitioners.

Open Source in Defense: Consuming it is Nice, but Building it is Better

I'll be participating in a panel discussion next week at the 4th Annual DoD Open Technology Conference in DC. The Panel is about open source software in defense and will be moderated by John Scott, an author of Sue Payton's Open Technology Development Roadmap (pdf). Dan Risacher, who recently discussed the DoD CIO's upcoming policy memo with GCN, will also be on the panel. We'll be talking about what makes open source valuable to the department - consuming it, contributing to it, and even building it outright. We'll also be talking about the policy, legal, and accidental process roadblocks that make it more difficult today than it should be.

Yesterday, while I was doing some preparation, I ran across this sources sought on Fed Biz Ops. It got me thinking about the down to Earth practical stuff that is necessary to make a difference in encouraging open source in defense. I am going to come back to the details of this sources sought in a moment.

The DoD has broken the seal when it comes to consuming open source, at least in packaged form. I'm not certain where I got this factoid, but I think the US Army is now Redhat's single biggest customer. But like I've said before, consuming open source is no big deal and really isn't occasion for a big celebration. Where the DoD stands to gain much more value is in producing open source software.

Every industry has at least some domain-specific software needs. The stuff that makes up their industry-specific "stack" and that isn't readily provided in the cross-industry products from the major software vendors. For example, the financial services industry depends on things like high speed messaging buses and high availability transaction monitors. Web firms use things like Perlbal, Hadoop, and of course Apache that help them build a massively horizontally scaled web presence. Telecom has specifications like H.323 and now SLEE and SIP and products built on them.

In the old days, if the industry was big enough, domain-specific software vendors would spring up to provide them with the infrastructure that they needed (e.g. Tibco's Rendezvous). If they were REALLY big, a large software vendor might even offer a domain-specific product, or at least a version of their product.

These days though there is an alternative, Open Source Quasi- Joint Ventures. Well, nobody really calls them that, but that's how I think of them. They are like accidental joint ventures that do resource sharing the way a traditional joint venture would, but they rely on open source licensing to make the risk of participation low. Plus, they avoid most of the legal and ownership wrangling that happens in a real joint venture.

A great example of this approach is the Advanced Message Queuing Protocol (AMQP) (and associated AMQ implementations such as OpenAMQ). It was initiated by JPMorgan but has grown to include many large banks as participants. The banks don't give up any competitive advantage by participating because messaging is about passing information to trading partners, but they save money by more efficiently providing for their own infrastructure.

Things like OpenID and Hadoop also fit into this mold. Companies like Yahoo and SixApart are taking active roles in funding and guiding the development of their industry-specific technology. Again, it's far enough down in their stack that they aren't giving up a competitive position; but they are saving money by sharing their development resources.

I don't mean to say that these aren't normal open source projects in every way. I'm simply making a distinction about how and why they are funded in terms of the specific needs of the industry that is funding them. By joining together to build components for an industry-specific stack and then intentionally commoditizing it within that industry, these projects seem to be filling in where JV's or domain-specific software companies might have focused before. This open source approach is better than a traditional JV though because new participants can join up at any time and they avoid many of the up front issues of starting a JV.

Back to the DoD. Defense has saved a great deal over the last decade recognizing that it can leverage COTS hardware and software. However, it still has many unique needs for its information technology stack - the DoD operates in a different operational environment and has many specialized requirements. So, while the DoD today is beginning to consume "package" commodity open source projects such as Linux, there is still a great opportunity to steal a page from JPMorgan's playbook and build defense-specific infrastructure as open source. The DoD builds defense-specific stack components all the time, but they rarely do it in a way that makes it easy for other programs to adopt them (or even know about them). An open source approach would better spread the funding and would also ensure that once the money is spent the pieces could be widely used and adopted.

This brings me back to the WebTAS sources sought.

Open Source in Defense: A Culture Virus

I had a lot of fun this week giving a short talk at Ignite Philly. There are lots of good reasons to use open source in government, but this talk was about my "secret" reasons. In short, I think open source isn't just a good way to make software, I think it's a culture virus.

Why We Work (and Why We Would Work Harder)

I was recently asked "to provide some thoughts about how contractors should or should not perform differently or be incentivized differently during wartime." The query came as part of a research project started by an acquisition officer who is concerned that they are not seeing "exceptional performance or motivation" from either the contracting base or the government's own acquisition apparatus.

Perhaps they are worried that we are at the mall with the rest of America.

I don't know who's at the mall, but the sad truth of the matter is that the average javascript programmer at a dot com circa 2000 worked a hell of a lot harder than the typical defense contractor or government acquisition person works today (speaking frankly, I know I did). The reality in this post-mobilization-age war is that America isn't at war. While it's professional class of pointy-end-of-the-spear warfighters are doing their third or fourth tour in the combat zones, the rest of us have been sort of intentionally left out of it so that we can keep the economy humming that pays for the spear. Sometimes it seems like our government just doesn't want to inconvenience us with this whole "we're at war" thing (I might add that that's an odd position to take in a Democratic Republic where the government is *of the people*).

But we defense contractors aren't just anyone in America, we're participants. So why did those javascript slinging philosophy majors work 18 hour days while we continue work, at least collectively, a lot less urgently? What's the difference?

In short, immediacy of impact. They had it, we don't.

I do not believe that money or monetary incentives would make one whit of difference. Face it, if it was really about the money we'd be in financial services or somewhere else where the money pools around your ankles. If you want your defense industrial base to work like it's at war, it has to feel like it is at war. You have to strip away the layers of industrial-age bureaucracy that makes our work feel less like winning and more like an abstract intellectual exercise. There's a reason why WWII war heros toured defense manufacturing plants.

My work is in defense IT. So, sticking at least a bit to what I know, here are some suggestions to create the kind of immediacy that will lead to meaningful performance; performance that comes from the sense that the work has real meaning.

One. Connect the warfighter to the people building stuff.

It's no mystery that direct contact with your customer raises the stakes. This is one of the reasons why scrum development works to improve the degree of customer engagement. I think of this every time I'm leaving our office and pass the team area of one team that is always here late. It might be that they just have a hard ass team leader, but I don't think that's it. I think it's the fact that the software they work on is deployed around the world and they have a direct conduit to the warfighters that use it through a 1-800 help line that is staffed inside their team. Sure they have to do standard acquisition system stuff like program reviews, but they regularly talk directly to the warfighters that use their stuff.

Remember the Automated Deep Operations System (ADOCS) story? Acquisition people hate it because it "was off the reservation," used non-standard architectures, and yada yada. But it has provided tremendous value, has served the warfighter well, and was built by engineers who were working overseas side by side with the warfighters that were using it. Let us do more of that. It works. No one in the commercial world would work as hard as the DoD does to separate the people slinging code from the people running it.

While we are at it, let's better connect our acquisition conduit to the warfighter too. There are fewer acquisition people in the DoD now than there were ten years ago, but there are still more of them than there are front line M-16 carrying troops. In an era when even Navy submarine officers are being rotated into Iraq between sea tours I think we can rotate our acquisition people in there too. They can see first hand how the stuff we are building for them is doing and gain an intuitive sense of what works that will be so useful later. Is it relevant? Is it usable? Is it providing value for the dollar? Will the operators be able to reconfigure it to meet their local needs? Etc.

CDD's and requirements documents are low bandpass filters. Being on the ground eating their own dog food in a war zone will make it easy for our acquisition folks to put the high frequency band back into the requirements comms. Be a warfighter for a while, think like one forever.

In situations where you can't put people over seas, bring the warfighter back to us through simple and available technologies like help lines, web forums, email, VoIP, mailing lists, etc. Heck, build those little active chat help things right into the apps if it is on an unclass network. Actively work to make a direct connection between the people writing code and the people running it.

Two. Add value to your customers, now.

The warfighter needs stuff now. Have us build stuff that you are going to rapidly deploy to them now, rather than spend all the money on stuff you think you'll deploy in 2011. Sure, we'd like to help build the longer term big programs that have well known and well worn acronyms, but let's get real, we're at war. If we want IT to help us stay inside our enemy's OODA loops we need to be roll out stuff our troops can use now. Imagine how frustrating it must be for the deployed warfighter to ask for something during their first tour knowing there is little chance of them seeing it before their fourth. Use those acquisition folks you've deployed into the war zone in step one to be on the ground product managers. Plan to deliver stuff to the warfighters they are embedded with while they are still there.

Put another way, let's figure out how we can deliver a range of capabilities. In the commercial world big things like an ERP system are done centrally and with a lot of oversight while smaller scale departmental initiatives are done quicker and locally. We shouldn't be thinking about how to deliver everything the warfighter needs from the center, we should be thinking about how to let the warfighter build or buy stuff locally. Need a web site to share information among all the platoons in a brigade? Let your javascripting sergeants build it locally on platforms that the center is providing for you. Enable the long tail of development.

Three. Build it incrementally.

I have never felt nervous perspiration on the back of my neck when facing a deadline that was three years away (and I don't mean those intermediate soft deadlines, I mean the real ones). Use agile incremental development along with the power of product platforms to quickly and incrementally deliver stuff to the field. Give us deadlines that are three months out, not three years. Exercises don't count.

Don't think in terms of "requirement" = "tool." Think in terms of generative platforms that will let the warfighter self serve on the little problems and will let small contractors quickly and innovatively deliver on the slightly more complex problems.

How is that possible you say? How can we deliver stuff without operationally testing it first? By engaging much closer with the warfighter while you build it (e.g. like the way ADOCS was built), using product platforms that have gone through pre-certification and operational use, and etc. First, believe you can be relevant to the warfighter now and then make it happen; we would love this. We are dying to be relevant today.

Some years ago a company I was working for sent me to a career planning seminar. The instructor gave us a framework for thinking about our careers that said satisfaction comes from balancing compensation, competency, and meaning. I think many of us in this business derive a large amount of satisfaction from the meaning associated with our work. Thinking back to my time in a B2B startup the meaning came from knowing that the thing I was building today was going to be deployed and earning revenue next week.

If you want to get even more from us, help us improve the meaning dimension by making it easier for us to do work that makes an impact now. Clear out the bureaucracy, help us get closer to the end customer, and deliver stuff quickly that will really make a difference. You might be surprised what happens.

Warfare's Long Tail - Cyber Warfare in Georgia

A quick summize search will take you to a bunch of articles like this one focusing on Georgia and Russia's cyber war; the one that is paralleling the much more deadly kinetic exchange on ground and in the air. While giving the reader ample opportunity to reminisce about Estonia, the reporting raises the usual questions. Where does state sponsorship (and intent) end and "volunteerism" begin? Who is responsible for the attacks and how are they carried out? That sort of thing.

I guess I'm struck as much by who's not talking about the cyber side of things. As you read the articles you find the intelligence reports coming from non-profits (e.g. Shadowserver Foundation), private companies (renesys) and private citizens (e.g. Armin) rather than from intelligence agency public briefings. The defensive infrastructure is being supplied by companies like Deutsch Telekom (data links) and Google (blogs to replace downed government web sites). And, well, the offense is probably coming from the RBN.

What you don't see are press room appearances by a NATO Cyber Defense spokesperson offering assistance or Russian military and government spokespeople talking about their glorious cyber offensives. While the conventional military channels focus on the kinetic war, the "amateurs" talk about (or participate in) the cyber campaign.

Historically, the line between professional militaries and the rest of a society were less clear than it has been since the first world war. Since then, at least in most of the industrialized world, kinetic warfare has been the domain of the professional. We civilians were expected to participate in metal drives and watch news reels and that was about it.

Later, the broadcast medium of television had a significant impact on how professional wars were fought by more tightly coupling success or failure on the battlefield back to the political domain. It dumped real time public reaction and sentiment into the mix and tended to shorten the cycle of political consequence. However, television did little to change who actually did the fighting.

Now it's different. In the same way it is impacting so many facets of modern life, the Internet appears to be bringing the long tail of participation to at least the cyber domain of modern warfare. As the professional militaries continue to develop capability in this domain we may ultimately see the tail and the head link up in a more seamless manner than we've seen in conventional warfare for a long time.

Barcamp.mil Follow Up

The inaugural barcamp.mil went down yesterday in Crystal City without a hitch and, at least for me, was a real blast. Thanks to John Scott and Mercury Computer Systems for providing space, pizza, and even some celebratory Tsing Tao beer. The turnout was good and we ended up with strong tracks in GIS and DoD open source as well as some great sessions on security (high assurance computing), cloud computing in DoD, and enabling innovation. There were probably others but being able to be in only one place at a time... The lunchtime session on evolving open source policy in the DoD was of particular interest. There are definitely people in the department that get it which is heartening.

John and I both agree that we'd like to do more of these things; perhaps tied in with major conferences such as I/ITSEC, the annual DISA Partner Conference, or similar venues that draw DoD geeks together. If you are running a conference and you'd like to facilitate a simul-barcamp.mil let John or I know.

Some people are probably wondering how it is possible to mix the barcamp ethos with the defense space. All I can say is, you should check it out. Despite the bureaucracy we deal with in this space there is really cool work being done (and some of it can even be talked about). Beyond that, I believe there is a growing cultural gap developing along generational lines within the industry. We didn't all grow up here and we are bringing our culture, values, and methods of working with us. From agile to open source the defense space is undergoing big changes. barcamp.mil is just another way for like minded people to connect, share ideas, and spread the culture virus.

Industrial age mechanisms simply aren't working anymore for the connected forces the DoD envisions. We have an opportunity to really make a difference in how tomorrow's force is equipped, but even more importantly, how it works. Hey, it's a Democratic Republic and it's our military too.

Barcamp Mil

John Scott and I are planning a barcamp in Crystal City (right next to a DC metro stop) on August 8 and 9. John and I both advocate open source in the defense space so we are hoping that there will be people coming to demo what they've built; either in defense or that would be useful in the space. However, there are lots of other areas to explore as well so if you are in the area and have something to talk about or show, please sign up here.

Army Cyber Warrior - Slashdot Interview

Following on the heel's of General Lord's interview on Slashdot some months back, the Army's Lt Col John Bircher responds to questions about the Army's role in Cyberspace.

I really like the fact that the armed services are making this effort to engage. We are no where near creating the kind of uniform to volunteer continuum that Richard Bejtlich discusses in his recent post about the USAF Cyber Symposium and I think there is much more practical day to day participation that can be done, but these kinds of baby steps toward are important and should be applauded.

On a recent panel at USAF Cyber Symposium I suggested that the USAF Cyber Command should avoid looking inward for the answers to every question and should look for opportunities to engage a wider community. As an example, I suggested that they make their Internet boundary router tap data available to researchers and volunteers as a way to get more eyes looking for patterns of potential intrusions. At one level this kind of proposal is simple pragmatism; get more eyes on the data ala wikipedia or Linux and do a better job solving problems. However, there is a an idealistic angle as well. I believe that in a Democratic Republic it is imperative that people stay involved at some meaningful and personal level with things like governance and defense. On an intuitive level I get nervous when the government and military turn inwards and see themselves as separate and distinct from the people they are governing and defending.

In his response to question number 8 Lt Col Bircher himself discusses this growing concern. There are things in the sphere of Cyberwarfare that are unlikely to become fully transparent anytime in the near future, but the more the membrane between the military professional caste and the people it supports is made permeable, the more room there will be for trust to develop and the more effective the military will be.

Sharing vs. Protecting, Generativity on DoD Networks

Before I dive into the topic of this post I should just say I can't believe it has been March since I've posted anything here. Epic lame. I've had lots of ideas for posts but just haven't seemed to get around to posting them. One minor excuse is that I've been posting as a contributor at radar.oreilly.com (you can find my posts here) and many of the little things I might have posted here before now just end up on my feed at twitter. They both function as sort of Limn This post relief valves. I'm going to try to get back into the swing of things here though. Now, back to the topic...

Last week I participated in a lively panel discussion moderated by Col Leslie Blackham of Electronics Systems Command at the second USAF Cyber Symposium. Panelists included Noah Shactman from Wired's Danger Room, Grant Wagner from the NSA and SE Linux fame, Richard Bejtlich from GE and Tao Security, Cisco's CSO John Stewart, and Marianne Hedin from IDC. The theme of the panel was "Sharing vs. Protecting" and exploring that tension gave lots of opportunity for a broad range of conversation that included open technology, organizational issues, social computing in the defense space, and even the role government in a Democratic Republic (well, maybe just a little bit of that).

A theme similar to Sharing vs. Protecting is discussed in Jonathan Zittrain's excellent book The Future of the Internet and How to Stop It. In the book Zittrain explores the tension between "generativity" and lockdown that is playing out on the public Internet and in corporate networks. In those venues security concerns are creating pressure to replace the open PC (you can run any program you like on it) with tethered appliances that remain under the control of their makers such as the blackberry, iPhone, Xbox, TiVo, etc. We see the same trend in the defense space as devices like Sun's SunRay find a market. As a result we are squeezing out venues for the kinds of experimentation that led us to the Internet we have today (e.g. this article is as an example of the impact of this change on the gaming industry).

In the defense space, where the ability to rapidly innovate and deploy new code can be equivalent to maneuver on the traditional battlefield, this generativity vs. lockdown argument has real consequences. Networks like the Navy Marine Corps Internet (NMCI) are like ships that have permanently set Condition Zebra. Such a ship would have great watertight integrity, but can you imagine having to get commanding officer permission every time you need to open a watertight door to go to chow? Networks like NMCI are so constrained that they are difficult to use for their intended missions and absolutely worthless for generative activities that lead to innovation. The DoD's next FalconView-like grass roots endeavor is not going to come out of one of these networks.

The question Zittrain attempts to answer in his book distills down to "how do we maintain a highly generative environment while eliminating the security concerns that plague the current Internet?" The DoD has the same challenge even if it would use different words. One of the more interesting questions put to the panel, and that speaks directly to this question, was "How do we manage local innovation and capability development in a web 2.0 environment?" I posited that at least part of the answer might be found in something like Facebook's platform or SalesForce.com's platform as a service offering but tailored for things that the military focuses on, like command and control and situational awareness tools. I believe that a highly secure application platform, acting as an aggregation point for a variety of data and content, and exposing application capability and data via simple API's would unleash all manner or rapid innovation from warfighter and contractor alike.

On the Internet Zittrain worries that platforms such as these, though highly generative, are problematic because they are contingent; they can be shut down or modified to be less open at any time by their owners. On the Internet, the contingency of generative platforms can have a chilling effect for those smart enough to realize the inherent risk in committing to them, or can have a post facto impact on innovations later thought by the platform owners to be competitive or otherwise disruptive. I think this is less of an issue in the defense space as long as the government and not one of its' contractors owns the platform (this is a really big "if", just look at Boeing's stranglehold on SOSCOE).

This idea of C2 Platform as a Service probably deserves its own post later (particularly that part about ownership) so for now I'll just introduce it as a possible mechanism to provide secure generativity on the DoD Internet. However, before I close out this post I just want to touch one other item that was discussed on the panel, the value of the social web.

We were asked weather the benefits of the social web outweigh the security risks. In my response I suggested that the word "social" tends to skew the conversation toward trivialities and away from meaningful participation in communities of practice. Richard Bejtlich made the point that there are about 12 other people in the world that do what he does and they form the nucleus of an incredibly important community to him. Despite the fact that he is an ex Air Force officer, none of his peers in that community are in the Air Force. His natural peers in the Air Force aren't collaborating across organizational lines with him; are they even peering effectively with each other inside the organization? Analogously, today's Air Force pilots don't participate in communities of practice with airline pilots or stunt pilots, but it wasn't always this way. Earlier in the development of military aviation flying clubs were important feeders into military cockpits and places where best practices were developed and communicated. In the cyber domain it is still more like 1914 than like 2008.

I believe we are also missing the opportunity to leverage these kinds of technologies to turn the currently binary "deployed" / "not deployed" bit into something much more seamless. Why is that ALL of a departing soldier or airman's expertise and participation goes with him or her when they rotate out of theater? It seems to me that there is a massive opportunity to "virtualize" the organization such that hard won knowledge and expertise could still be made available in theater in some combination of ad hoc and formalized ways.

This post is already too long so I'll save other topics that we discussed (e.g. open technology, the opportunity to harness volunteerism, etc.) for future posts. Let me know if you have also been thinking about how to preserve generativity while simultaneously protecting our important network assets.

"DISA Inside" - NCES should be a SaaS Appliance

The title of this post is not part of my secret plan to obscure meaning through the liberal use of acronyms. It really just came out that way.

Here's what the acronyms mean:

DISA = Defense Information Systems Agency. The once and future DoD phone company, now also responsible for stuff like enterprise application hosting, service oriented architecture (SOA) infrastructure and the command and control systems that will use it. Today DISA's center of gravity remains firmly in the data center.

NCES = Net Centric Enterprise Services. The SOA infrastructure that I mentioned above plus stuff like enterprise chat and search. Current state has a high ppt to compute ratio.

SaaS = Software as a Service. I know you already knew that one but I felt compelled to include it for completeness.

So, now on to the meat of the post.

If DISA is going to have relevance outside of the data center, and therefore relevance to the warfighter, it needs to have an impact on the experience at the warfighting network edge. Today's data center focus combined with the realities of network availability at the edge makes that unlikely. The NCES' core services of messaging, security, search, and things like that are simply going to be useless to the warfighter at the end of an intermittent or low bandwidth network. This is really nothing new as everyone already recognizes the difficulty in supporting shipboard command and control systems with remotely provisioned NCES services. What's missing today though is a focused strategy to obtain relevancy at the edge.

This article's mention of SaaS appliances reminded me of a conversation I had with DISA engineers about two years ago. The gist of the conversation was "what if you guys were to think like a company and stretch NCES out to the edge by building (or commissioning) a line of NCES appliances as a combined product and services offering? You could build them into the two or three common form factors so that they could fit into server racks (like in a Tactical Operations Center) or into a vehicle (in an Integrated Computer System form factor). Then design them so that you could add value by remotely providing systems management, domain spanning messaging, and things like that. Finally, paint them white and put big blue 'DISA Inside' logos on them so everyone knows they came from you."

The appliances would allow for modular software deployment (including messaging and message routing, mediation, information assurance functions, search and etc.) either as different appliances or as independent functions within a single appliance depending on the deployment environment. Relying on local caching it would serve as a complete (if narrowly aware) NCES environment on the battlefield when intermittent network realities left it disconnected from the borg, but it would intelligently re-synch when connected.

Done right, I can imagine all kinds of programs spec'ing such an appliance as both a core piece of local infrastructure and also as the bridge back to the enterprise. Army Battle Command System, the USAF Objective Gateway, Future Combat System, Shipboard command and control systems, USAF Air Operations Center, just to name a few, could all use common services packaged this way and whose enterprise awareness seamlessly expanded and collapsed as network availability allowed.

So what do I think "done right' means? I think it means an adoption-friendly appliance line based on an open stack that all of those consuming programs can inspect and contribute to. I imagine a DISA-managed collaborative ecosystem of related open communities delivering the stack and service components into a related ecosystem of appliance hardware vendors. In a perfect world the collaborative eco-system would be market driven and would span the DoD's garden walls. By avoiding excessive gating it would serve as an effective two way technology transfer mechanism into and out of the defense establishment; stack components coming in technologies like context-aware message routers flowing out.

I know this probably sounds kind as crazy to the DoD establishment as it does mundane to the open source world outside it. It's all a matter of perspective I guess. You have to dream it to do it.

DoD Software Development - the Wrong Center of Effort

I spent some time last week going through the details of a well known DoD three letter acronym IT program. There's no point in disclosing the particulars because the observation I'm about to make applies to most of them. But, as I was looking at the capabilities of the software and seeing it demonstrated I asked "Hmmm... I always thought there was more to this thing. Is this really all it is?" A: "Well, yeah, it's mostly just an integration of a COTS database with middleware. The code itself is a few hundred lines of database data definition language and probably less than ten thousand lines of Java, probably a lot less."

During the rest of the conversation we talked about the fact that, since this is "joint" program it has a staff of program managers and systems engineers at each service plus at OSD and JFCOM to develop the requirements, "design the architecture" and so on. Each group also has its own testing and certification staff to prove out the code before it can be integrated into their versions of the C2 node this thing is designed to integrate... What you get looks like this (where conceptually, the cost per line of code can be estimated by integrating the area under the curve and dividing by the lines of code):

10,000 lines of code. I've seen commercial software development programs with 500 developers and millions of lines of code with less management and oversight. Besides a willingness to accept extraordinarily slow delivery of software, this approach so burdens each line of code with overhead, that the world of defense IT has developed bloated expectations for what things cost. The kind of overhead burden that makes quotes like "you don't understand the pressures we're under, we only have a budget of $250M" seem to make sense.

When it comes to IT, we in defense systems development need to move past this industrial age systems engineering choked approach and get to something that is agile, fast, and more cost effective. Just make it look like this:

General Lord Responds on /.

General Lord responded today to questions posted on Slashdot. Like I said before hats off to Gen Lord for even attempting this. On a relative scale this is an amazing amount of openness. The General's responses had just enough authenticity that I believe he probably authored the original content. But, I suspect it went through a process something like this before it was released:

1st Draft:

YGTBKM! But seriously, I'd love to have a strategy like the one the Chinese have and harness you guys into some kind of ass-kicking lead-turning civil cyber patrol but you know I can't, turns out that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? By the way, you didn't hear it from me, but there is a list of interesting IP addresses stego-hidden in my bio picture at af.mil, in the big jpeg one.

2nd Draft after once over by his deputy:

YGTBKM! But seriously, I'd love to have a strategy like "some other places" and harness you guys into some kind of really good civil cyber patrol but you know I can't, turns out that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? Please email me for a list of interesting IP addresses (margin note: what's stego?).

3rd Draft after legal review:

YGTBKM! LOL! I CAN haz cheezburger!

4th Draft after second deputy review:

YGTBKM! LOL! But seriously, I'd love to have a strategy like some other places and harness you guys into some kind of really good civil cyber patrol but you know I can't, that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? Really, email me.

5th Draft, PAO Approval:

Mmmmm... Cheezburger.

...


Final Draft after all corrections:

YGTBKM! LOL! I like your enthusiasm, but you know the Air Force neither encourages nor condones criminal activity.