Limn This

Barcamp Mil

John Scott and I are planning a barcamp in Crystal City (right next to a DC metro stop) on August 8 and 9. John and I both advocate open source in the defense space so we are hoping that there will be people coming to demo what they've built; either in defense or that would be useful in the space. However, there are lots of other areas to explore as well so if you are in the area and have something to talk about or show, please sign up here.

Army Cyber Warrior - Slashdot Interview

Following on the heel's of General Lord's interview on Slashdot some months back, the Army's Lt Col John Bircher responds to questions about the Army's role in Cyberspace.

I really like the fact that the armed services are making this effort to engage. We are no where near creating the kind of uniform to volunteer continuum that Richard Bejtlich discusses in his recent post about the USAF Cyber Symposium and I think there is much more practical day to day participation that can be done, but these kinds of baby steps toward are important and should be applauded.

On a recent panel at USAF Cyber Symposium I suggested that the USAF Cyber Command should avoid looking inward for the answers to every question and should look for opportunities to engage a wider community. As an example, I suggested that they make their Internet boundary router tap data available to researchers and volunteers as a way to get more eyes looking for patterns of potential intrusions. At one level this kind of proposal is simple pragmatism; get more eyes on the data ala wikipedia or Linux and do a better job solving problems. However, there is a an idealistic angle as well. I believe that in a Democratic Republic it is imperative that people stay involved at some meaningful and personal level with things like governance and defense. On an intuitive level I get nervous when the government and military turn inwards and see themselves as separate and distinct from the people they are governing and defending.

In his response to question number 8 Lt Col Bircher himself discusses this growing concern. There are things in the sphere of Cyberwarfare that are unlikely to become fully transparent anytime in the near future, but the more the membrane between the military professional caste and the people it supports is made permeable, the more room there will be for trust to develop and the more effective the military will be.

Sharing vs. Protecting, Generativity on DoD Networks

Before I dive into the topic of this post I should just say I can't believe it has been March since I've posted anything here. Epic lame. I've had lots of ideas for posts but just haven't seemed to get around to posting them. One minor excuse is that I've been posting as a contributor at radar.oreilly.com (you can find my posts here) and many of the little things I might have posted here before now just end up on my feed at twitter. They both function as sort of Limn This post relief valves. I'm going to try to get back into the swing of things here though. Now, back to the topic...

Last week I participated in a lively panel discussion moderated by Col Leslie Blackham of Electronics Systems Command at the second USAF Cyber Symposium. Panelists included Noah Shactman from Wired's Danger Room, Grant Wagner from the NSA and SE Linux fame, Richard Bejtlich from GE and Tao Security, Cisco's CSO John Stewart, and Marianne Hedin from IDC. The theme of the panel was "Sharing vs. Protecting" and exploring that tension gave lots of opportunity for a broad range of conversation that included open technology, organizational issues, social computing in the defense space, and even the role government in a Democratic Republic (well, maybe just a little bit of that).

A theme similar to Sharing vs. Protecting is discussed in Jonathan Zittrain's excellent book The Future of the Internet and How to Stop It. In the book Zittrain explores the tension between "generativity" and lockdown that is playing out on the public Internet and in corporate networks. In those venues security concerns are creating pressure to replace the open PC (you can run any program you like on it) with tethered appliances that remain under the control of their makers such as the blackberry, iPhone, Xbox, TiVo, etc. We see the same trend in the defense space as devices like Sun's SunRay find a market. As a result we are squeezing out venues for the kinds of experimentation that led us to the Internet we have today (e.g. this article is as an example of the impact of this change on the gaming industry).

In the defense space, where the ability to rapidly innovate and deploy new code can be equivalent to maneuver on the traditional battlefield, this generativity vs. lockdown argument has real consequences. Networks like the Navy Marine Corps Internet (NMCI) are like ships that have permanently set Condition Zebra. Such a ship would have great watertight integrity, but can you imagine having to get commanding officer permission every time you need to open a watertight door to go to chow? Networks like NMCI are so constrained that they are difficult to use for their intended missions and absolutely worthless for generative activities that lead to innovation. The DoD's next FalconView-like grass roots endeavor is not going to come out of one of these networks.

The question Zittrain attempts to answer in his book distills down to "how do we maintain a highly generative environment while eliminating the security concerns that plague the current Internet?" The DoD has the same challenge even if it would use different words. One of the more interesting questions put to the panel, and that speaks directly to this question, was "How do we manage local innovation and capability development in a web 2.0 environment?" I posited that at least part of the answer might be found in something like Facebook's platform or SalesForce.com's platform as a service offering but tailored for things that the military focuses on, like command and control and situational awareness tools. I believe that a highly secure application platform, acting as an aggregation point for a variety of data and content, and exposing application capability and data via simple API's would unleash all manner or rapid innovation from warfighter and contractor alike.

On the Internet Zittrain worries that platforms such as these, though highly generative, are problematic because they are contingent; they can be shut down or modified to be less open at any time by their owners. On the Internet, the contingency of generative platforms can have a chilling effect for those smart enough to realize the inherent risk in committing to them, or can have a post facto impact on innovations later thought by the platform owners to be competitive or otherwise disruptive. I think this is less of an issue in the defense space as long as the government and not one of its' contractors owns the platform (this is a really big "if", just look at Boeing's stranglehold on SOSCOE).

This idea of C2 Platform as a Service probably deserves its own post later (particularly that part about ownership) so for now I'll just introduce it as a possible mechanism to provide secure generativity on the DoD Internet. However, before I close out this post I just want to touch one other item that was discussed on the panel, the value of the social web.

We were asked weather the benefits of the social web outweigh the security risks. In my response I suggested that the word "social" tends to skew the conversation toward trivialities and away from meaningful participation in communities of practice. Richard Bejtlich made the point that there are about 12 other people in the world that do what he does and they form the nucleus of an incredibly important community to him. Despite the fact that he is an ex Air Force officer, none of his peers in that community are in the Air Force. His natural peers in the Air Force aren't collaborating across organizational lines with him; are they even peering effectively with each other inside the organization? Analogously, today's Air Force pilots don't participate in communities of practice with airline pilots or stunt pilots, but it wasn't always this way. Earlier in the development of military aviation flying clubs were important feeders into military cockpits and places where best practices were developed and communicated. In the cyber domain it is still more like 1914 than like 2008.

I believe we are also missing the opportunity to leverage these kinds of technologies to turn the currently binary "deployed" / "not deployed" bit into something much more seamless. Why is that ALL of a departing soldier or airman's expertise and participation goes with him or her when they rotate out of theater? It seems to me that there is a massive opportunity to "virtualize" the organization such that hard won knowledge and expertise could still be made available in theater in some combination of ad hoc and formalized ways.

This post is already too long so I'll save other topics that we discussed (e.g. open technology, the opportunity to harness volunteerism, etc.) for future posts. Let me know if you have also been thinking about how to preserve generativity while simultaneously protecting our important network assets.

"DISA Inside" - NCES should be a SaaS Appliance

The title of this post is not part of my secret plan to obscure meaning through the liberal use of acronyms. It really just came out that way.

Here's what the acronyms mean:

DISA = Defense Information Systems Agency. The once and future DoD phone company, now also responsible for stuff like enterprise application hosting, service oriented architecture (SOA) infrastructure and the command and control systems that will use it. Today DISA's center of gravity remains firmly in the data center.

NCES = Net Centric Enterprise Services. The SOA infrastructure that I mentioned above plus stuff like enterprise chat and search. Current state has a high ppt to compute ratio.

SaaS = Software as a Service. I know you already knew that one but I felt compelled to include it for completeness.

So, now on to the meat of the post.

If DISA is going to have relevance outside of the data center, and therefore relevance to the warfighter, it needs to have an impact on the experience at the warfighting network edge. Today's data center focus combined with the realities of network availability at the edge makes that unlikely. The NCES' core services of messaging, security, search, and things like that are simply going to be useless to the warfighter at the end of an intermittent or low bandwidth network. This is really nothing new as everyone already recognizes the difficulty in supporting shipboard command and control systems with remotely provisioned NCES services. What's missing today though is a focused strategy to obtain relevancy at the edge.

This article's mention of SaaS appliances reminded me of a conversation I had with DISA engineers about two years ago. The gist of the conversation was "what if you guys were to think like a company and stretch NCES out to the edge by building (or commissioning) a line of NCES appliances as a combined product and services offering? You could build them into the two or three common form factors so that they could fit into server racks (like in a Tactical Operations Center) or into a vehicle (in an Integrated Computer System form factor). Then design them so that you could add value by remotely providing systems management, domain spanning messaging, and things like that. Finally, paint them white and put big blue 'DISA Inside' logos on them so everyone knows they came from you."

The appliances would allow for modular software deployment (including messaging and message routing, mediation, information assurance functions, search and etc.) either as different appliances or as independent functions within a single appliance depending on the deployment environment. Relying on local caching it would serve as a complete (if narrowly aware) NCES environment on the battlefield when intermittent network realities left it disconnected from the borg, but it would intelligently re-synch when connected.

Done right, I can imagine all kinds of programs spec'ing such an appliance as both a core piece of local infrastructure and also as the bridge back to the enterprise. Army Battle Command System, the USAF Objective Gateway, Future Combat System, Shipboard command and control systems, USAF Air Operations Center, just to name a few, could all use common services packaged this way and whose enterprise awareness seamlessly expanded and collapsed as network availability allowed.

So what do I think "done right' means? I think it means an adoption-friendly appliance line based on an open stack that all of those consuming programs can inspect and contribute to. I imagine a DISA-managed collaborative ecosystem of related open communities delivering the stack and service components into a related ecosystem of appliance hardware vendors. In a perfect world the collaborative eco-system would be market driven and would span the DoD's garden walls. By avoiding excessive gating it would serve as an effective two way technology transfer mechanism into and out of the defense establishment; stack components coming in technologies like context-aware message routers flowing out.

I know this probably sounds kind as crazy to the DoD establishment as it does mundane to the open source world outside it. It's all a matter of perspective I guess. You have to dream it to do it.

DoD Software Development - the Wrong Center of Effort

I spent some time last week going through the details of a well known DoD three letter acronym IT program. There's no point in disclosing the particulars because the observation I'm about to make applies to most of them. But, as I was looking at the capabilities of the software and seeing it demonstrated I asked "Hmmm... I always thought there was more to this thing. Is this really all it is?" A: "Well, yeah, it's mostly just an integration of a COTS database with middleware. The code itself is a few hundred lines of database data definition language and probably less than ten thousand lines of Java, probably a lot less."

During the rest of the conversation we talked about the fact that, since this is "joint" program it has a staff of program managers and systems engineers at each service plus at OSD and JFCOM to develop the requirements, "design the architecture" and so on. Each group also has its own testing and certification staff to prove out the code before it can be integrated into their versions of the C2 node this thing is designed to integrate... What you get looks like this (where conceptually, the cost per line of code can be estimated by integrating the area under the curve and dividing by the lines of code):

10,000 lines of code. I've seen commercial software development programs with 500 developers and millions of lines of code with less management and oversight. Besides a willingness to accept extraordinarily slow delivery of software, this approach so burdens each line of code with overhead, that the world of defense IT has developed bloated expectations for what things cost. The kind of overhead burden that makes quotes like "you don't understand the pressures we're under, we only have a budget of $250M" seem to make sense.

When it comes to IT, we in defense systems development need to move past this industrial age systems engineering choked approach and get to something that is agile, fast, and more cost effective. Just make it look like this:

General Lord Responds on /.

General Lord responded today to questions posted on Slashdot. Like I said before hats off to Gen Lord for even attempting this. On a relative scale this is an amazing amount of openness. The General's responses had just enough authenticity that I believe he probably authored the original content. But, I suspect it went through a process something like this before it was released:

1st Draft:

YGTBKM! But seriously, I'd love to have a strategy like the one the Chinese have and harness you guys into some kind of ass-kicking lead-turning civil cyber patrol but you know I can't, turns out that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? By the way, you didn't hear it from me, but there is a list of interesting IP addresses stego-hidden in my bio picture at af.mil, in the big jpeg one.

2nd Draft after once over by his deputy:

YGTBKM! But seriously, I'd love to have a strategy like "some other places" and harness you guys into some kind of really good civil cyber patrol but you know I can't, turns out that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? Please email me for a list of interesting IP addresses (margin note: what's stego?).

3rd Draft after legal review:

YGTBKM! LOL! I CAN haz cheezburger!

4th Draft after second deputy review:

YGTBKM! LOL! But seriously, I'd love to have a strategy like some other places and harness you guys into some kind of really good civil cyber patrol but you know I can't, that would be illegal. Oh well. But hey, if you kick a little ass and I didn't ask you to, what can I do about it? Really, email me.

5th Draft, PAO Approval:

Mmmmm... Cheezburger.

...


Final Draft after all corrections:

YGTBKM! LOL! I like your enthusiasm, but you know the Air Force neither encourages nor condones criminal activity.

CHIPSAFE for Critical Components?

There has been a surge of articles like this one lately that raise the issue of supply chain integrity for critical computing components especially in the context of security. The emerging trend toward open hardware as exemplified by Sun OpenSPARC will only make the sourcing provenance issue more important as the openness of the design will make it that much easier to knock off a slightly modified chip.

I was the Quality Assurance officer on a U.S. Navy Submarine for a while (too long, it was an awful if important job) and this problem made me think of the SUBSAFE program. Started after the loss of the Thresher it put in stringent controls on material for all seawater systems that cover how the materials are sourced, inspected, tracked to point of installation, installed, tested in place and etc. The reactor systems are covered by similar controls all of which are designed to make sure that the specified thing was manufactured, bought, shipped, and installed, and that it lived up to its specification as implemented. To me, the QA officer at the end of a long chain, it meant material control tags, work package reviews, periodic work-in-progress inspections, a stack of pre-underway paperwork next to my desk that reached from the floor to the desktop, and many many many signatures.

The NSA's Trusted Foundry Access program sounds like the sourcing piece of SUBSAFE for silicon. I hope it never needs to be extended into a CHIPSAFE program that covers the entire supply chain as the expense and difficulty imposed by such a system would be staggering (there are a lot more things using computers than there are submarines). However, I suspect eventually it will for at least some uses. However, computing is so pervasive that I'm not sure a system like SUBSAFE with carefully controlled and focused scope can be effective.

Cyber Command's General Lord Agrees to Interview on Slashdot

As a quick follow up to my Blogs of War post I thought I'd post this link to Slashdot. The funny thing is that based on the recent actions of Gen Lord's very own AFNOC, neither the Slashdot thread or this blog post will be visible from a USAF computer. But...

Apparently General Lord, the provisional Cyber Command commander (nice alliteration there huh?) has agreed to take questions from Slashdot readers. I think this is pretty cool and along with other recent comments about culture make me think Gen Lord is really making an effort to attract talent to his nascent command.

Some of the comments showing up on the slashdot thread are pretty good and some are just plain funny. Worth scrolling through.

The Blogs of War

The USAF has recently clamped down on access to blogs on internal networks as reported at Wired and elsewhere. There is no point in my joining in the general outcry. The AF is already getting an earful. I guess I will say that I'm disappointed that my readership is going to plummet from its recent peak of about ten to more like two as it has already been confirmed that it is blocked at Electronic Systems Command. The two of you still here from Army CECOM stick around, I'll try to switch back to SOSCOE soon...

By emphasizing only "official news sources" the Air Force will clearly be limiting its access to all manner of valuable information such as Michael Yon's inside look at what's going on in Iraq. But maybe more specifically, I can't imagine why Cyber Command would want to limit access to blogs like The Dark Visitor and Tao Security. The long tail IS where it's at in this space. You can't rely on CNN to learn about Chinese hackers and network security monitoring trends.

I spent a decade in the Navy and I can intuitively understand how a hierarchical military culture will respond to external stimuli such as the web. Control it, contain it, deny it and certainly don't let anyone even appear to be having fun with it! (Fun will only be authorized between the hours of 1900 and 2100 in designated areas with approved fun augmentation devices such as a volleyball - unless of course the command is sponsoring mandatory fun at the Morale Welfare and Recreation Club in which case fun will be required between 1500 and 1700 and may involve egg tossing).

When I left the service and began the arduous process of de-institutionalization I learned some important lessons from one of my first bosses, Jay Brown. Chowderhead (a colleague of mine, also ex-Navy, and with a somewhat unconventional and unfortunate call sign that stuck with him) and I would be trying to figure out how to better control some situation or another and Jay would just sit back and say nothing. Jay simply excelled at letting stuff play out a bit, letting it percolate, and eventually evolve into a good solution. He encouraged us to do the same thing more often than it seemed to make sense. For a couple of ex Naval Officers with a strong "J" preference at the end of our Myers-Briggs, this "letting things happen" could be absolutely maddening. But, over time I really came to appreciate what he was teaching us. How (and when) to let things go and trust in the evolving wisdom of the people that worked for us.

As warfare becomes more networked and traditional hierarchical command and control becomes less applicable to at least some areas of the modern fight, I think the lesson that Jay taught Chowderhead and I is becoming more and more relevant to a whole generation of military leaders. It boils down to expectations. If you expect your people to frivolously waste time and that only by you actively controlling them will they accomplish anything useful, you will process their visiting blogs in one way. If you expect them to be aggressively learning, adapting and sharing information and trust them to do it, you will see their use of blogs in a completely different light. Some of them will violate that trust and goof off, but so what? Everyone else will be so much more productive and engaged.

So, with that as background I'm going to briefly take on the orthodoxy a bit. Out here in blog land it is easy for us to know better and say it - "Let them read blogs!" Back seat drivers, Monday morning quarterbacks, ... we are like them in that we can take a position without owning the risk. Though I suspect that at least part of this blog shutdown is driven by the kind of reactions that I describe above, I doubt the Air Force is saying everything that factors into the decision. I'm betting that the AF is also seeing trends like this (from Sinan Eren's recent Blackhat presentation)...


...and reacting to avoid widespread deployment of new classes of botnets on NIPRNet.

Getting Owned Across the Air Gap

I attended a fascinating talk yesterday at Blackhat given by Sinan Eren from Immunity in which he described a recent for-hire Information Operation.

In the talk he took pains to differentiate between a standard penetration test and the kinds of things they were doing; the primary differences being time scale and scope. In this case the time scale was long (though undisclosed) and the goal was compromise of some particularly sensitive data. He didn't say but it was probably product design or source code.

To maintain a stealthy ingress they decided to avoid easily exploited client side weaknesses and instead found something much more difficult to detect, a poorly implemented anti virus scanner on the mail transfer agent. After fingerprinting, building an equivalent MTA in their lab, and coding a unique one-time exploit of the poorly implemented AV file parser, they were in. Consolidation and expansion was done at a leisurely pace, greatly aided by the social engineering benefits of the MTA's access to all of the email traffic. Within a reasonable period of time they were able to relationship map many of the target's personnel, expand to the other side of the firewall, quietly exploit a number of client machines, and gain a good understanding of who was likely to have access to the information they were looking for.

Then interesting stuff happened.

They began to find file references to the stuff they were looking for on a user workstation. The references ultimately ended up pointing to a USB drive that had been accessed at some time prior. It turns out that the target company was running a separate air-gapped internal network where they segregated development or other sensitive activities. However, one of the developers had questions that needed answering over email and was using the USB drive to carry bits and pieces across the air gap so that he could email them as attachments with his questions. Unfortunately, it sounds like the USB drive was used for backup as well and had more than just the snippets on it. After finding, testing, and deploying an exploit that would suck the contents out of the USB drive the next time it was inserted, the attackers just needed to wait until the next time the developer had a question.

Once they had the contents from the USB they ended their IO and reported out to their client. However, had they been dedicated to ongoing operations against the target organization it is not inconceivable that they could have gone further than just retrieving data from the USB. A planful low and slow continuation would probably have kept the USB-copy-and-retrieve going until it had panned out and they were no longer retrieving significant new information. With that vein mined, they might have escalated the level of detection they were willing to risk and try to deploy an exploit across the air gap by writing to the USB drive the next time it was inserted (that's not my idea, it was quickly suggested by members of the audience who sounded like they had a good idea of how it would be done). With a long view, a cleverly designed set of USB-transported exploits, and those occasional sneaker-enabled transits and you've got an effective measured impedance between those two networks of near zero.

Obviously it is interesting that the air gapped network was as vulnerable as it turned out to be. I'm sure a government on government exercise like this would have a lot of people thinking hard about existing assumptions. But perhaps more interesting is the fact that almost all of the exploits used were purpose built for this operation and so were completely invisible to AV signature matching or rules based detection schemes. Well trained, motivated, and aggressive internal analysts with the right tools might have discovered what was going on, but no automated tool was likely to as there would have been no pattern or signature for them to match against.

I am curious how many "typical" intrusion attempts were discovered and warded off during the same period by the target. The "usual stuff", by creating a high noise baseline that keeps defenders feeling like they are accomplishing something with their automated tools would probably help hide the signals of a determined and unique operation such as this one.

Culture in Cyber Command

I love the fact that Gen Lord is talking about cultural differences between the nascent cyber command and the rest of the service. Culture will certainly matter in this space where it isn't just about operationalizing what we already know, but is about out innovating the enemy. The rest of the article goes on to talk about the politics of location. I hope that the impact of location on culture and the ability of the new command to make connections outside of the traditional band of merry metal benders is being considered as well.

Campaigning, Technology, and Supporter Generated Content

I'm accustomed to receiving invitations to donate money to political campaigns but today I received an invitation to participate more directly in a calling campaign as part of Obama's virtualized campaign phonebank. Of course McCain and Hillary have them too. It's not a groundbreaking use of technology per se, but it feels groundbreaking in the political sphere where we are generally expected to passively absorb.

They aren't really doing much more than providing a number to call and a script. Perhaps if they provided an embedded VoIP client they wouldn't have to include this statement about costs (which I guess are essentially contributions that are beyond contribution accounting).


Obama's site is noteworthy for leveraging the power of game mechanics and creating an opportunity to be recognized as an uber supporter.

Though these virtual phonebanks are simple uses of technology, I hope they are a hint of the power of networks to create a more participative future political environment.

FCS, SOSCOE, and the Big Bang

It bums me out to read statements like this one in this article about FCS/SOSCOE:

The software program "started prematurely. They didn't have a solid knowledge base," said Bill Graveline, a GAO official involved in the government's ongoing review. "They didn't really understand the requirements."

That isn't to say that I think FCS/SOSCOE is on track and being developed the best way, it's just that I think these kinds of statements perpetuate the idea that software of this magnitude should be written as a Big Bang after every requirement is fully understood. There are 3000 developers working nine years at a cost of $6B and the expected value curve is supposed to look like this:

Contrast this with something like Linux where value has tracked much more closely with effort:

Why the difference? Well, unlike an open source project like Linux that slowly moves its way up the food chain from departmental web servers to mission critical applications as it matures, the government acquisition system tends to assume that at 8 years 364 days SOSCOE is an entry in an earned value report. Then, suddenly as the calendar turns over 9 years it is hatched as a fully functioning completed system ready for operational deployment. In the meantime, as it hasn't been "delivered" it isn't available to be used anywhere.

I would love to see large scale software developments like this thought of in much more incremental / evolutionary terms. I'd also like to see a greater degree of transparency and openness so that incremental value could be provided along the way, even to completely unrelated programs. After all, many of the component parts of SOSCOE are lego blocks that could be readily used in other environments.

In fact, my first graph is probably completely wrong. Without the hardening that comes from incremental use it is much more likely that the budgeted 9 years / $6B grows dramatically (like it did for Vista) before the value bit can be flipped.

This article made the hair stand up on the back of my neck. What freedoms will we ultimately be willing to trade for our safety?

This isn't normal territory for this blog, but I thought this video was wonderful at conveying how I feel about the subject of torture and America. Waterboarding, rendition, imprisonment without trial, and the like simply aren't consistent with what I always believed to be fundamental American values. I agree wholeheartedly with the gentlemen in the video who claim that our values didn't change on 9/11 and that if we are silent we are complicit.

Net Snooping

This article made the hair stand up on the back of my neck. What freedoms will we ultimately be willing to trade for our safety?

The Gulag Archipelago

This isn't normal territory for this blog, but I thought this video was wonderful at conveying how I feel about the subject of torture and America. Waterboarding, rendition, imprisonment without trial, and the like simply aren't consistent with what I always believed to be fundamental American values. I agree wholeheartedly with the gentlemen in the video who claim that our values didn't change on 9/11 and that if we are silent we are complicit.