Worst Case Scenarios

ecss_game-over-manDespite an endless ocean of advice on what to do (and not to do) in order to avoid enterprise implementation disaster, reality is often an imperfect one. Whether it’s a failure to understand the requirements, secure solid management support, evaluate the options, or install a Locutus equivalent to champion the transformation, mistakes get made. Some are recoverable with effort, others not so much. Ever wonder what happens if you happen to get it all wrong simultaneously, Titanic style?  That kind of thing never happens, right? A Senate Committee on Homeland Security and Governmental Affairs has just proven otherwise and documented the truly horrific failure of the Air Force’s Expeditionary Combat Support System (ECSS) for the world to see. It’s subtitled “a cautionary tale” but it frankly reads with the same hopelessness as a squad of colonial marines holeing up against an entire colony of xenomorphs on LV-426.

Ars Technica provided some coverage of the report’s release including some rather sobering dollar figures. If you’re an American and reading this, you might want to sit down for a moment, pop a few pills, and breathe in deeply, because this is your tax dollars at work:

“When the Air Force cancelled the program in November of 2012, a report issued by the service said, “ECSS has cost $1.03 billion since 2005 and has not yielded any significant military capability. We estimate it would require an additional $1.1 billion for about a quarter of the original scope to continue and fielding would not be until 2020.” Instead, the Air Force decided that it would have to rely on existing systems with minor modifications in order to meet a Congressional mandate for audit compliance by 2017. The service has not been able to certify itself as audit-ready.”

Only corporal Hudson has appropriate words for this dire situation. Well that’s just great. That’s just freakin’ great. What are supposed to do now, man? How about: Congressional powers activate! In the scathing report, the committee pulls no punches to lay down blame for the project squarely on the Air Force’s shoulders:

“The Air Force failed in its procurement of [ECSS]… because it lacked a clear objective and the organizational will to implement changes to its internal business processes vital to integrating ECSS into the organization,” the Senate investigators wrote in the report. “In doing so, the Air Force violated many crucial guidelines and best practices for information technology acquisition.”

Why don’t we build a campfire? Sing a couple of songs? Why don’t we try that? Right now, you probably want to kick the Air Force project governance just so hard. And, admittedly, it’s probably justified. But before you break out the crow bar from the back of the pickup truck let’s look at some of the lofty recommendations by a congressional committee with Hubble caliber hindsight and think about them. You may find their expectations sound reassuring, but oversimplify the problems.

“The Air Force’s failure to clearly define program requirements and effectively communicate program objectives, both BPR tenets, deviated from acquisition best practices, causing massive cost overruns and scheduling delays.”

The bottom line here is the people who remotely knew what the new system would need to do had little involvement selecting and choosing the technologies and vendors. Of course, the labyrinthine governance dictated by the DOD played a role here, and the Air Force gets thrown under the bus for following orders. But just fixing the governance would not necessarily help such acquisition. The core issue is a larger problem centered on the inability to evaluate technologies/products objectively in a timely fashion because of all the long-term impacts that require more prophecy than outright technical expertise. I’ve described this scenario before for Product Lifecycle Management (PLM), and the tenets of that argument hold true here. The amount of effort to execute these evaluations overwhelm even the largest of organizations, so they take the cheap way out with old-fashioned sales pressures. Don’t believe me? The report cites that only 41% of applicable IT systems were acquired using the formal process. There’s a reason for that, guys.

“The Air Force lacked strong, continuous leadership, as called for by Business Process Reengineering (BPR). During the eight years that the Air Force tried to implement ECSS, this program had six program managers and five program executive officers”

This one is priceless. Try staffing anyone on a project for 8 years. Now try staffing someone actually worth a damn for 8 years. Aw, hell no. Here’s a message from Earth: The project was way too long. They tried to eat a whole elephant and choked on the first bite. For the very same reasons waterfall development has fallen out of favor, ridiculously scoped IT moon-shots should be avoided. You’ve just blown the transaxle, you’re just grinding metal.

“Failure to Overcome Cultural Resistance to Change Business Processes Among Air Force Personnel.”

but still

“Forego any modifications to the commercial software”

There is an inherent conflict here; these are diverging objectives. The fundamental problem here is all about industry best practice not fitting everyone. The value proposition of Commercial Off the Shelf (COTS) enterprise suites hinges on a flawed hypothesis: that everyone needs exactly the same stuff. Here’s the thing – to win over end users and win a cultural war, you have to show them something that is awesome. They might not see it from their myopic perspective, but if you can demonstrate it at a scale they understand, they will follow. The problem with COTS is it won’t fit every situation. It’s just reality. Example: an open-minded end user points out situations X and Y for which the best practice has no equivalent. And what if that user has a real point? With strict orders to avoid modifications, you end up telling that user to just deal with it. Even if it doesn’t make sense. Do you think that helps overcoming cultural resistance? You’ve just lost your most open-minded user, and the others will quickly follow in his or her dissent, because it’s no longer a truly objective implementation. Then out come the torches and pitchforks. This is how organizational cultural wars are lost – one user at a time.

A project of this size and scope is insanely difficult. Ripley might suggest perhaps we should take off and nuke the entire site from orbit. It’s the only way to be sure. And that’s pretty much what happened in this case. The lessons here are important not just for the future of Enterprise Resource Planning (ERP), but for all enterprise software. We need more resilient approaches.