Lessons Learned: Army SOA
Written by Chip Raymond
SOA is, however, a new way of thinking about how to use technology. The architecture needed to construct an SOA is such a key critical success factor that without it the entire effort will fail. Although it can appear daunting at first, it is not that difficult, and the money saved by doing it correctly can be significant. Reusability and flexibility are the key features of SOA.
About four years ago, while I was the director of the Army Enterprise Solutions Competency Center, we first started experimenting with SOA and how it could be used. We made our share of mistakes, and what follows are some lessons learned from that experience.
Right off, we learned that SOA was a way of doing business rather than just a technology. Like any enterprise solution, SOA is transformational in form and function. It changes the way you do business and has to be approached from that perspective. So our first task was to overcome the enormous hype about SOA and develop an education program for our leadership so they could be better-informed buyers.
The landscape of products that offer SOA, including the enterprise service bus, is varied and contains anything from a complete suite of tools and functionality to the smallest piece of essential software needed to produce a Web service. There are also standards that have been developed over time that allow for interoperability among the various products.
At first, this was all very confusing, and our lesson was that we needed to have an education program for leadership and management to get everyone speaking the same language. We accomplished this by having a series of senior-level workshops on SOA and publishing a pocket-sized reference guide. Later, we added a Website and content about SOA. All of these educational tools became so popular that it was difficult to keep the reference guides in print.
The bottom line for this experience is that you must have a concerted education program aimed at the business leaders who will undergo the changes required to be effective and efficient users of the SOA as a computing environment. Part of that education is the obligatory warning about getting tied to a single solution. We quickly found that we lost a lot of flexibility by being tied in to a single solution with some proprietary characteristics. In an open-systems environment, this is not advisable.
LIFE CYCLE MANAGEMENT
Our second lesson was learned when we realized that there was no life cycle management model, or methodology, for SOA. When a large enterprise like the Army embraces an approach like SOA, the worst-case scenario is realized when the development environment is chaotic, many different services are being created without any standards, and there is no overarching governance of the transition to SOA.
We felt strongly that an Army-like Life Cycle Management Model (LCMM) was needed. We divided the model into three general organizing categories of activities: selection, adoption and exploitation. We also wanted the LCMM to act as a governance tool for the areas of acquisition, governance, methodology, security and technical aspects of how to design then adopt/maintain SOA. The development of the LCMM took over two years and was vetted by several international standards groups.
The sad part of the story is that the LCMM has not yet been adopted, although a new version will be vetted by OSD-ALT in April. The lesson for us was clearly that we needed to have a network of senior leaders understand the importance of having such a critical organizing document before you attempt implementing this large a transformational approach.
Another important lesson we learned is that there is not a penny of savings in SOA if you just adopt it as a different method of computing. This is also the case with enterprise resource planning. The real benefits in these technologies accrue during the phase we called exploitation.
Up to 70 percent of these implementations fail, and one of the primary reasons is that the customer is led to expect benefits to accrue during the adoption phase. When they do not appear, the project is killed and the technology is blamed for the failure. With very few exceptions, this is not the case at all.
To ensure that benefits accrue—benefits that are dollar-quantifiable and can be tracked—an exploitation phase of the life cycle model must be put in place and followed with rigorous attention to detail. That phase is largely missing when you purchase the technology and what is supplied with the acquisition. Also, what is supplied is not geared to the Army enterprise. The lesson is not to expect benefits until you learn to exploit the SOA’s power to deliver business capability. The benefits accrue rapidly as the network effect kicks in along with the supply and demand for new services.
Perhaps the single greatest SOA lesson we learned early on concerned the data needed to produce the Web service desired. Here we experienced a dramatic and shocking realization: Today the speed at which enterprise data is obtained, and saved, is increasing at an increasing rate. It’s getting easier and faster to save huge amounts of data. There is no end in sight, and as the cost of getting and storing that data is decreasing, it’s becoming a critical issue.
The speed at which you obtain important new information from all that saved data, however, lags far behind. This appears to be because the current software we use to make new information has reached a plateau in functionality, and until there is a leap ahead in technology, we are faced with titanic amounts of raw data and no cost-effective means of generating new information fast enough. We did see very clever ways of visualizing data, but those we considered only marginal improvements.
Another practically insurmountable challenge was convincing data owners to loosen up their death-like grip on the data we needed to build the Web service. We failed most of the time. Sharing data was seen as giving up power and control, and in any enterprise, including the Army, data is protected by a nearly invincible bureaucracy. Yet without free-flowing access to data, the SOA will not work.
One way we did enjoy some success was by convincing data owners that what we really needed was an information service rather than direct deep access into their databases. The idea was that we needed information that they would certify as authoritative. The information would be published as a service and could include multiple data fields or attributes. We were not concerned where this information came from as long as it was authoritative. This allowed the data owners to protect their sources of data and maintain ownership, change it around, or any other activities they needed to perform.
As long as they provided certified authoritative information (clusters of data), everyone was a winner. Attacking the culture of data ownership, control and power was a losing effort. Perhaps someday this will not be such a tedious endeavor. The Army could take advantage of the pending migration to APCs in the Network Service Center of the future by limiting it to authoritative data only.
CHIEF DATA OFFICER
During this period, the concept of a chief data officer (CDO) for the Army surfaced as serious consideration. We set out to find one and invite him or her to address one of our senior leader workshops. We were successful in getting the CDO of a multinational financial services corporation to speak at our workshop and were pleased to hear about the challenges and opportunities he was faced with in a global 24/7 operation. The common theme again was data ownership and sharing.
The CDO’s greatest challenge was the same as we had in the Army. Data is power, and control of it is tight and parochial. Their simple solution was to migrate data ownership to the CDO through the power of the purse. However, we did not think that would work in the Army because of the institutional resistance to sharing across the enterprise and protected budgets.
There had to be another way to gain control, and that appeared at first to be through standard enforced data architecture and a governance mechanism. The Army now has a data center of excellence and is doing the needed hard work of building this critical part of the SOA architecture, which is currently referred to as the Army Data Service Layer (ADSL). This is an outstanding effort.
There are a lot of other lessons we learned over the years we worked with SOA. Some of them were quite humorous. For example, the enterprise service bus (ESB) took on the patina of a silver bullet. It was viewed as the star of the show, while the truth was that you really didn’t need an ESB for some of the work on an SOA. Some organizations even bought ESBs, lit them up and then tried to figure out how to use them. That was the absolute wrong approach and resembled the old problem of solutions wandering around looking for problems.
At the same time there evolved warring camps on just how many buses the Army needed. The hype was unbelievable. Our position was it took as many as it took, but some thought one bus could handle the entire Army. Most of these issues were sadly political and difficult to attack. We thought in the end that what they really meant was one bus architecture (there are different architectures for ESBs). It would be nice if it were that simple, but it isn’t, and in the end the Army will likely have a federated SOA, along with a federated data architecture, with many ESBs of various configurations throughout the enterprise.
One of the rules I set for our SOA lab was that everything had to comply with accepted standards in a plug-and-play heterogeneous environment. I wanted see if this could lower the total cost of ownership and increase the flexibility in a plugand- play scenario. We thought that would give the Army the benefit of not being tied to a homogeneous solution with an expensive price tag. I was not popular with some in the vendor community.
Web services are interesting machines. If you build them correctly, they can be used across the enterprise as a shared service. It makes no sense at all to build multiple services that do the same thing. For example, “to order” and “to pay” for something should be a single shared service across the enterprise. We wanted to prove that you could have shared services work together to produce a positive outcome as an enterprise service.
SHARED SERVICE SAVINGS
We were successful, and it proved to us that a significant effort should be engaged in determining what were the major shared services in the Army, who should own them, and who should pay for them. It’s no stretch to imagine the amount of money that could be saved if this were done. We never did get answers for those questions, but they remain as major considerations.
I’m now a chief technology officer for a large systems integrator, and the Army is our largest customer. I came to the job well armed with the lessons learned in the past, and now my effort is spent making sure that our solutions are well thought out and focused on cost effectiveness.
The SOA lessons learned here are not dissimilar from those I learned in the Army. I have a very large pool of experience to draw from our various centers of excellence. The advanced research and engineering going on is amazing, and the industry partnerships are all focused on doing what is right for our warfighters. And in every case the lessons learned are being fed back into the engineering required to provide solutions.
The lessons here are dramatically straightforward. You must have a strong governance structure; the scope must align with the business strategy; change management is critical; business leaders must lead the effort; a solution architecture is key; organizational culture can defeat the project; and leadership must be engaged in all facets of the transformation. Without these key enablers, the SOA business transformation will fail. ♦
_______________________________________
Chip Raymond is chief technology officer for Army programs of the Defense Division of CSC.
