For example, a number of REST advocates are declaring victory because of the RESTfulness of the Bloglines web service APIs. This is a bit like Schwinn declaring victory whenever someone in a sunny climate buys a bicycle rather than an armored car to commute a mile to work. It's not all clear to me why anyone at Bloglines or elsewhere would even think of using the WS-* technologies to do this job. After all:

- The sources of information are generally known to the consumers by their URI, and can easily be discovered with mechanisms such as Google.

- The information being aggregated is public and not particularly sensitive, i.e. there would be little plausible benefit from attempting to steal or mis-attribute it.

- The information is already in XML form (or something very close to XML, net some misunderstandings of the finer points of the spec).

- It is already on the web, directly accessible via HTTP.

- The consumers of the API are experienced software developers, who presumably understand HTTP and XML natively and don't need a hand-holding "import a description of the API and generate all the code" tool to get the job done.

So, this is a job for which HTTP+XML is very well suited, and for which WS-* technologies add very little potential value. As a loyal Bloglines "customer" [just what is their business model anyway????] I'm happy to see them use the right technologies for this job. One might contrast this to Google, which drank deeply from the web services koolaid pitcher when developing its own Web API a couple of years ago and doesn't seem to have created much real value with them.

Let's consider, however, another scenario in which some business or agency needs to present an integrated view of diverse information sources, but has additional requirements:

- New sources of information must be discovered in real time with minimal human intervention.

- The information being aggregated is sensitive and confidential, so must be encrypted, digitally signed, and contain access control assertions.

- The systems from which information is gathered include mainframes, and some communicate with the outside using industrial strength middleware such as IBM's MQ technology or Software AG's EntireX rather than HTTP.

- Many of the communications links involve multiple hops, some of which go over intrinsically unreliable networks.

- The effective address of some information sources is discovered dynamically as part of a multipart interaction rather than via a published URI, and that address needs to be

- The information needs to be incorporated into a variety of off-the-shelf client programs by people with only basic programming skills.

OK all you folks who love to point out that HTTP+XML meet the needs of 99% of your customers, how do you meet the needs of the very real organizations that do have these kinds of requirements? Chances are you'll end up inventing something that is at least as complex as the WS-* stack, and you'll not get the benefit of the dozens of person-years of discussions about trials, errors, pitfalls, nitpicks, clarifications, and so forth that have gone into these specs. Nor will you get the benefits of the network effect from all the real products that support WS-* by rolling your own solution to these requirements.

I have no stake in the WS-* stuff -- it was developed outside the W3C system in which I spent a lot of time for the last several years, and I haven't been to any of the review workshops. Nor will I argue that WS-* will give market success to those who have bet on it, or that it hits the 80/20 point in the tradeoff between functionality and complexity. All I will argue is that a) it is designed to address real problems faced by real organizations who have tried to integrate enterprise systems with Web technology, and b) there is an immense amount of intelligence, thought, and real experience distilled into these things that needs to be given a lot of respect. If people in the REST camp want to actually change minds rather than exchange virtual pats on the back in their echo chamber, they'll have to explain how to do the hard things RESTfully, not belabor the pointlessness of doing easy things with WS-*.

It's a nice day, maybe I'll ride my bike to the public library at lunch .... but that doesn't mean you shouldn't use an armored car if you need to haul cash around Newark on a rainy night.]]> 2008-01-02T17:42:16Z 2004-09-19T21:35:18Z tag:weblogs.java.net,2004:/blog/mchampion/37.135 2004-09-19T21:35:18Z

Of course, the mainstream XML community lives in a glass house and should be careful about throwing stones at the web services people. Tim mentions the XML Schema spec as a victim of Design by Committee, and XQuery is often mentioned as another example; of the thousands of pages of XML-specs coming out of the W3C in the last few years, how many are really based on best practice? The Semantic Web specs are an interesting case; they contain the best practice that has survived over several generations of theory (LISP parantheses mutating to XML angle brackets along the way!), but I have no idea how truly field-tested they are. I'm actually becoming somewhat optimistic that we're seeing the inverse of what Bray talks about there -- people are discovering that they have been stumbling around the concepts that are captured and formalized in OWL, and are taking a look at how to apply them to domains that nobody thought of as needing an "ontology". We may be seeing one of those rare cases where academic theory successfully does precede practice.

The metaphor I like here is "intellectual capital." The internet and the Web looked like overnight successes (and committees fairly quickly established the core Web standards) during the 1990s, but they drew on at least 20 years of academic and government R&D. The industry seems to have learned the wrong lesson, believing that it was the committees writing the specs that created the intellectual capital upon which the Web was built, and hoped that more committees would keep the momentum going during this decade. I suspect that the capital formation occurred in a lot of research labs and email lists and conferences over a decade or two, and the standards committees just refined and refactored out what was relatively common.

The community of web services developers should certainly keep doing what they're doing to lay down the next layer of intellectual capital, and not be deterred by the skeptics and fundamentalists who think that XML+HTTP will always suffice. Consumers of this stuff, however, should be extremely wary, realizing that WS-* is about the future, not for today. People with real projects to plan simply have to judge for themselves which of these approaches and technologies are Best Practice (ideally with a solid theory behind it), and which are Best Guesses of self-appointed experts. My take on today's reality is very much in line with Tim's -- "XML, HTTP, URIs, SOAP, WSDL, and that’s about it."

Adam Bosworth of BEA delivered the opening keynote address on Wednesday. He started by reminding us of the dream that XML geeks shared back in 1998: Information should not get lost in presentation. Actual XML practice has to some extent diverged into two separate streams -- documents on one hand, and application data on the other -- but together they have helped take back the world from the "hideous complexity and fragility" of information presented in .DOC, .EXE, etc. files.

I started to summarize the talk itself, then came upon Matt May's excellent near-transcript . I'll just elaborate on a few points that resonated with me.First, Bosworth notes that one of the negative aspects of the current XML world is that the KISS (Keep It Simple, Stupid) principle is being widely ignored, as XML and Web services technologies are becoming extremely complex. Bosworth has pointedly noted the complexity of W3C XML Schema and XQuery in the past; this time he said something like "there's only one guy in our company who *really* understands WSDL" ... and "we don't need all these layers of coordination/orchestration specs on top of SOAP, we need something like a 'SOAP cookie'." It's heartening to see more and more people pick up on the theme that the XML family of specifications is too complex, since I've been beating this drum for a long time now. It's definitely time for some serious refactoring of the the family of XML specs, and that won't happen until more people of Bosworth's stature start telling the unpleasant truths about what a few thousand person years of Design By Committee will do to a technology that used to be simple. The video clip from Bosworth's presentation in Jon Udell's weblog has my favorite quote from the talk: After admitting that 50 people would come up with 75 different ways of building distributed systems ."Some will win, some will lose. That's just fine. It's called evolution. It works really well. It's why we're all sitting here today." Maybe a bit of Darwinian Refactoring is what is called for on this stuff.

Another major theme in the keynote was that XML developers are asked to use "APIs from Hell." For example, a programmer working with a purchase order in XML format must deal with events, or child/sibling nodes in a tree, rather than application-level concepts such as products and quantities. Hmm, that's a posting in and of itself, because it ties in with a town hall meeting on storing/querying XML that turned into a discussion of XML APIs. More later.

Probably the most unconventional topic Bosworth spent time on was the importance of getting XML data models and APIs suitable for handling the synchronization of intermittently connected devices to Web-based master databases or applications. He noted that he spends much of his business week using only his Blackberry device. Effective use of such web-enabled, but slow and UI-challenged devices will require better synchronization tools: queries are difficult to generate with a handheld UI, and their limited bandwidth (if connected at all) means that it is important for queries to be very optimized to return back only the information the user really wants. Since this is so far beyond the state of the art, it may be easier for the device to anticipate the types of data the user will want, and trickle than information into the device in advance, as bandwidth is available, rather than on demand.

Bosworth presented a slide outlining a "Mobilized Data Model" that might help guide work on this. I suspect that many other listeners were also intrigued, but a bit mystified by this-- a planned demo wasn't ready in time for the conference. It has, however, generated some interesting discussion on the Web, not so much on the still-fuzzy details of the data model, but on the deeper issues. For example, Tim Bray notes that he is skeptical of the basic assumption behind the widespread need for synchronization, since "the trend is clear: anyone who wants to will be able to have a fast pipe that's always on." I'm not sure which side I come down on, but there is definitely two world views here: Those who assume that there will be a significant amount of data on the handheld device that must be synchronized with a central repository, and those who assume that in general the remote device will be able to query the central repository for the data it needs for a given task. Bosworth did offer one data point that might argue against Bray's position: At best, in the best-served places in the world a GPRS (General Packet Radio Service -- used for "always on" connection to the Internet by mobile devices) request takes 1 second to fulfill, and more if any significant amount of data is transmitted. Unless we get a WiFi network that is as extensive as the GPRS network today, it's not clear that the "fat pipe" assumption is realistic.

Another point that Bosworth has explored, not so much in his speech but in his weblog. How can one address the difficult challenge of synchronization without falling into the trap of complexity that will not scale to the Internet, or even work on a limited power and bandwidth device? He's asking a lot of questions about REST in the weblog, getting lots of answers, but one gets the impression that they are not satisfactory.

Inspired by a posting by Vanessa Williams, I'll put in a plug for the ideas behind JXTASpaces (sortof tuplespaces/Javaspaces using XML rather than objects) as a way of bridging the gap between the web services ideas that Bosworth talked about and the REST stuff that intrigues him. (Williams doesn't make the link to Bosworth in the posting, but has done so privately) . Kimbro Staken picks up the thread and mentions how an XML DBMS that supports XPath can make the template lookup feature of tuplespaces easy to implement if XML documents are the "tuples." I think they are definitely on to something here -- See Robin Cover's summary of some technologies and discussions, including some comments by me. The one point that's most relevant to synchronizing mobile devices is that coordination via "spaces" allows loose coupling in time as well as space -- not only can components in a distributed system employ different platforms, languages, and native data formats, they don't even have to be running at the same time. On the other hand, product ideas such as Ruple went nowhere and even the open source JXTASpaces project is a not exactly thriving. Is this just an idea whose day has not yet come, perhaps like hypertext before HTML? Or is there not really a "there" there? I'm pretty sure that if any application domain is tailor made for an XML Spaces approach, it's intermittently connected mobile devices!

Finally (just when I thought I was finally done blogging this speech!), Joe Chiusano asks about the apparent contradiction between Bosworth's focus on simplicity and BEA's co-authorship of a somewhat daunting list of Web services specifications. It will be interesting to see if Joe gets an authoritative answer from BEA; my guess is that they are telling it like it is in a recent press release that suggests [at least in my reading between the lines!] that customers are giving them the message that the current way of doing things is too complex.

Jon Udell gave a keynote speech on Tuesday that pierced the jaded, slightly cynical shell I've acquired after about 8 years in the XML world. He didn't talk about "maybe someday..." or "if only ...", he showed what a little imagination can do with the widely deployed XHTML, CSS, and XPath technologies today.

He began with an explanation of why the idea of an XML document that can be shared by different application is "magical," using two speeches by Bill Gates to illustrate the point. 10 years or so ago, Gates' slogan was "information at your fingertips," in which information workers could bring together text and data from a variety of applications and easily find, reuse, and revise it. The vision was a powerful one, but we are only beginning to realize it today. Part of the reason for the delay, Udell argued, was that in the OLE architecture that Microsoft promoted (and in the OpenDoc architecture of its rivals), each application owned its own proprietary data, and OLE/OpenDoc provided a means by which they could talk to each other. This worked to some extent, but created tight couplings (I'm not sure if Udell used that term) among the applications, object models, and operating systems and programming languages that support them.

In a more recent version of Gates' stump speech, the key idea is a "universal canvas" enabled by XML. Rather than applications communicating to access each others' proprietary data, they share a common XML representation of the data -- some may be numbers manipulated with a spreadsheet, others text manipulated by a wordprocessor, still others structured data manipulated via a forms interface. This vision is already very close to reality in Microsoft's Office 2003 product (and Udell's keynote was followed by a presentation from Adobe that demonstrated their very similar vision in action).

What makes XML a good "universal canvas"? In Udell's words, "contextual metadata is XML's gift to mankind." To demonstrate this, he showed how he had been eating his own dogfood in the presentation slides, which were being displayed in Mozilla on a Mac, produced with ordinary XHTML, with CSS styling, and some script. He showed some queries that exploit Mozilla's XPath support on data that wasn't really designed to be queried, but exploits the metadata that XHTML authors create without thinking about it -- links, attribute values put in to support CSS styles, etc. When the idea of XML-defined context is more explicitly supported, as in Kimbro Staken's Syncato weblog tool (or is it an "XML fragment management system?"), the power of XML's ability to model context and XPath's ability to query it becomes even more apparent. In any event, knowing that one will use XPath to retrieve information in the future motivates one to think about the structure of content as one writes -- creating the contextual metadata magic without much extra effort.

Udell went on to talk about how these advantages can be extended to email and instant messages, in which most real business communication takes place (and thus where the real content is created). We need to build tools and products that tap the small packets of structure and context and pull it into useful business information. Although Microsoft Office's XML support does not extend into Outlook, there are ways to archive email and IM in XML, and Udell challenged the audience to help devise real products and integrations to exploit the potential this would create. In any event, we don't need new standards, or a solution to the thorny issue of the diversity among flavors of RSS/Atom, to make progress -- XHTML, CSS, and XPath provide the basics of what we need. Udell argued that we need to figure out how to use what we have in a smarter way, and to "smuggle" metadata into content because it leverages the universal urge to make things look cool.

So why did this pierce my cynical shell? Most would agree that we need more metadata on the Web for it to live up to its full potential -- that's the very premise of the Semantic Web effort in which Tim Berners-Lee has invested much of the W3C's resources (and credibility). On the other hand, the historical difficulty of getting real people to put metadata in their content is believed by many to doom such efforts to failure. (Cory Doctrow's essay is the most colorful and cogent, if widely reviled, statement of this position). Udell's insight is that we can leverage the technology we have, salted by human vanity, to get usable metadata without technological breakthroughs or unrealistic demands on humans.

In 1993 or thereabouts, we saw the emergence of TCP/IP, HTML, HTTP, Mosaic and the Web. From our perspective, all of these were simplistic emulations of a tiny subset of what we'd been doing in Notes for years. TCP/IP instead of Netbeui or IPX/SPX, HTML instead of CD [Lotus Notes "compound document" ] records, HTTP instead of the Notes client/server protocols, httpd instead of a Notes server. And we were many years ahead in other ways: embedded compound objects, security, composition of documents as opposed to just "browsing" them, and a sophisticated development environment. I am quite embarassed to say that we frankly didn?t "get" what was so innovative about this newfangled "Web" thing, given the capabilities of what had already been built.

So why did the "simplistic subset" succeed so much more dramatically than Notes? Uhh, probably mainly because it is a simplistic subset! Consider the argument of Clay Shirky

The very weaknesses that make the Web so infuriating to serious practitioners also make it possible in the first place. In fact, had the Web been a strong and well-designed entity from its inception, it would have gone nowhere... Only solutions that produce partial results when partially implemented can succeed. The network is littered with ideas that would have worked had everybody adopted them. Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen.

Put differently, the Web succeeded because it hit the famous "80/20 point" -- it gave most of what other solutions offered, but at a much lower cost and complexity.

Joel Spolsky has a rather different take on this.

A lot of software developers are seduced by the old "80/20" rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. Unfortunately, it's never the same 20%. Everybody uses a different set of features.

So, the 80/20 rule rules sometimes, but not always. Obviously, it doesn't rule when different groups need a different subset of the features. More generally, it doesn't appear to rule at the application level -- the masses seem willing to keep pour money into Microsoft's jaws to get the latest version of Office, when far fewer than 20% use features that weren't in Office 97. OpenOffice, which appears to be at about the Office 97 level of functionality and is free, has a miniscule market share. (I'm part of it, so don't flame me!). A few hypotheses:

• Minimalism ( fundamental simplicity) matters more at the infrastructure level, where it really really matters how universally something is adopted, how reliable it is, and how well it performs.
• Minimalism ( fundamental simplicity) matters more when standards are involved.
• Ease of use (simplicity of operation) is obviously critical when you are dealing with a large, relatively unsophisticated audience. The early Web didn't have to worry about this, and by the time it took off it had a "cool factor" that overcame a lot of resistance. One can't count on that happening again anytime soon!
• "Good" software, at both the infrastructure and UI levels, manages to maintain the "illusion of simplicity" by hiding the ugly details while still providing most of the functionality that one immersed in the details can employ. That's hard. Office 2003 is just starting to approach this.
• Beware of the argument that complexity isn't a problem so long as it is hidden. True enough, in the abstract, but only if it is the detail being hidden and not the complex concepts that an API or GUI is simply ignoring. For example, XSLT is a powerful tool in expert hands, but AFAIK none of the GUI stylesheet generators go much beyond the basics. The same is often said of IDEs.

If analysts are already inviting themselves to the funeral of the XML DBMS, maybe we should check to see if the departed is lying quietly waiting to be buried. Perhaps like the unfortunate Lal Bihari of Uttar Pradesh in India, it been declared dead by those who would prematurely appropriate its inheritance. Here's a few arguments for their continued viability:

• It is widely believed that less than a quarter of enterprise data is currently stored in RDBMS systems. This suggests that the market is not "making do" with what the relational database products offer today, but using a wide variety of technologies.
• The main reason OODBMS didn't hit the sweet spot, AFAIK, is that they created a tight coupling between application code and the DBMS. Potential performance gains this allows can outweigh the maintenance challenges in extremely business critical, high transaction volume environments (e.g. where IBM's IMS or Software AG's Adabas DBMS still thrive), but not in the kinds of innovative, rapidly evolving environments where OODBMS got a foothold.
• XML DBMS, on the other hand, inherit XML's suitability for loosely coupling systems, applications, and tools across a wide range of environments. Phil Howard mentions "integration" as if it were a niche. That may be true in a business sense, but the ability of the XML DBMS to efficiently, scalably, and reliably power the intrinsic integration-friendliness of XML is likely to find a wide range of uses outside the "application integration" business space.
• Again AFAIK (having only played with OODBMS personally), there is relatively little portability across OODBMS systems; code written for one would be very expensive to adapt to another. Investing in the technology required one to make a risky bet on the vendor who supplied it. This created an environment where the object-relational vendors could prosper by offering only a subset of the features but the absolute assurance that they would be in business for years to come. In the XML DBMS world, on the other hand, all support roughly the same schema, query language, and API standards;
• The standards of the XML world provide a clearly defined and fairly high bar for those who would seek to take away the market pioneered by the XML DBMS vendors. For better or worse, the XML family of specs is complex and quite challenging to support efficiently in a DBMS system. It's one thing to support, as the RDBMS vendors now do quite well, XML views of structured, typed, relatively "flat" data such as are typically found in RDBMS applications. It is quite another to efficiently and scalably support queries and updates on "document-like" XML with relatively open content models, lots of recursion, mixed content, and where wildcard text comparisions are more frequent than typed value comparisons. The dominant DBMS vendors obviously have talent and money to throw at the problem, but analysts should not assume that they will surpass theese capabilities of the XML DBMS systems anytime soon.

This is not to predict the demise or even decline of the one-size-fits-all RDBMS+kitchen sink products from the dominant vendors. Microsoft's Longhorn project apparently hopes to handle much of the other 75% in an RDBMS-powered but XML-capable replacement for the filesystem. As much as one can admire this vision and expect it to be at least partially successful, there will be an awful lot of "pure" XML data out there, partly due to Microsoft's other efforts. A substantial number of developers and end-users will continue to need "pure XML" DBMS technology that handles XML extremely well, even given the availability of DBMS technology that handles all kinds of data in a "good enough" way.

Finally, it's important to consider the gap between technology mindshare among pundits and analysts, and technology deployment by businesses who need to get the job done efficiently and reliably. I'm reminded of another technology analyst who has dismissed XML itself by pointing to its similarities to IBM's IMS product, a "discarded in practice" hierarchical DBMS. So IMS is also dead, eh? Let's see ...IBM says "IMS serves 200 million end users, managing over 15 billion Gigabytes of production data and processing over 50 billion transactions every day." Another corpse that won't quietly play its part in the funeral the analysts have thrown for it!

I'm not sure when pure XML databases will be managing exabytes of data, but It looks to me like XML DBMS's will join IMS and Mr. Bihari in the ranks of those who continue to lead active and productive lives despite having been declared "legally dead" by the presumed authorities.

Lots of people get annoyed that it takes a group so long to do what any competent individual could do more quickly and cleanly. The point is, uh, standardization. Sure you, or your favorite tool's vendor, can define terms, formats, APIs, etc. quicker and in some ways better than a committee can. There's a bazillion jokes on the basic theme "there would be so much less discord if the rest of you would just do it MY way! But then you usually end up with dozens of ways of doing the same thing, and that invites gravitation toward the biggest vendor in a space..

Standardization is hard -- it requires people with different perspectives and interests to (usually very laboriously) figure out what they really have in common and what that implies for the standard terminology (or data format, or API, or whatever). Even I (after 6 years of spending about half my working time on various W3C groups) am continually amazed at how hard it is to get agreement on seemingly trivial issues. Uhh, that's usually because they only look trivial ...

Standardization is messy. There plenty of jokes about standards and sausage making, along the lines of "those who appreciate sausages or the law should not watch either being made". Personality / politics is a big part of the process, don't let anyone tell you otherwise. But that's life: Lots of profound accomplishments were achieved by highly flawed personalities, groups, and processes. Just picking the first examples that come to mind, Isaac Newton was a seriously disturbed man and quibbled constantly about his "intellectual property," but he built the foundation for modern science and math; the politics that produced the US Constitution were pretty intense and gave the document itself some embarassing warts, but the overall result was extremely successful; and the interpersonal processes in the group that discovered the structure of DNA were pretty ugly, but again the result was profound. (Oops, I see that Tim Bray made this point better than I could while I've been fiddling with this article).

The result of a standards process is rarely elegant. Here there is one canonical joke: "A camel is a horse designed by a committee." The analogy between racehorses and camels is actually a good one, but I'd turn it around: If you want to go fast around a well-prepared track, get yourself a racehorse. If you want to reliably reach a distant destination across unknown terrain, you are probably better off on a camel. Those committees can be sortof surrogates for evolution, pruning out the beautiful but high-maintenance features and insisting on the "humps" that look strange but add real value.

As much as I sometimes wish that we could somehow avoid all this, it's increasingly clear that well-defined specifications (not universally adopted legal standards!) are increasingly necessary. Some argue (see the RSS article linked above) that a formal standards process primarily benefits the big vendors who can invest in it and promote their own technologies. That's true enough, but the alternative is even worse: without rigorously defined criteria for what a "valid" document, message, or API is, the dominant vendor's implementation becomes the de facto standard. Furthermore, you can hardly benefit from test driven development if the tests themselves are best guesses based on what works with the dominant vendor's implementation; you'll never know if the test is right and your code is wrong, or whether you're simply introducing "bug for bug compatibility" that will hurt you if the original bug is fixed.

So what is to be done? I don't presume to think that much can be done to change the standards processes, but I do think that consumers of "standards" can adjust their attitudes in ways that work for them in the short run and the overall Web standardization culture in the long run.

1. Ask yourself "can I reject documents that don't conform to the standard" before worrying too much about the details. The numerous variants and decendents of "RSS" are a case in point. I'm not going to ignore a source of news or opinion if they don't conform to one of the many alleged standards in this space, I'm going to use the most "libaral" tools that manage to extract useful information. But in other cases -- especially those involving money or security, or when transaction volumes are very high -- validity of a message against an agreed-upon standard is a prerequisite for accepting it, and the details matter very much.
2. Judge the specs on their results, not their inputs or processes or the legal standing. The SAX XML API is a case in point -- it has no official standing whatsoever, but is so simple and useful that it is almost universally supported. That makes it more of a "standard" than something like ISO DSSSL or W3C XLink that simply haven't generated much of a network effect. If, somehow, the messy politics of the content syndication or web services sectors result in specs that are clean, useful, and widely supported, then the process has succeeded despite itself.
3. Vote with your feet for or against specs based on whether they solve real portability and interoperablity problems. Vendors don't really have the luxury of ignoring official standards, but users do. (Actually, recent specs are more likely to have "interoperability profiles" or "minimum conformance levels" that reflect the business realities in the field rather than the political realities in the conference room, but that's another essay!).
4. Accept that diversity, competition, and evolution is inevitable. No matter how convenient it would be for there to be one universal, stable standard in each domain, it ain't gonna happen, fuggidaboudit. In the early days of a technology, premature standardization is a worse problem than confusion. When a technology is mature, the way forward often involves taking a few steps backward. ( The Innovator's Dilemma has numerous examples that illustrate this). Knowing when it's time to non-standardize or un-standardize to allow innovation to flourish is as important as knowing how to create consensus and standardization when it's time for the network effect to work its magic.

At least one group of analysts more or less agrees, making the additional point that "the whole point to Web Services is to be interoperable. 'Fragmented Web Services standards' is an oxymoron that makes no business sense. And if there's no money to be made, then companies won't do it. " ]]> 2008-01-02T17:42:16Z 2003-07-07T15:05:18Z tag:weblogs.java.net,2003:/blog/mchampion/37.69 2003-07-07T15:05:18Z

Another thing that did not inspire confidence was that many people who profoundly disagree about lots of other things all seem to agree that Service Oriented Architectures are a Good Thing. If the term is broad enough to include "everything", is it meaningful enough to exclude anything?

I've overcome a good bit of this cynicism lately. Sure, SOA is a kindof fuzzy concept, but after spending much of the last year trying to rigorously define what a Web service really is and is not, I'm more sympathetic to the recursive use of "servce" in these definitions. Likewise, a lot of the discussion blurs the distinction between the definition of SOA and the characteristics of good architectures, service oriented or otherwise. For example, "loosely coupled" and "standards based" are considered two of the pillars of SOA". These seem more like principles of most successful architectures than principles specific to SOA. But there does seem to be a very powerful idea buried down in the current SOA hype: the idea that the central abstraction is this fuzzy thing called "service."

It's hard to define what a "service" is -- it could be some software component written in an OO language and deployed via an application server, or some "glue" exposing a legacy mass of COBOL , or a plain vanilla Web server providing the, uhh, service of serving up documents. But it's a lot less hard to talk about how one invokes a "service" -- you always (I think!) send it messages. So, to implement a "service," take some discrete chunk of functionality and provide a messaging interface to it. All that consumers of the service have to know to use it is the format of the messages they send it and get back from it, the protocol(s) for transmitting the messages, and the "message exchange pattern" of the interaction between the service provider and service consumer. In other words, the critical definition of a service is its interface to its consumers, not any particular characteristic of its design, implementation, deployment, etc. One can develop SOAs with COM, CORBA, BEEP, JMS, proprietary stuff, XML, HTTP, SOAP, and all sorts of combinations of these and many other technologies. One can describe the service interface using some combination of IDL, WSDL, an XML schema language (there are several!), DAML-S or some other ontology language, or if all else fails, a phone conversation.

This, in my mind anyway, explains why so many people who disagree on other things tend to think of their architectures as "service oriented." What they're disagreeing about are things like whether to encapsuate the service invocation messages behind an API or expose the messages directly to the service consumer, whether to handle reliabiility and transaction integrity down in the service infrastructure or up in the application layer, whether to have the service semantics described by human-readable documents or machine-processable ontologies, and so on. But if one looks at the messages, exchange patterns, and data exchanged, a lot of these differences tend to blur into the background.

I think this view of distributed systems architectures helps maintain focus on what is important for scalability, interoperability, evolveability, etc. and prevents us from becoming distracted by extraneous and politically loaded concerns. For example, there is no intrinsic conflict betweeen REST and SOA or SOAP. One can and should debate the specific conditions under which it makes more sense to directly manipulate "resources" by transferring representations of their state using the "services" supplied by HTTP, and when it makes more sense to hide the resources behind a more complex and customized service interface, but let's not call these alternative "paradigms". They're just different flavors of SOA, and developers are free to mix and match them to meet the needs of a distributed application.

First, as of SOAP 1.2 (very soon to be a W3C Recommendation) there is no conflict between SOAP and "having a GETable URI." SOAP has a web method feature:

Bindings to HTTP or such other protocols SHOULD use the SOAP Web Method feature to give applications control over the Web methods to be used when sending a SOAP message.

In short, SOAP 1.2 allows and encourages the use of HTTP GET to invoke services that simply return data, thus allowing one to hyperlink to SOAP services, exploit HTTP caches of the results of frequently-accessed services, etc.

But, one might ask (and people very frequently do!), what benefit does one get from SOAP if just GETing or POSTing XML data works just fine in your application? One very reasonable answer is "not much." Sure, once WSDL 1.2 (which wll support the web method feature in SOAP) is out we can expect tools to make it very easy to generate programming language code to invoke "GETable URI" SOAP services, but that is in the future sometime. A better answer, I think is that SOAP provides an architectural foundation for extended services as requirements evolve.

For example, "raw XML over HTTP" works just fine for simple services over a secure intranet (or the public internet when SSL provides adequate security). But what happens as the requirements on the service creep upwards and one must: support:

• Routiing and reliable messaging across multi-node networks, such as when one must perform content-based routing from an HTTP gateway to the appropriate back-end service (e.g., the one nearest to the consumer).
• End to end encryption (from consuming application to service rather from consuming application to SSL gateway)
• integration of legacy services that may not have an HTTP URI
• Non-HTTP communications protocols and interfaces such as BEEP, MQ and JMS
• Multipart service transactions that must be committed or rolled-back/compensated as an entity
• More complex interactions between service suppliers and consumers that need to be described and choreographed.

So, if you are likely to have some of these challenges, building SOAP into the architecture lets you leverage emerging technologies such as WS-Security, WS-Routing, one of the several reliable messaging proposals, WS-Transaction, WSBPEL, etc. that build on the SOAP infrastructure.

If, on the other hand, you are saying to yourself "that's WAY more complicated than anything I want to do," then SOAP may not be for you. Likewise, if you are saying "I need to do that stuff MY way rather than the way some industry committee's way," then perhaps you are better off putting this stuff into your own application rather then buying an off-the-shelf solution, and SOAP doesn't offer anything you need. But this is something to be analyzed -- it's just as "wrong" to blindly reject SOAP as to blindly accept. it.

Still, at a more fundamental level I think Sean and I agree on the basic architectural principle here :"I have learned to really appreciate the power of the "distributed systems as stateless document exchange," which I believe lies at the heart of what makes HTTP so stunningly successful. " It often makes more sense to think of XML-powered applications as transformation pipelines (see Propylon's PropelX and Software AG's EntireX Mediator as pioneering tools that support this approach) rather than objects to be exposed with APIs. More on this in a future edition ...but in either scenario, SOAP may add enough value to justify its use.

To start things off, there has been a spate of articles lately related to something I've wondered about for a long time, namely the relationship of the world of objects to the world of XML. On one hand, the OO principle of "data hiding" suggests treating the details of data formats as implementation details to be hidden behind access methods -- relates to the world of XML. The whole point of XML is to expose the data rather than to hide it. Norm Walsh emphasizes the point that "XML is Not Object Oriented," pointing out that:

the constituent elements and attributes of an XML vocabulary are not generally related to each other by inheritance, nor do they naturally correspond to objects with any kind of precision.

On the other hand, XML makes an awfully handy serialization format for Java objects.Also, there is a generic XML object model that has much support in the Java world and provides a useful, if very low-level, set of methods for working with XML documents.

So, as the old joke goes, XML really is a bit of a "floor wax and a dessert topping." The trick is to know which approaches, tools, APIs, etc. are best for which applications. That is a subject we will explore in subsequent editions of this weblog.