Michael Champion's Blog

Web Services and XML Archives

Bicycles, Trucks, and web services specs

Not surprisingly, peace has not broken out in the ongoing dispute over web services specifications described in my last post. The continuing attacks on the WS-* family of specifications by advocates of simply using HTTP and XML together somehow reminds me of a bicycle company positioning their products against those of an armored truck manufacturer: "It's more agile in traffic, it gets an infinite number of miles per gallon of petroleum fuel, it's better for the environment, and it improves your health!" All true, of course, but completely irrelevant: you can't do the secure heavy hauling work that the truck is designed for with bicycles, except perhaps with extreme ingenuity and monumental inefficiency.

For example, a number of REST advocates are declaring victory because of the RESTfulness of the Bloglines web service APIs. This is a bit like Schwinn declaring victory whenever someone in a sunny climate buys a bicycle rather than an armored car to commute a mile to work. It's not all clear to me why anyone at Bloglines or elsewhere would even think of using the WS-* technologies to do this job. After all:

- The sources of information are generally known to the consumers by their URI, and can easily be discovered with mechanisms such as Google.

- The information being aggregated is public and not particularly sensitive, i.e. there would be little plausible benefit from attempting to steal or mis-attribute it.

- The information is already in XML form (or something very close to XML, net some misunderstandings of the finer points of the spec).

- It is already on the web, directly accessible via HTTP.

- The consumers of the API are experienced software developers, who presumably understand HTTP and XML natively and don't need a hand-holding "import a description of the API and generate all the code" tool to get the job done.

So, this is a job for which HTTP+XML is very well suited, and for which WS-* technologies add very little potential value. As a loyal Bloglines "customer" [just what is their business model anyway????] I'm happy to see them use the right technologies for this job. One might contrast this to Google, which drank deeply from the web services koolaid pitcher when developing its own Web API a couple of years ago and doesn't seem to have created much real value with them.

Let's consider, however, another scenario in which some business or agency needs to present an integrated view of diverse information sources, but has additional requirements:

- New sources of information must be discovered in real time with minimal human intervention.

- The information being aggregated is sensitive and confidential, so must be encrypted, digitally signed, and contain access control assertions.

- The systems from which information is gathered include mainframes, and some communicate with the outside using industrial strength middleware such as IBM's MQ technology or Software AG's EntireX rather than HTTP.

- Many of the communications links involve multiple hops, some of which go over intrinsically unreliable networks.

- The effective address of some information sources is discovered dynamically as part of a multipart interaction rather than via a published URI, and that address needs to be

- The information needs to be incorporated into a variety of off-the-shelf client programs by people with only basic programming skills.

OK all you folks who love to point out that HTTP+XML meet the needs of 99% of your customers, how do you meet the needs of the very real organizations that do have these kinds of requirements? Chances are you'll end up inventing something that is at least as complex as the WS-* stack, and you'll not get the benefit of the dozens of person-years of discussions about trials, errors, pitfalls, nitpicks, clarifications, and so forth that have gone into these specs. Nor will you get the benefits of the network effect from all the real products that support WS-* by rolling your own solution to these requirements.

I have no stake in the WS-* stuff -- it was developed outside the W3C system in which I spent a lot of time for the last several years, and I haven't been to any of the review workshops. Nor will I argue that WS-* will give market success to those who have bet on it, or that it hits the 80/20 point in the tradeoff between functionality and complexity. All I will argue is that a) it is designed to address real problems faced by real organizations who have tried to integrate enterprise systems with Web technology, and b) there is an immense amount of intelligence, thought, and real experience distilled into these things that needs to be given a lot of respect. If people in the REST camp want to actually change minds rather than exchange virtual pats on the back in their echo chamber, they'll have to explain how to do the hard things RESTfully, not belabor the pointlessness of doing easy things with WS-*.

It's a nice day, maybe I'll ride my bike to the public library at lunch .... but that doesn't mean you shouldn't use an armored car if you need to haul cash around Newark on a rainy night.

More WS-* specs, more questions about architectural viability

It's as regular as the seasons: as the leaves start to fall from the trees here in Michigan, more web services specifications flutter down from WS-IvoryTower, and more hunters take up their rhetorical shotguns to blast at them. This year it is WS-Transfer and WS-Enumeration , Microsoft has a new web services architecture vision document to accommodate them , and a number of people (Tim Bray is perhaps the most visible) have already wondered why we need Yet Another web services spec that appears to do something the Web already does.

I'm in somewhat of the same quandry that Tim is in, since there are an awful lot of smart people on each side of the fence here. (I recall a W3C Web Services Architecture working dinner when I was the only person at the rather long table without a Ph.D!). Likewise, I agree that the WS-* stack is monumentally difficult to keep track of, that there are some very visible success stories for "web services" that don't use WS-*, and that "design by committee" is not likely to be successful.

But one thing Tim says really helped me to understand why there are such diverging opinions among smart and well-informed people. " I’m deeply suspicious of multiple layers of abstraction that try to get between me and the messages full of angle-bracketed text that I push around to get work done ". Ahh, I think I see it now -- the "XML and the Web suffice" people push around XML to get their work done, and don't have the pain that WS-* tries to cure. The "WS-* is needed" folks, on the other hand, are working on the behalf of people to whom this is more or less unthinkable. "Real" people building enterprise applications of the sort that the WS-* specs target work with objects, databases, transaction monitors, and reliable message queueing systems .. which they need to make more accessible via HTTP *gateways." They don't have the option of simply exposing these systems as stateless Web resources with whom one exchanges XML respresentations, without a massive amount of code rewriting or adapter building, and there is nobody writing books or selling tools to assist them. There are, however, reams of whitepapers, articles, and books explaining how to apply the WS-* approach to the problem, and a considerable amount of success to show for all this.

On the other side, I don't think that very many of the people pushing simple XML over HTTP really have to solve nasty enterprise integration problems to earn a living. (Sean McGrath is a notable exception, by the way, but I recall (but can't find the link) that he on record as noting that the problems that WS-* addresses are real even if solutions that have been built by committee are not likely to actually work). So, I'm hypothesizing that the WS-* bashers are those who find XML and Web technologies the sharpest tools in their toolbox, and think of the problems of linking them with enterprise transaction systems to be an implementation detail; the proponents are those who have to expose enterprise systems over the web, and think of HTTP as just another transport and XML just another serialization. It's not a matter of one side being right and the other wrong, it's a matter of which tools are needed for which jobs. Tim Bray's point about Amazon, eBay, etc. not needing the WS-* stuff to get their job done is well taken, but it's also quite clear that these were built from the ground up to work with the Web, whereas the fertile ground for WS-* are the enterprise systems that were not designed with the web in mind.

What I'd really like to see here is the application of a well-known conflict management technique in which each side has to state the other side's position, to the other's satisfaction, before discussing the disagreement. For example, the new Microsoft web services architecture document praises and aligns itself with the Web in its introduction:"Web services take many of the ideas and principles of the Web and apply them to computer/computer interactions. Like the World Wide Web, Web services communicate using a set of foundation protocols that share a common architecture and are meant to be realized in a variety of independently developed and deployed systems. Like the World Wide Web, Web services protocols owe much to the text-based heritage of the Internet and are designed to layer as cleanly as possible without undue dependencies within the protocol stack."

So, why don't the Web standards suffice for computer/computer interactions? People have been talking past each other on this topic for years now. How about it? Maybe Mark Baker could re-state his understanding of why Don Box thinks they don't ... and vice versa ... before wrapping this permathread around the blogosphere one more time.

At a minimum, I'd like to see documents such as the Microsoft one address the well-known critique that the Web specs already hit the 80/20 point, and be much more specific on why they think these can't be simply extended to handle computer/computer interactions. I'd also like to see the RESTifarians explain and give examples of how they think goals such as "autonomous services" and "managed transparency" can be achieved without building something like the WS-* framework.

WS-Simplicity?

Tim Bray has yet another must read piece that apparently emerges from the collision of his deep understanding of XML concepts with realities he experiences at Sun. According to Tim, today's web service best practice is built on the foundations of asynchronous XML messages, produced with the assistance of programmer-friendly tools that hide the arcana of SOAP and XML (but not their basic principles). The grand theory expresses in the WS-* stack of proto-standards consists of interesting but unproven concepts including composable message architectures, dynamic service discovery, and declarative application building powered by metadata rather than procedural code. That theory is not well-grouned, however: 'This essay is about theory and practice, and we ought to have learned by now that standards are terrific when applied to proven industry practice but high-risk in the domain of theory and science. ..Here at Sun I’ve talked to a lot of people internally and worried out loud about the WS-* stack: whether it will actually work, whether it will interoperate, whether the complexity will be tractable, whether it will be secure, whether it’s going to have patent lock-ins. Mostly, they say “yep, yep, yep, we worry about those too.”'

Of course, the mainstream XML community lives in a glass house and should be careful about throwing stones at the web services people. Tim mentions the XML Schema spec as a victim of Design by Committee, and XQuery is often mentioned as another example; of the thousands of pages of XML-specs coming out of the W3C in the last few years, how many are really based on best practice? The Semantic Web specs are an interesting case; they contain the best practice that has survived over several generations of theory (LISP parantheses mutating to XML angle brackets along the way!), but I have no idea how truly field-tested they are. I'm actually becoming somewhat optimistic that we're seeing the inverse of what Bray talks about there -- people are discovering that they have been stumbling around the concepts that are captured and formalized in OWL, and are taking a look at how to apply them to domains that nobody thought of as needing an "ontology". We may be seeing one of those rare cases where academic theory successfully does precede practice.

The metaphor I like here is "intellectual capital." The internet and the Web looked like overnight successes (and committees fairly quickly established the core Web standards) during the 1990s, but they drew on at least 20 years of academic and government R&D. The industry seems to have learned the wrong lesson, believing that it was the committees writing the specs that created the intellectual capital upon which the Web was built, and hoped that more committees would keep the momentum going during this decade. I suspect that the capital formation occurred in a lot of research labs and email lists and conferences over a decade or two, and the standards committees just refined and refactored out what was relatively common.

The community of web services developers should certainly keep doing what they're doing to lay down the next layer of intellectual capital, and not be deterred by the skeptics and fundamentalists who think that XML+HTTP will always suffice. Consumers of this stuff, however, should be extremely wary, realizing that WS-* is about the future, not for today. People with real projects to plan simply have to judge for themselves which of these approaches and technologies are Best Practice (ideally with a solid theory behind it), and which are Best Guesses of self-appointed experts. My take on today's reality is very much in line with Tim's -- "XML, HTTP, URIs, SOAP, WSDL, and that’s about it."

XML 2003 reflections - day 1

This is the first of several reflections on what I think I learned here at the XML 2003 conference in Philadelphia. Sorry if it's too XML-geeky and not of sufficient interest to Java people, but I think a lot of what I heard people talking about have considerable relevance beyond the XML community they were aimed at.

Jon Udell gave a keynote speech on Tuesday that pierced the jaded, slightly cynical shell I've acquired after about 8 years in the XML world. He didn't talk about "maybe someday..." or "if only ...", he showed what a little imagination can do with the widely deployed XHTML, CSS, and XPath technologies today.

He began with an explanation of why the idea of an XML document that can be shared by different application is "magical," using two speeches by Bill Gates to illustrate the point. 10 years or so ago, Gates' slogan was "information at your fingertips," in which information workers could bring together text and data from a variety of applications and easily find, reuse, and revise it. The vision was a powerful one, but we are only beginning to realize it today. Part of the reason for the delay, Udell argued, was that in the OLE architecture that Microsoft promoted (and in the OpenDoc architecture of its rivals), each application owned its own proprietary data, and OLE/OpenDoc provided a means by which they could talk to each other. This worked to some extent, but created tight couplings (I'm not sure if Udell used that term) among the applications, object models, and operating systems and programming languages that support them.

In a more recent version of Gates' stump speech, the key idea is a "universal canvas" enabled by XML. Rather than applications communicating to access each others' proprietary data, they share a common XML representation of the data -- some may be numbers manipulated with a spreadsheet, others text manipulated by a wordprocessor, still others structured data manipulated via a forms interface. This vision is already very close to reality in Microsoft's Office 2003 product (and Udell's keynote was followed by a presentation from Adobe that demonstrated their very similar vision in action).

What makes XML a good "universal canvas"? In Udell's words, "contextual metadata is XML's gift to mankind." To demonstrate this, he showed how he had been eating his own dogfood in the presentation slides, which were being displayed in Mozilla on a Mac, produced with ordinary XHTML, with CSS styling, and some script. He showed some queries that exploit Mozilla's XPath support on data that wasn't really designed to be queried, but exploits the metadata that XHTML authors create without thinking about it -- links, attribute values put in to support CSS styles, etc. When the idea of XML-defined context is more explicitly supported, as in Kimbro Staken's Syncato weblog tool (or is it an "XML fragment management system?"), the power of XML's ability to model context and XPath's ability to query it becomes even more apparent. In any event, knowing that one will use XPath to retrieve information in the future motivates one to think about the structure of content as one writes -- creating the contextual metadata magic without much extra effort.

Udell went on to talk about how these advantages can be extended to email and instant messages, in which most real business communication takes place (and thus where the real content is created). We need to build tools and products that tap the small packets of structure and context and pull it into useful business information. Although Microsoft Office's XML support does not extend into Outlook, there are ways to archive email and IM in XML, and Udell challenged the audience to help devise real products and integrations to exploit the potential this would create. In any event, we don't need new standards, or a solution to the thorny issue of the diversity among flavors of RSS/Atom, to make progress -- XHTML, CSS, and XPath provide the basics of what we need. Udell argued that we need to figure out how to use what we have in a smarter way, and to "smuggle" metadata into content because it leverages the universal urge to make things look cool.

So why did this pierce my cynical shell? Most would agree that we need more metadata on the Web for it to live up to its full potential -- that's the very premise of the Semantic Web effort in which Tim Berners-Lee has invested much of the W3C's resources (and credibility). On the other hand, the historical difficulty of getting real people to put metadata in their content is believed by many to doom such efforts to failure. (Cory Doctrow's essay is the most colorful and cogent, if widely reviled, statement of this position). Udell's insight is that we can leverage the technology we have, salted by human vanity, to get usable metadata without technological breakthroughs or unrealistic demands on humans.

Simplistic subsets

Ray Ozzie [IMHO but IANAL] effectively demonstrates that 1993-vintage Lotus Notes had "prior art" that -- in a rational world -- would invalidate the Eolas patent on embedded hypermedia. This patent has resulted in a large judgment against Microsoft and raised the very real possibility that the Web browser as we know it must change drastically or infringe on the patent. Jon Udell suggests that this may be the most influential weblog entry of all time. We shall see about that, but it does have one of the most intriguing tidbits I've come across in awhile:

In 1993 or thereabouts, we saw the emergence of TCP/IP, HTML, HTTP, Mosaic and the Web. From our perspective, all of these were simplistic emulations of a tiny subset of what we'd been doing in Notes for years. TCP/IP instead of Netbeui or IPX/SPX, HTML instead of CD [Lotus Notes "compound document" ] records, HTTP instead of the Notes client/server protocols, httpd instead of a Notes server. And we were many years ahead in other ways: embedded compound objects, security, composition of documents as opposed to just "browsing" them, and a sophisticated development environment. I am quite embarassed to say that we frankly didn?t "get" what was so innovative about this newfangled "Web" thing, given the capabilities of what had already been built.

So why did the "simplistic subset" succeed so much more dramatically than Notes? Uhh, probably mainly because it is a simplistic subset! Consider the argument of Clay Shirky

The very weaknesses that make the Web so infuriating to serious practitioners also make it possible in the first place. In fact, had the Web been a strong and well-designed entity from its inception, it would have gone nowhere... Only solutions that produce partial results when partially implemented can succeed. The network is littered with ideas that would have worked had everybody adopted them. Evolvable systems begin partially working right away and then grow, rather than needing to be perfected and frozen.

Put differently, the Web succeeded because it hit the famous "80/20 point" -- it gave most of what other solutions offered, but at a much lower cost and complexity.

Joel Spolsky has a rather different take on this.

A lot of software developers are seduced by the old "80/20" rule. It seems to make a lot of sense: 80% of the people use 20% of the features. So you convince yourself that you only need to implement 20% of the features, and you can still sell 80% as many copies. Unfortunately, it's never the same 20%. Everybody uses a different set of features.

So, the 80/20 rule rules sometimes, but not always. Obviously, it doesn't rule when different groups need a different subset of the features. More generally, it doesn't appear to rule at the application level -- the masses seem willing to keep pour money into Microsoft's jaws to get the latest version of Office, when far fewer than 20% use features that weren't in Office 97. OpenOffice, which appears to be at about the Office 97 level of functionality and is free, has a miniscule market share. (I'm part of it, so don't flame me!). A few hypotheses:

• Minimalism ( fundamental simplicity) matters more at the infrastructure level, where it really really matters how universally something is adopted, how reliable it is, and how well it performs.
• Minimalism ( fundamental simplicity) matters more when standards are involved.
• Ease of use (simplicity of operation) is obviously critical when you are dealing with a large, relatively unsophisticated audience. The early Web didn't have to worry about this, and by the time it took off it had a "cool factor" that overcame a lot of resistance. One can't count on that happening again anytime soon!
• "Good" software, at both the infrastructure and UI levels, manages to maintain the "illusion of simplicity" by hiding the ugly details while still providing most of the functionality that one immersed in the details can employ. That's hard. Office 2003 is just starting to approach this.
• Beware of the argument that complexity isn't a problem so long as it is hidden. True enough, in the abstract, but only if it is the detail being hidden and not the complex concepts that an API or GUI is simply ignoring. For example, XSLT is a powerful tool in expert hands, but AFAIK none of the GUI stylesheet generators go much beyond the basics. The same is often said of IDEs.

Reports of the "Demise of the XML Database" are dubious

Phil Howard of Bloor Research presents anargument I've heard more than once recently: "The reason why there is this trend away from pure XML storage is because advanced XML capabilities are being introduced by all the leading relational vendors." As the developers of Object Oriented DBMS discovered, he says, "the truth is that (like it or not) the market will make do with what the relational database products offer".

If analysts are already inviting themselves to the funeral of the XML DBMS, maybe we should check to see if the departed is lying quietly waiting to be buried. Perhaps like the unfortunate Lal Bihari of Uttar Pradesh in India, it been declared dead by those who would prematurely appropriate its inheritance. Here's a few arguments for their continued viability:

• It is widely believed that less than a quarter of enterprise data is currently stored in RDBMS systems. This suggests that the market is not "making do" with what the relational database products offer today, but using a wide variety of technologies.
• The main reason OODBMS didn't hit the sweet spot, AFAIK, is that they created a tight coupling between application code and the DBMS. Potential performance gains this allows can outweigh the maintenance challenges in extremely business critical, high transaction volume environments (e.g. where IBM's IMS or Software AG's Adabas DBMS still thrive), but not in the kinds of innovative, rapidly evolving environments where OODBMS got a foothold.
• XML DBMS, on the other hand, inherit XML's suitability for loosely coupling systems, applications, and tools across a wide range of environments. Phil Howard mentions "integration" as if it were a niche. That may be true in a business sense, but the ability of the XML DBMS to efficiently, scalably, and reliably power the intrinsic integration-friendliness of XML is likely to find a wide range of uses outside the "application integration" business space.
• Again AFAIK (having only played with OODBMS personally), there is relatively little portability across OODBMS systems; code written for one would be very expensive to adapt to another. Investing in the technology required one to make a risky bet on the vendor who supplied it. This created an environment where the object-relational vendors could prosper by offering only a subset of the features but the absolute assurance that they would be in business for years to come. In the XML DBMS world, on the other hand, all support roughly the same schema, query language, and API standards;
• The standards of the XML world provide a clearly defined and fairly high bar for those who would seek to take away the market pioneered by the XML DBMS vendors. For better or worse, the XML family of specs is complex and quite challenging to support efficiently in a DBMS system. It's one thing to support, as the RDBMS vendors now do quite well, XML views of structured, typed, relatively "flat" data such as are typically found in RDBMS applications. It is quite another to efficiently and scalably support queries and updates on "document-like" XML with relatively open content models, lots of recursion, mixed content, and where wildcard text comparisions are more frequent than typed value comparisons. The dominant DBMS vendors obviously have talent and money to throw at the problem, but analysts should not assume that they will surpass theese capabilities of the XML DBMS systems anytime soon.

This is not to predict the demise or even decline of the one-size-fits-all RDBMS+kitchen sink products from the dominant vendors. Microsoft's Longhorn project apparently hopes to handle much of the other 75% in an RDBMS-powered but XML-capable replacement for the filesystem. As much as one can admire this vision and expect it to be at least partially successful, there will be an awful lot of "pure" XML data out there, partly due to Microsoft's other efforts. A substantial number of developers and end-users will continue to need "pure XML" DBMS technology that handles XML extremely well, even given the availability of DBMS technology that handles all kinds of data in a "good enough" way.

Finally, it's important to consider the gap between technology mindshare among pundits and analysts, and technology deployment by businesses who need to get the job done efficiently and reliably. I'm reminded of another technology analyst who has dismissed XML itself by pointing to its similarities to IBM's IMS product, a "discarded in practice" hierarchical DBMS. So IMS is also dead, eh? Let's see ...IBM says "IMS serves 200 million end users, managing over 15 billion Gigabytes of production data and processing over 50 billion transactions every day." Another corpse that won't quietly play its part in the funeral the analysts have thrown for it!

I'm not sure when pure XML databases will be managing exabytes of data, but It looks to me like XML DBMS's will join IMS and Mr. Bihari in the ranks of those who continue to lead active and productive lives despite having been declared "legally dead" by the presumed authorities.

Standards, Stability, and Confusion

There are lots of articles and weblogs about standards-related issues recently. Simple news/weblog content syndication "standards" are in the midst of a power struggle, ... Web services management vendors may be squaring up for a standards fight ... Reliable SOAP messaging politics are bogged down or stupid, and so on. A few thoughts ..

Lots of people get annoyed that it takes a group so long to do what any competent individual could do more quickly and cleanly. The point is, uh, standardization. Sure you, or your favorite tool's vendor, can define terms, formats, APIs, etc. quicker and in some ways better than a committee can. There's a bazillion jokes on the basic theme "there would be so much less discord if the rest of you would just do it MY way! But then you usually end up with dozens of ways of doing the same thing, and that invites gravitation toward the biggest vendor in a space..

Standardization is hard -- it requires people with different perspectives and interests to (usually very laboriously) figure out what they really have in common and what that implies for the standard terminology (or data format, or API, or whatever). Even I (after 6 years of spending about half my working time on various W3C groups) am continually amazed at how hard it is to get agreement on seemingly trivial issues. Uhh, that's usually because they only look trivial ...

Standardization is messy. There plenty of jokes about standards and sausage making, along the lines of "those who appreciate sausages or the law should not watch either being made". Personality / politics is a big part of the process, don't let anyone tell you otherwise. But that's life: Lots of profound accomplishments were achieved by highly flawed personalities, groups, and processes. Just picking the first examples that come to mind, Isaac Newton was a seriously disturbed man and quibbled constantly about his "intellectual property," but he built the foundation for modern science and math; the politics that produced the US Constitution were pretty intense and gave the document itself some embarassing warts, but the overall result was extremely successful; and the interpersonal processes in the group that discovered the structure of DNA were pretty ugly, but again the result was profound. (Oops, I see that Tim Bray made this point better than I could while I've been fiddling with this article).

The result of a standards process is rarely elegant. Here there is one canonical joke: "A camel is a horse designed by a committee." The analogy between racehorses and camels is actually a good one, but I'd turn it around: If you want to go fast around a well-prepared track, get yourself a racehorse. If you want to reliably reach a distant destination across unknown terrain, you are probably better off on a camel. Those committees can be sortof surrogates for evolution, pruning out the beautiful but high-maintenance features and insisting on the "humps" that look strange but add real value.

As much as I sometimes wish that we could somehow avoid all this, it's increasingly clear that well-defined specifications (not universally adopted legal standards!) are increasingly necessary. Some argue (see the RSS article linked above) that a formal standards process primarily benefits the big vendors who can invest in it and promote their own technologies. That's true enough, but the alternative is even worse: without rigorously defined criteria for what a "valid" document, message, or API is, the dominant vendor's implementation becomes the de facto standard. Furthermore, you can hardly benefit from test driven development if the tests themselves are best guesses based on what works with the dominant vendor's implementation; you'll never know if the test is right and your code is wrong, or whether you're simply introducing "bug for bug compatibility" that will hurt you if the original bug is fixed.

So what is to be done? I don't presume to think that much can be done to change the standards processes, but I do think that consumers of "standards" can adjust their attitudes in ways that work for them in the short run and the overall Web standardization culture in the long run.

1. Ask yourself "can I reject documents that don't conform to the standard" before worrying too much about the details. The numerous variants and decendents of "RSS" are a case in point. I'm not going to ignore a source of news or opinion if they don't conform to one of the many alleged standards in this space, I'm going to use the most "libaral" tools that manage to extract useful information. But in other cases -- especially those involving money or security, or when transaction volumes are very high -- validity of a message against an agreed-upon standard is a prerequisite for accepting it, and the details matter very much.
2. Judge the specs on their results, not their inputs or processes or the legal standing. The SAX XML API is a case in point -- it has no official standing whatsoever, but is so simple and useful that it is almost universally supported. That makes it more of a "standard" than something like ISO DSSSL or W3C XLink that simply haven't generated much of a network effect. If, somehow, the messy politics of the content syndication or web services sectors result in specs that are clean, useful, and widely supported, then the process has succeeded despite itself.
3. Vote with your feet for or against specs based on whether they solve real portability and interoperablity problems. Vendors don't really have the luxury of ignoring official standards, but users do. (Actually, recent specs are more likely to have "interoperability profiles" or "minimum conformance levels" that reflect the business realities in the field rather than the political realities in the conference room, but that's another essay!).
4. Accept that diversity, competition, and evolution is inevitable. No matter how convenient it would be for there to be one universal, stable standard in each domain, it ain't gonna happen, fuggidaboudit. In the early days of a technology, premature standardization is a worse problem than confusion. When a technology is mature, the way forward often involves taking a few steps backward. ( The Innovator's Dilemma has numerous examples that illustrate this). Knowing when it's time to non-standardize or un-standardize to allow innovation to flourish is as important as knowing how to create consensus and standardization when it's time for the network effect to work its magic.

At least one group of analysts more or less agrees, making the additional point that "the whole point to Web Services is to be interoperable. 'Fragmented Web Services standards' is an oxymoron that makes no business sense. And if there's no money to be made, then companies won't do it. "

XML can define agreements, but can also help deal with chaos

The contentious world of RSS and the "(not) Echo" project have been featured in a number of java.net weblogs recently by Simon Phipps and Daniel Steinberg. I've been intrigued by RSS for awhile because it illustrates both the challenges one faces in the real world in getting agreement on what seems like a simple problem, but also on the ability of XML to provide robust solutions even in the absence of agreement.

One of the biggest issues in the (N)Echo debate is whether a new, presumably improved format is worth the disruption it will cause to established users and software developers. Some are saying "If it ain't broke, don't fix it." -- Tinkering is more likely to break the existing applications such as RSS aggregators than to provide a solid foundation for further development. Stability in formats and protocols is, in this view, what the weblogging world needs to continue to expand and prosper.

This argument would be quite compelling in a world without XML, but is somewhat moot now that XML is pervasive: The whole point of XML's "self-describing" tags [1] is to allow loose coupling between producers and consumers of data. In one widely held view, this means "all that the producers and consumers of information have to agree on is the XML format, and software that supports it can evolve freely." I'd contend, however, that if a community could agree on the data format there might be little need for XML -- a CSV or ASN.1 or Java serialized object format without tags would work more efficiently, be easier to integrate with procedural code, require less network bandwidth, etc. The problem with agreed-upon formats -- besides the difficulty of achieving agreement, of course!-- is their fragility in the face of inevitable change.

The power of XML's tags (namespaced or otherwise) is to allow variation and evolution. The history of RSS bears witness to this very clearly. Even in a world of chronic disagreement, rapid innovation, and several contending "standards," the actual software that syndicates weblogs and aggregates diverse feeds has been remarkably robust. In fact, software to produce and consume (N)Echo appeared almost immediately after . This rapid response wasn't due, AFAIK, to late night hacking but to simple tweaks to the scripts, stylesheets, etc. that had evolved to support the diverse flavors of RSS previously seen.

I don't want to understate the overall business importance of having authoritative "standards" (de jure, de facto, ad hoc, or whatever) in this area. Tim Bray has made a compelling case for this and that seems to have been one of the galvanizing factors in the formation of the (N)Echo community. But whether or not Bray's prototypical Mr. Safe can cope with controversy and diversity, XML's basic technology is definitely up to the job. An eventual "standard" for an RSS-like format will help corporate developers using drag-n-drop IDEs to more easily develop software to produce and process syndication streams, but stasis is not necessary for progress in the XML world. In fact, the evolutionary potential of XML's ability to support loosely coupled applications is its greatest strength.

[1] Sigh, am not under the illusion that XML instances are "self-describing" in any philosophical sense. XML is "self-describing" only by comparison to alternative formats such as CSV or ASN.1 that require a more rigid data format and does not "tag" individual data items. The XML markup may refer to "namespaces" in which the semantics of specific element names are rigorously defined using an ontology language, or they may simply be hints exploited by a heuristic algorithm, but they supply additional information that a processor can exploit.

When does SOAP add value over simple HTTP+XML?

Sean McGrath seems to be the first to link to this weblog, and I'll return the favor by publicly disagreeing with him <grin>. Actually, there's not a whole lot in the XML world that Sean an I disagree about, I strongly recommend his ITWorld columns . But apparently the benefits of SOAP are one of them, as he says: " I would argue that it is not the case that for more complex apps, SOAP is better. Sure, there comes a point where you cannot encode parameters into a URI but I don't see why it follows that these more complex web services need to throw out the huge advantages of having a GETable URI."

First, as of SOAP 1.2 (very soon to be a W3C Recommendation) there is no conflict between SOAP and "having a GETable URI." SOAP has a web method feature:

Bindings to HTTP or such other protocols SHOULD use the SOAP Web Method feature to give applications control over the Web methods to be used when sending a SOAP message.

In short, SOAP 1.2 allows and encourages the use of HTTP GET to invoke services that simply return data, thus allowing one to hyperlink to SOAP services, exploit HTTP caches of the results of frequently-accessed services, etc.

But, one might ask (and people very frequently do!), what benefit does one get from SOAP if just GETing or POSTing XML data works just fine in your application? One very reasonable answer is "not much." Sure, once WSDL 1.2 (which wll support the web method feature in SOAP) is out we can expect tools to make it very easy to generate programming language code to invoke "GETable URI" SOAP services, but that is in the future sometime. A better answer, I think is that SOAP provides an architectural foundation for extended services as requirements evolve.

For example, "raw XML over HTTP" works just fine for simple services over a secure intranet (or the public internet when SSL provides adequate security). But what happens as the requirements on the service creep upwards and one must: support:

• Routiing and reliable messaging across multi-node networks, such as when one must perform content-based routing from an HTTP gateway to the appropriate back-end service (e.g., the one nearest to the consumer).
• End to end encryption (from consuming application to service rather from consuming application to SSL gateway)
• integration of legacy services that may not have an HTTP URI
• Non-HTTP communications protocols and interfaces such as BEEP, MQ and JMS
• Multipart service transactions that must be committed or rolled-back/compensated as an entity
• More complex interactions between service suppliers and consumers that need to be described and choreographed.

So, if you are likely to have some of these challenges, building SOAP into the architecture lets you leverage emerging technologies such as WS-Security, WS-Routing, one of the several reliable messaging proposals, WS-Transaction, WSBPEL, etc. that build on the SOAP infrastructure.

If, on the other hand, you are saying to yourself "that's WAY more complicated than anything I want to do," then SOAP may not be for you. Likewise, if you are saying "I need to do that stuff MY way rather than the way some industry committee's way," then perhaps you are better off putting this stuff into your own application rather then buying an off-the-shelf solution, and SOAP doesn't offer anything you need. But this is something to be analyzed -- it's just as "wrong" to blindly reject SOAP as to blindly accept. it.

Still, at a more fundamental level I think Sean and I agree on the basic architectural principle here :"I have learned to really appreciate the power of the "distributed systems as stateless document exchange," which I believe lies at the heart of what makes HTTP so stunningly successful. " It often makes more sense to think of XML-powered applications as transformation pipelines (see Propylon's PropelX and Software AG's EntireX Mediator as pioneering tools that support this approach) rather than objects to be exposed with APIs. More on this in a future edition ...but in either scenario, SOAP may add enough value to justify its use.

Exploring Where XML and Java Meet

For some users, the Java and XML roads come together in a smooth interchange where objects can be serialized as XML and schemas cleanly bound into classes. But for others, they come together in a maze of alternative sidestreets, and there are a lot of interesting things happening there that one one misses by simply taking the "databinding interchange." The overall theme of this weblog is to explore some of the sidestreets in the "place" where XML, Java, and related technologies such as DBMS and Web services meet, and sometimes look into the dark alleys leading off them. I plan to use this weblog as a forum in which to point to interesting ideas and generate discussions about them, so please contribute thoughts, corrections, commentary, and suggestions.

To start things off, there has been a spate of articles lately related to something I've wondered about for a long time, namely the relationship of the world of objects to the world of XML. On one hand, the OO principle of "data hiding" suggests treating the details of data formats as implementation details to be hidden behind access methods -- relates to the world of XML. The whole point of XML is to expose the data rather than to hide it. Norm Walsh emphasizes the point that "XML is Not Object Oriented," pointing out that:

the constituent elements and attributes of an XML vocabulary are not generally related to each other by inheritance, nor do they naturally correspond to objects with any kind of precision.

On the other hand, XML makes an awfully handy serialization format for Java objects.Also, there is a generic XML object model that has much support in the Java world and provides a useful, if very low-level, set of methods for working with XML documents.

So, as the old joke goes, XML really is a bit of a "floor wax and a dessert topping." The trick is to know which approaches, tools, APIs, etc. are best for which applications. That is a subject we will explore in subsequent editions of this weblog.

This work is licensed under a Creative Commons License.