John Reynolds's Blog: March 2006 Archive

Posted by johnreynolds on March 17, 2006 at 01:49 PM | Permalink | Comments (7)

Recently I had the pleasure of sitting down with Ken Orr (of Warnier-Orr Diagram fame) to discuss his views on software design and on what programmers really need to know.

In the 1970's, Ken Orr expanded on Jean-Dominique Warnier's methodology to develop Data Structured System Design (DSSD).

DSSD is primarilly a data driven methodology for designing systems with (gasp!) direct user involvement. Through the use logic and data charts that are easily explained to the users, the requirements of a project are collaboratively captured in a format from which much of the code and data schemas can be generated... This may sound like "no big deal" today, but Ken published these techniques in his book Structured Systems Development in 1977.

Ever the opportunist, I cornered Ken during lunch one day to pump him for insights (as fodder for this blog)...

John: "Ken... What advice would you give Java programmers?"
Ken Orr: "Don't define yourself as Java programmers."

I am going to have to paraphrase from here on out... both because Ken is from Kansas, and because I wasn't taking notes (and I didn't think to bring a recorder).

Being a good programmer really has very little to do with the languages that you know... Being a good programmer is mostly dependent on the methodologies that you know and on how you employ those methodologies.

Good process design skills and good data modelling skills are the foundations for being a good programmer.

This holds true whether you employ Object Oriented Design, Aspect Oriented Design, or Service Oriented Design. If the process is not well engineered, or the data model is not well engineered, "you are just putting lipstick on a pig".

Process design and data modelling are best done in collaboration with your users (Don't confuse this with letting the users design the process or model the data, both require considerable skills that aren't easily mastered). Ken offered this advice when developing a new design with a client:

It is better to be clearly wrong than to be obscurely right.

If you have something wrong in your design, and it is clearly wrong, then your client will point it out and you can correct it. If the design is obscure, the client may never know if it is right or if it is wrong.

Tools are the key for communicating with your clients... processes (and data) are expressed well through pictures (flow charts, etc.), and you are missing the boat if you don't adopt tools that map these pictures to code.

I got the feeling that Ken's biggest regret is that Computer Aided Software Engineering (CASE) seems to have been on hiatus for most of the 90's and the first years of this century. Model Driven Architecture is surely a form of CASE, but Ken feels that the focus isn't quite right.

UML was developed to meet the design needs of programmers. We need tools focussed on meeting the design needs of users.

This led us into a discussion of SOA and BPEL. Ken is very "hype-sensitive", and cautions that SOA must be understood before you take the plunge.

One of the most promising aspects of SOA is that it brings the art of process design back into focus. In many cases, each service can serve as a step in a process, and BPEL can be used to describe the process itself.

My spin on the process/services connection is the following: By viewing each service as a step within a larger process, you can get a "feel" for the correct granularity of the service interfaces...

If the granualarity of the services is too fine, then the process definition will probably have too many steps.
If the granularity of the services is too coarse, then you won't be able to modify the process without modifying individual services.

Ken's parting advice was perhaps the most important thing to remember...

Master more than one design/programming paradigm.

No paradigm is perfect... that's why there are so many of them. Each fills a niche, and in some cases you might need to apply multiple paradigms to solve a single problem.

Ken gave the example of AOP vs. OOP. Objects are very good at solving "vertical" problems... Hierarchies of relationships being the most common. Aspects are very good at solving "horizontal" problems... Cross-cutting concerns require thinking beyond object boundaries.

I suspect this is why Ken has never become too old to program. By mastering the basics of data modeling and process design, and by mastering newer paradigms as they are developed, Ken has remained a vibrant force in the programming world.

I don't want to make Ken blush, but in my book he's a darn good role model.

Software for Business People (like my dad)

Posted by johnreynolds on March 09, 2006 at 08:10 AM | Permalink | Comments (0)

My father ran a small-town employee credit union in the 1960s and early 70s when most small organizations still used paper to store their records. Information for each account at the credit union was kept on large sheets of card stock. When a deposit or a withdrawal or a loan payment was made, or when interest was accrued, my father would retrieve these paper records, insert them into a Burroughs “posting” machine, and type in the current balance and the new information. The “posting” machine would then mechanically compute the new account balances, and print the relevant information onto the paper record. Very primitive by today’s standards, but all of the relevant information about the account was there on a card for all to see (Don’t ask how my father to dealt with errors… the thought still makes me shudder).

My father’s credit union worked because he had a system, or more precisely a process, for updating accounts. When folks would deposit or withdraw their money, forms would be filled out and placed into what was essentially an inbox. The money would change hands immediately, but the “official” paper record would not be updated until a time that was set aside for “posting” the updates. During most of the year my father and one helper (my mom) were the only staff available; so given the complexity of “posting” changes, you could not do this immediately. Once each year (at tax time) my father had to update all of the accounts and prepare letters that were sent to all of the shareholders. During these “crunch periods” my dad would hire extra staff, some of whom (me and my brother) were very unskilled. The "process" during these times was modified… instead of one person cycling through all of the steps, the more skilled workers would focus on key steps of the process for each account, and their results would be placed into bins. The temporary workers would take their work from these bins, each worker going at their own speed (I was notoriously slow).

From a "40,000 foot view" of the business perspective, very little has changed in the world of credit unions. Computers have enabled process automation, and that has made it much easier for folks like my dad to process transactions. Computing interest accruals is no longer a scary proposition. Everything happens much faster, and fewer staff members are needed, but the basics of the business are still about the same.

Every business involves a collection of processes that result in services or products that are delivered to customers and business partners. As conditions change, the processes need to change or the business suffers (or fails). Computers and software have enabled the automation of these processes… but they generally have not significantly altered these processes (unless automation by itself enabled a better process).

Processes are composed of events, tasks and decision points. Execution of one or more tasks leads to a decision point, the outcome of which leads to the execution of more tasks (or the completion of the process). As with my father’s process, we’d like to be able to “hire more workers” to perform these tasks if our business grows (and we’d like to be able to “lay off workers” if the business slows). Said another way, we want processes that scale well… as the business grows we should be able to add hardware to keep pace without having to change our software.

We also want processes that can be easily changed. If a new task is required, or an existing task is no longer needed, or the criteria for making a flow decision changes, we want to be able to easily implement the changes.

Java and the JVM clearly play a huge role in crafting the infrastructure underlying "Business Process Environments". With the combination of Java-based application servers, messaging systems, user interface components, database support, rules engines, identity and security solutions, enterprise service busses, etc. the Java ecosystem is by far the best choice as the basis for solutions.

To capitalize on Java's strengths, we need to step back into the shoes of those pre-computer business folks (like my dad) and consider what we need to provide to make all of this "stuff" really work for them.

Back before computers, you could sit in an office and observe the business processes. The process flow and process steps could be easily monitored and analyzed. The need to modify the flow, or to improve the efficiency of a specific task was something the business owner could easily discern.

From this perspective, you can probably understand why I am interested in languages like BPEL, and in the SOA design paradigm. The tasks and decision points that make up processes can be mapped very closely onto Services and BPEL orchestration logic.

BPEL is far from perfect, and the use of Services (particularly those with XML/HTTP interfaces) adds overhead, but the paradigm is the right one for many businesses. I’m comfortable to predict that better “process oriented” languages will be written, the performance overhead associated with using Services will be lessened, and the ESBs will become a whole lot more capable.

The NetBeans 5.5 Orchestration Designer is very cool, but it is only the tip of the iceberg. We should soon see tools that let someone who really understands a business (like my dad) make (appropriate) process changes without having to involve a programmer (To use a car analogy: “real” programmers will build and maintain cars, but business people will be able to tune the engines.)

If my dad were still with us, he'd probably still be hesitant to give up his paper records... but in the near future I'll bet that we could win him over ;-)

Is an Enterprise Service Bus (ESB) appropriate for volume-performance critical consumer applications?

Posted by johnreynolds on March 02, 2006 at 04:24 PM | Permalink | Comments (8)

In response to my SOA/ESB Level Set blog entry, I got the following question from cparziale:

"In researching esb direction, I'm trying to assess if a "service bus" is a just another service 'on the side' of the primary services, to be used when needed, like UDDI, or if it will become a physical mediator of all service consumer-provider conversations. I ask because I'm trying to visualise what to do when we have high volume-performance critical consumer applications (such as in a financial institution) who have well defined providers and don't need externalized orchestration, transformation or even routing and do not want to to incur the overhead or support of additional mediation tiers. So will the ESB concept become mediation-integration fabric, like DNS, or will it become an application tier that must be managed and accounted for in a queuing/performance analysis? Any thoughts in this space are appreciated "

This is a great question, and it prompted me to write this blog entry to answer this and some related questions...

I believe the value proposition for an ESB drops considerably unless it is the mediator for all service consumer-provider conversations. The advantage of an ESB over point-to-point service invocations is mostly about managing services across the enterprise. If "rogue applications" bypass the service bus, you will lose much of its value.

At the heart of cparziale's question is the gnawing concern:

"We do not want to incur the overhead or support of additional mediation tiers."

This is obviously a legitimate concern, and one that every ESB architect must address.

What additional overhead (latency) is incurred by invoking a service via an ESB versus invoking it directly?

There are many factors to consider beyond this concern to determine whether or not an ESB is appropriate for your company, but we must still answer this question as best we can.

The answer will depend on the architecture of the specific ESB, but I believe that it is quite possible that the additional overhead could be quite low.

An ESB allows you to abstract the address of the service endpoint that you wish to invoke. To make this work, the ESB has to map the logical address of the service to a “physical” address. Presumably, a service registry (akin to UDDI) will be used to look up the “physical” address. Looking up the address each time a specific consumer accesses a service could be costly, but if the consumer’s ESB "agent" caches "physical" addresses the overhead could be quite low… it all depends on the implementation of the ESB itself (Check out the links to various ESB implementations at the end of this blog entry).

Tha value of an ESB comes into play when you consider the burden that would be placed on individual service consumers if the ESB was not there... so we have to look at the benefits that the ESB provides and decide whether or not we need them.

One benefit of an ESB is the aforementioned abstraction of the "physical" address. If the location of a service moves, or if a new version of a service becomes available, the consumers of the service will not have to be modified. You could of course build this logic into the service consumer itself, but imagine where that leads.

An ESB could also be a mechanism for providing scalability and load balancing. Assume that overall performance could be enhanced by instantiating multiple copies of a specific service: The ESB could function as a load balancer, hiding from service consumers any knowledge that multiple service instances exist. Of course you could accomplish the same thing by hosting a service on a clustered server, but the ESB offers the promise of creating service instances "on demand" if coupled with the appropriate provisioning software.

Some ESB implementation provide these capabilities, but others certainly do not, and it is very important for us to ask these questions before commiting ourselves. If "high volume-performance critical consumer applications" are a part of our enterprise, then we'd better be very sure that the ESB that we adopt is up to the task... Ask the right questions early, or we could end up very disappointed.

Powered by
Movable Type 3.01D

java.net RSS Feeds