John Reynolds's Blog

"We don't see the need for BPM"

Several folks said "We don't see the need for BPM tools"... and I would like to address that in this blog entry.

I'm really not surprised that the average Java developer doesn't see a need for BPM... In fact I'm not surprised that the average worker in any industry doesn't see the need for BPM.

All of us who work for a living participate in business processes. All of us who interact with businesses participate in business processes.

Almost of us participate in business processes. We don't think about it much because we generally don't care what the full process is... We generally only care about the tasks that we have to perform within any process.

For example: When you apply for a loan you are a participant in a "Loan Origination Process". In general you really don't care about all of the steps of the process. You don't care about all of the tasks that the lender has to perform. You only care about the process step where you fill out the loan application and the process step where you respond to the lender's decision.

When you develop software you are a participant in a "Software Development Process". Speaking for myself, when I am working in "programmer mode" I usually don't care about all of the steps of the process. I only care about understanding the requirements, writing the code, and fixing any bugs that QA finds.

My Project Manger (PM) cares about many more steps of the software development process than I do. Project Managers have to coordinate the programmer tasks and those of everyone else on the project. Most of the PMs that I've met don't think of their projects as instances of the software development process... They tend to view their projects as schedules, which explains why those lovely Gant charts in Microsoft Project are pretty much all that they need in terms of tool support.

It's only when you get to a much higher level in your organization, perhaps to the VP of Development, that you find someone who might really worry about the whole Software Development Process. If your organization is large enough, the VP might be responsible for several projects that are running at the same time, with each project at a different point in the process.

As long as all of your company's projects are completed on time and under budget, your VP is probably going to leave your Software Development Process (SDP) alone. For some strange reason, all of the VPs that I have worked for kept tweaking their SDPs.

Development VPs mess with their SDPs to try to lower costs, to have more predictable schedules, and to just generally drive their employees crazy. They want to make more money. Meddlesome Development VPs like to find problems in the process, "improve" the process, and then do it all over again.

Your company has lots and lots of business processes besides your Software Development Process. Some are really mundane, some our critical to your company's bottom line.

All of your Corporate Officers are really interested in managing all of those business processes. Inefficient processes cost more, both in terms of expense and in terms of lost revenue, than efficient processes.

Managing a process...

The first step in managing a process is to know what the process actually is. This includes the steps that are done by people and the steps that are done by machines. I once found it necessary to gather a dozen people in a room to find out what everyone thought the process was, and then later discovered the process wasn't at all what they thought it was. Big companies like to spend lots of money to diagram their processes... which they then file away for five years until they do it again.

The second step in managing a process is to gather metrics... How long (on average) does it take for each step of the process to take? How often are steps repeated (presumably due to errors)?

The third step and fourth steps in managing a process are to analyze and actively manage the process. Use the metrics that you've gathered to analyze process performance, develop a better process, try it out and start all over again. That's Continuous Process Improvement.

All good BPM suites provide infrastructure support for all the steps needed to manage processes...

First: Know the process

At the heart of all BPM suites is a process engine that executes process definitions. These definitions are usually represented graphically using BPMN notation. Process diagrams are usually generated in conjunction with the business people who own the process... Please note that it's much more important that the business folks understand how to "read" the process diagram than it is for them to be able to draw the diagram... The process engine is going to execute the diagram, so if it's wrong the process will be wrong.

I like to say that processes are composed of Human Powered Services and Autonomous Services... Most BPM suites provide tools that help you build the Human Powered Services. These tools are basically form designers, and they're good for doing many business-task-oriented user interfaces, but they aren't meant to be used for designing "filthy rich" client interfaces... All of the good BPM suites provide interfaces that allow you to hook in virtually any client that you wish to construct. If you want a Swing application to perform a task, then you can write a Swing application and hook it in. If you want to use Netbeans or Eclipse to perform a task, you can do that too.

BPM is not about replacing all of the applications that you use. It's about coordinating the use of those applications to perform a process.

But back to knowing what the process is.... You know what the process is because you can always see it. The process definition is not scattered across multiple applications and background tasks. It's contained in one precise definition.

Second: Gather metrics

Since the process engine is managing the process, the process engine knows when each task is started and ended. The process engine knows who performs each task. This makes it incredibly easy to instrument a process.

Third: Analyze the process

All good BPM suites provide tools for analyzing all of those nifty metrics that you've gathered. The really good suites even let you use your historical data to simulate how a proposed process change would have performed in the past.

Fourth: Modify the process

With that analysis in hand, you can now go back and modify your process... This is really easy because, as you may remember, the process definition is in a single place.

I cannot emphasize how useful it is to have that process definition driving everything else. In essence, you have a process-centric view of your code base. In my pre-BPM days I would join a project and have to spend weeks to track down the code that was responsible for some specific process behavior. With a BPM tool, you can always start at the process diagram and "drill down" to the code you need to change.

But can't I just put all these features together myself?

Of course you could. What do you think all the BPM vendors did?

Wrong question... Will a client pay you to put all of these features together AND pay you to develop a process on top of that?

The optimist in me says: "Why of course they will!"

The realist in me says: "Why would they?"

Sounds boring... Why shouldn't I just ignore BPM?

There's no reason to pay any attention to BPM as long as you don't make your living building process-centric applications.

If the applications that you write only involve one participant, you'll never need BPM... but I encourage you take a good hard look at those applications because they may actually function better if they are process-aware.

For example... Every issue-tracking system implements multi-participant processes. Sadly the processes are usually hard-coded in these tools, but more and more you'll find that issue tracking tools (like Jira ) are providing web-service interfaces to interact better with the other tools that are used in a wider process.

Another example... Your favorite IDE. More and more we are seeing process oriented functionality creeping into these tools. Why not leverage an existing process engine (like Apache ODE ) and make those processes more flexible?

Parting thought...

From the very beginning, the core of most programming has been Process Definitions and Data Structures. BPM is our latest reminder of that.

Comments

• I can understand your arguments from the "process" view. Just like you pointed out, programmers and business process folks are two different groups of people. Although they can work closely with each other. They have different responsibilities. An average programmer probably is not that enthusiastic about process, and an average process folk is not interested in programming. BPM vendors try to find a sweets pot that will please both campus. But it ends up displease all of us. Except for the product demo, I have not yet seen any teams use their BPM software to communicate the business processes to the clients. A 30 minutes PowerPoint probably does a better job on that. On the other hand, drag-drop programming means high productivity? I will appreciate vendors to create better tools that are more specific targeted to their intended users. Other tools such as data miner, transformer can be used to fill the gap between programmers and process folks. I would like to see how those vendors use the BPM tools within their own enterprise. You get to eat your own dog food.

  Posted by: sunh11373 on October 11, 2007 at 08:06 PM

• I've heard about how such tools would make programmers obsolete by allowing managers to just input their business requirements into the tool and have a complete application up and running a few minutes later for over a decade.
  And as said the demo usually works, after a fashion. If the case is simple (one or two screens with a few fields on each), and the person inputting the specs knows how to translate human language into the special scripting language (symbolic or graphical) of the tool (hint, isn't that what programmers do?), he can indeed create something generic that does the job. But when it gets more complex such tools all break down. They're terrible at linking data, collating stuff, basically doing anything with data collected or available outside the one screen they're working on. Basically they have the proficiency of a kid learning to program who has just been taught to write an application that has a single screen and one class. And that hasn't changed over the space of a decade.
  Even worse (from the point of view of a manager at least) is that the screen layouts these tools produce are terrible, and usually can't be customised in any way. Many companies want their logo everywhere in their in-house applications (and especially the output they produce). Most these tools don't allow for that, you got to eat the standard layout they produce without modification.
  You can of course hire programmers (doh!) to customise the generated code, but each time you generate a new iteration of the system you're working on all that customisation has to be done again from scratch (and programmers get rather irritated if they have to do the same thing time after time after time).

  Posted by: jwenting on October 11, 2007 at 11:11 PM

• Can it be you are going to start up a Revolution? Pushing
  business folks out of programming area? I can here the masses screaming. "BPM Out! No BPM! " and hunt down the process managers the street right out of their offices.
  On the other side there are the few BPM people protected by
  the Guards. In this case i think programmers will pervail - because there are more developers than BPM people.
  But in fact its an ironic fact that developers tend to make tools that might make them superflous. But in the end some programmers still are neeeded to improve the BPM software and to innovate new BPMs in order to write new tools for this ...
  Its the same with any new innovation - until it doesnt go into the bin right after it started any new inno will produce new folks carrying it on and reducing some other folks work.

  Posted by: alexanderschunk on October 12, 2007 at 05:08 AM

• I am at the receiving end of the results of BPM and BPM software. I don't use BPM software, I am just a customer/user/consumer running into businesses or, even worse, government departments, who handle my requests, applications, orders, etc. with some software or tool build according to some business process model.

  One thing is is always obvious. These systems are incredibly inflexible. They can't handle any unusual request and can't respond to exceptional situations. They are in fact rather stupid, disgusting pieces of software.

  I know the standard answer from the BPM camp is that someone failed to properly model their process so the resulting software can't handle a particular case. That is is not a failure of the BPM tools. That view is right and wrong.

  The fundamental idea of BPM, that a process has well defined borders so it can be completely modeled, is flawed. Processes can easily be open ended. Whatever you do, you can not model an open-ended process. You can not anticipate each and every special case which might arise. Processes interact with an environment they don't have under control. Conditions change. And whenever humans are involved, e.g. as customers, processes have no clue about human creativity. Humans can and do create all sorts of unpredictable interactions and dependencies.

  But even if a process has well defined borders it does not mean that it can be modeled flawlessly. From a certain complexity the probability of introducing a modeling error approaches 1.

  BPM tool vendors hype BMP and promise that BPM can do things which are impossible.
  

  Posted by: ewin on October 12, 2007 at 07:05 AM

• ewin,
  I don't disagree with your statements that modelling processes is hard... That's why people hire me ;-)

  Sometimes you can't model a process because the client hasn't got a process to model... but I think you've miss the whole point of why organizations adopt BPM. BPM is not about automating a business process and then "throwing it over the wall" to the users.BPM is about automating a business process in such a way that the process can be modified as it is used... Continuous Process Improvement is the goal, not to generate applications quickly.Yes, the first iteration is often flawed, but that's why you developed the application using a framework that will make it easier to correct those flaws.Using traditional development techniques, process flow may be governed by many hidden little snippets of code... with a BPM system the process definition is well known and "in your face". If the process is wrong, it's more obvious how to go about changing it.
  All business processes change over time... sometimes to make business run smoother, sometimes because of new regulations or new competitors. BPM is an acknowledgement of that.
  -JohnR

  Posted by: johnreynolds on October 13, 2007 at 02:49 PM

• Continuous Process Improvement is indeed what makes the consultant rich. And that's why consultants love BPM. It's a constant source of income, while at the same time providing little value to the buyer. Continuous process improvements is against people's usual buyer behavior pattern and against general goals of a business.

  You do not buy a product for your business and expect that the manufacturer or a consultant has to constantly fiddle with it while you are using it. You buy a product and use it until it finally breaks or finally does not provide any value for your business. You do not buy a van and pay a mechanic who constantly changes the wheelbase and rearranges the steering wheel while you are making deliveries.

  This means two things. First, BPM is so immature that it does not produce stable results. The results are so inflexible that they need constant tweaking. Second, judging the processes I as a consumer/buyer have to deal with, it is apparent that many business people buy process modeling as a one time service and expect the result to be stable and just work. I can understand this behavior. They paid for it, and they paid more than the cost of a van, so they expected to get something at least as lasting and as stable as an average van. Only they don't get it.

  The idea that one first goes out with a faulty process, i.e. a faulty product, and then continuously improve it (at additional cost), is also a disservice to the users. Yes, others do that too, and from time to time car manufacturers have to recall a van. But making the delivery of a faulty product part of the methodology is disgusting. Some would call it a ripoff. Eau De Snake Oil.

  I also don't buy the argument "it is hard, so you have to constantly pay a consultant". Other things are hard, too. That's why the clever people don't do them if they can avoid them. They are not worth doing. And here we are back to your original taunt. Rejecting BPM is not about hate. BPM is simply not worth doing.

  Posted by: ewin on October 14, 2007 at 02:57 AM

• erwin,
  I'll agree with you that BPM is immature... but (to paraphrase you):
  "CPI is against the general goals of a business"
  Tell that to FedEx, Ford, Toyota, etc. and I'm pretty sure that they will disagree with you.
  You are absolutely right that you shouldn't have to hire an outside consultant to tweak your process software... That's why the BPM suites have such "simplistic" tools.Unfortunately there's more to it than tools... the Business person has to know at least "some" programming theory in order to use even the simplest tools.I think the answer will be more than tools, I think it will be to add "just enough" programming to the curriculum at Business Schools.I am not sure what "just enought" programming is, and would welcome your thoughts and those of others on just what that might be.-JohnR

  Posted by: johnreynolds on October 14, 2007 at 07:24 AM

• John,

  I too must be in the minority and I agree with you. If we can build services that are as granular and specific as possible, we can use BPM to enable to the business to then aggregate those how they see fit. It seems like most of the rework comes from poor requirements gathering and analysis and therefore the fixes are usually not large fundamental changes but tweaks to business logic at a much smaller level.

  Personally I don't want to have to make these mundane changes and I believe that code that should be instrumented in such a way that changes can be made dynamically and without a recompile. If the services we produce are able to do this, and we give the ability to business users to aggregate them into true "business processes", I'm all for it. I don't think we're there yet, but I believe it to be a worthy goal. To me it represents the best of breed. Developers can concentrate on delivering discreet services while business users/analysts (who should truly know the business) are responsible for the overall process.

  But maybe I'm just delusional...

  --J.

  
  

  Posted by: jlhflyingfish on October 14, 2007 at 01:28 PM

• erwin,
  You are msising an important point. Whether or not a 'BPM tool' is used, a software solution will have to model the process. All the problems you describe with process modelling will be there whether you build the software from scratch or use a BPM tool - so they are not valid arguments against a BPM tool.
  Whether it is easier to make on-going changes with or without a BPM tool a separate issue - obviously depends on the tool.

  Posted by: manjuka on October 14, 2007 at 10:25 PM

• "Sometimes you can't model a process because the client hasn't got a process to model... but I think you've miss the whole point of why organizations adopt BPM. BPM is not about automating a business process and then "throwing it over the wall" to the users."
  It is, usually. It's seen as a replacement for "traditional" software development processes. A system is created using some automated system, thrown over the wall to the users, who throw it right back because it doesn't work. At that point the programmers are gone, having been laid off because the tool makes them redundant according to the marketing hype. Some kids are hired to modify the generated application to work properly, and fail to do so (usually). Programmers in general are blamed for their lack of skill, rather than the tool creator for selling some POS.
  That's the general pattern as I saw it happening several times when I was doing consultancy work in shops where such tools had been introduced.

  Posted by: jwenting on October 15, 2007 at 02:25 AM

• jwenting,I certainly have no reason to doubt your experiences... and I hope that you don't doubt mine.The projects that I have worked on that were based on BPM were easier to maintain than those where the process "defnitiniton" was scattered across the code base.It just doesn't make sense to me to start from frameworks that are not "process-centric" when you are implementing business processes.It would be like implementing a muli-page web application without using something like Spring's web flow.-JohnR

  Posted by: johnreynolds on October 15, 2007 at 06:14 AM

• Of course not. As always "your mileage may vary".
  Always try to find the right tool for the job, and for some jobs a generator that churns out code based on some form of business process description may be the right tool for the job. Most problems as always arrise when people (usually at management level, but programmers aren't immune to it) try to make one tool the "universal solution" to everything and force its use where inappropriate.
  That way lies disaster, an example would be using a BPM tool to generate that same web application :)

  Posted by: jwenting on October 15, 2007 at 07:07 AM

• Dear Readers,

  If you are not already bored to tears with this subject, please check out my related blog posting: User Interfaces for Process Tasks.

  -JohnR

  Posted by: johnreynolds on October 15, 2007 at 12:44 PM

• Just a side note - Apache ODE is a BPEL-WS engine. BPEL is not strictly a BPM. For one thing, it lacks abstractions for people, roles, tasks and so on, the stuff that makes up a business process.

  BPEL is more aimed at defining services orchestration. It exposes its artifacts as web services (WSDL and all), which has nothing to do with BPM. The interaction between web services is a process (like many other aspects of imperative programming), sure, but it's far from being expressive enough to describe a business process.

  jBPM is a good example of an open source BPM engine. While not its focus, It is also capable of executing BPEL.
  -
  Ophir Radnitz

  Posted by: liqweed on October 16, 2007 at 12:52 PM

This work is licensed under a Creative Commons License.