John Reynolds's Blog

Field Validation Thoughts

Here at my company our business is all about forms. I’m not talking about HTML forms; I’m talking about business forms such as loan applications, tax reporting statements, etc.

We’ve got quite a collection of applications that deal with our forms, and we’ve implemented them in everything from COBOL to Java. We’ve got desktop based applications, browser based applications, mainframe batch processes, stored procedures and web services. The one business requirement that is common among these applications is this; the rules that govern what constitutes valid input for a specific form must always be the same.

This is by no means a unique situation. For example, the rules governing an IRS Form 10-40ez don’t change just because you’re using the web-based Turbo Tax rather then the desktop edition (If they did, Intuit would be in a world of trouble).

Given that all of the forms that my company deals with are well defined (and heavily regulated), one can easily envision a common repository of knowledge from which all of our applications programmatically draw the information necessary to validate field values. If the definition of one of our forms is amended (perhaps due to a regulatory change) the change will be picked up by all of our applications to insure consistent form processing across our enterprise.

Sadly, we have no such facility. We have a substantial knowledge base regarding our business, both in terms of human resources and extensive documentation, but we do not have a machine parse-able equivalent. When the definition of a form changes, we have to go through all of our systems and manually update them, and a considerable effort is expended to make sure that everything stays consistent.

This issue is one of my pet peeves, and I’m not alone. A few months back there was a good discussion on this subject on the Java Server Faces forum, and much of this blog owes thanks to the folks who chimed in at that time.

I am concerned that we Java folks are missing the boat in not generally recognizing the linkage between field validation and the underlying business forms. To illustrate my point, consider how field validation will be handled in Java Server Faces.

JSF encourages separation of concerns. Architects are encouraged to compartmentalize the design into Model, View (Presentation), and Controller concerns, and authors can be assigned to each layer. This is great.

What’s not so great (in my view) is that the current design of the JSF validation tags assigns the responsibility for defining field validations to the page author. To validate a field value, the pages author “registers the validator on a component by nesting the validator's tag within the component's tag" (excerpt from the JSF tutorial).

This bothers me because it requires the page author to have knowledge about a field’s meaning in order to select the correct validation tag.

I am not concerned that field validation is performed in the presentation layer. Validation should generally occur as close to the user as possible to provide the best possible feedback when an error is detected. I am concerned that validation is defined in the presentation layer.

Elsewhere in the JSF tutorial the responsibility for designing forms is assigned to the model author (as it should be). The form designer defines all of the fields that make up the form bean, and most likely has the knowledge of what values are acceptable for each field.

Consider the process of developing and maintaining an application: How will the rules that govern a form’s fields be conveyed from the form designer (model author) to the page designer (presentation author)? Unless this process is mechanical, it will be prone to error.

Perhaps we need a convention for specifying validation rules within form bean definitions... We now have the Java Bean getXXX and setXXX for dealing with the value of property XXX; perhaps we need "getContraintsXXX".

With such a convention, we could develop JSF validation tags to extract rules from the model, rather then having to communicate the rules to the page authors and running the risk of introducing errors. In addition, if the rules that govern a form’s fields reside with the form definition it will be easier to insure that all of the views remain consistent (even views that are web services rather then HTML).

Note that this solution only partially addresses my base concern since it deals with the HTML forms passed to the user rather then the underlying business forms, but it's a start.

Philosophically, the Struts Validator is closer to my preferred approach. With the Struts Validator, form validations for a specific application are specified in a single file named validator.xml. Unfortunately (from my perspective) the rules are defined separately from the target form, and from a packaging perspective I have not figured out how to bind the rules to every application that uses a specific form.

Another approach that I find interesting is the XForms model for constraining values. With XForms, rich declarative validation information can be included in each form definition. For a company like ours, I can envision creating detailed XForms definitions for all of our business forms, and letting those drive validation throughout the process.

Here is a summary of my thoughts:

• Field validation of business forms is an important concern of many applications
• The business owners should specify the field validation rules - this is business logic, not presentation logic
• The validation rules associated with fields should be specified as part of the model – the rules are tied to the meaning of the data, and the meaning should remain the same regardless of the presentation
• There may be presentation specific rules that are associated with a specific implementation, but the model’s rules should not be violated - the model's rules should percolate up to the presentation layers
• We need a standard for specifying field constraints within form definitions, and for accessing those definitions - XML web services standards and XForms might provide good starting points

Update: See my later blog on JSF and Field Validation

Comments

• Using XML Schemas
  I have been spending a lot of time on this subject lately. Our recent project here at IBM involved handling a very diverse set of xml data. We create xml schemas to model them all, so that we could validate them as they came in.

  We also had to create a Web GUI to create the xml, and I really wanted to be able to validate the data in the forms off the schema, even create the HTML from the schema. I didn't find a good solution to that problem however. We used an implementation of Xforms, but don't like how it doesn't use the XML schema completely to validate the data. Instead, much of the data rules have to be recreated in an XForms model. That is redudant.

  I think XML Schemas are the best way to validate data, and which that JSF, Struts, and XForms, would include support for validating from the schemas.

  Brian

  Posted by: branley on November 03, 2003 at 07:34 AM

• attach validation to model object, and use checked exceptions
  Here is an illustration of a technique which I have found useful, elegant, and compact. I have used it in web applications only, however, and not yet in Swing applications.

  The "MyUser" class, which models the registered user of a web site. Note how its constructor throws a checked exception.
  http://www.javapractices.com/apps/web4j/javadoc/src-html\hirondelle\web4j\model\MyUser.html#line.36

  Adding a "MyUser" given user input from an HTML form :
  http://www.javapractices.com/apps/web4j/javadoc/src-html\hirondelle\web4j\ui\MyActionAddUser.html#line.24

  These examples are taken from web4j :
  http://www.javapractices.com/Topic188.cjp

  In short, in this example, data-centric classes which model items in the problem domain :
  - are immutable
  - have their "class invariant" established/validated in the constructor
  - if a validation fails, a checked exception is thrown, which contains a description of every failded validation (not just the "first one" encountered)

  Everything happens in the constructor. All of the logic/validation is there. If any failure, then the caller gets a checked exception object, which can be used to display appropriate error messages to the caller.

  The use of checked exceptions is the most economical and elegant way I can think of to perform validation.
  - the validation code is not attached to the presentation
  - the validation is attached to the class which encapsulates the data. What could be more natural?
  - if validations are common across classes, a simple refactoring removes any duplication, as usual
  - checked exceptions are for items which are outside the direct control of a program. User input (especially over the Web) seems to fall into this category. Why not make use of it?
  

  Posted by: johanley on November 03, 2003 at 10:25 AM

• Using XML Schemas
  Doesn't simply using xml schemas to validate data leave you with a problem, in telling the user what constraint they failed to satisfy?

  Constraints in xml schema are exceedingly complex and you to not only validate, but show the user where on the form they went wrong. This is where validation in the UI layer begins to make sense - in "some intermediate layer" you have no idea if the field were validation failed was actually presented to the user or was derived below the UI layer.

  There is a strong argument for moving validation into the UI layer not because its efficient but because the errors won't be mappable to user input otherwise.

  Posted by: ba22a on November 03, 2003 at 11:38 AM

• Using XML Schemas
  If you simply use the stock validators, then yes you would have this problem. One of the reasons I used IBM's alphaWorks XForms implementation, was because it actually marks each field on the HTML form as having an error. Doesn't tell you exactly what the error was, but tells you its invalid, and then you simply respond by telling the user how the data should be formatted or typed.

  But, if we are spending all this time working on JSF and Struts to validate data in forms, why not write an XML validator that uses schemas, and gives you the specific information the user would need.

  Brian

  Posted by: branley on November 03, 2003 at 02:14 PM

• Using the JSR94 API for validation
  I cannot agree more with your philosophy behind field validation. I don’t like JSF validation either.

  However as much as I like the philosophy behind the Struts Validator, it is too weak for any validations other than trivial ones in the real world.

  Dont get me wrong. Some web sites just have trivial validations It works very well for trivial validations - Just the right tool.
  (As I understand, Struts Validator cannot do dependent validations.)

  My idea of validation revolves around how that term is defined. Checking any field in isolation ( for trivial errors) is what I like to classify under validation.

  Once the filed value is dependent on or driven by another field’s value (which is what we have at where I work), it is no longer a validation – it is a business rule.

  And I like to maintain business rules in a business rules repository. Like a rule engine for e.g. ILog JRules

  And then wrap it with Rule engine API (JSR 94) and invoke from anywhere – JSF, Struts Forms, or EJBs does not matter.

  Most Rule Engines in the market have started supporting the JSR94 API.

  I would like to build a adapter for JSR94 if needed – one each for JSF, Struts or anything else that comes my way .

  The adapter does the same task all the time, it gets the RuleRuntime from JNDI and invokes the rules on Stateless or Stateful RuleSessions something like this

  RuleRuntime ruleRuntime = (RuleRuntime)
  PortableRemoveObject.narrow
  (jndiContext.lookup(“RuleRuntime”),
  RuleRuntime.class);
  StatelessRuleSession srs =
  (StatelessRuleSession)
  ruleRuntime.createRuleSession
  (“rules://rule_exec_set_01”, null, RuleRuntime.STATELESS_SESSION_TYPE);

  List resultList = srs.executeRules( inputList );
  srs.release();

  The advantage is you have a centralized corporate repository of rules, used not just across forms in a system, but reused across the enterprise – a true validation nirvana ;-D

  NOTE: You can find all this plus a reference implementation of the rule engine and tons of ways to better design J2EE apps in my upcoming book called "J2EE Project Survival Guide - Vol1 Designing the web tier" to be published in Jan 2004.

  Have fun,
  Srikanth

  Posted by: srikanth on November 04, 2003 at 08:53 AM

• Using the JSR94 API for validation
  A piece that I missed.
  Consider building a adapter for Struts and Rule Engine X

  In the Struts Form validate() method, you can invoke the Adapter method and pass in
  a rule name as a argument.

  The deployer or application assembler can bind the rule name to a rule exec set name in the Rule Engine. Thats how you get the Rule Exec Set name in my previously shown code.

  After running the rules on the rule engine, it gets back a resultList which contains the vendor specific objects indicating success or failure.

  You have to convert the vendor specific errors into Struts specific ActionErrors (Thats why the adapter )

  NOTE: Althoguh the rule engine invocation is vendor independent, the actual rules, the heart and soul of your business is coded in the rule engine vendor's propreitary langauage - which may be familar as XML or something weird. You have to evaluate the rule engine a lot. Cannot stress that enough.
  Plus the rule engines cost a fortune ;-(

  Cheers,
  Srikanth

  Posted by: srikanth on November 04, 2003 at 09:10 AM

• You'll need something more than schemas
  I think there are 2 general types of validation, one of which you can do via some type of XML schema -- ie, require only numbers, 4 characters, not empty or similar very simple statements.

  But the next step of validation, actual business rule validation is much more complex. For example -- You can claim deducation X, if you meet criteria Y, but don't make over Z but under A and your account's name is Smith.

  This type of validation would either need a rules based syntax (I think some of the Business Processing XML is trying to generate this logic), but perhaps could be solved by developing a set of web services that exposed methods like "Validate Tax Form 1049A". By using Web services you make it available from a wider range of applications without requring you to write them in a particular language.

  Following the Tax model -- you may very well have fat clients written in VB.NET and a web app written in Java and some type of crude reporting app in Perl. All done by different departments, using the languages they know best.

  Posted by: mwilcox on November 04, 2003 at 09:48 AM

• Using XML Schemas
  Finally a discussion about this. I have found that most (haven't tried all) view frameworks have some ways of validating fields, but doesn't do the job proparly.

  I have found that XML Schema's can be the right "tool", but only if used the right way. I use XML Schema's and JAXB to create all model objects. This is to ensure that there is some consistency between the model and the view "fields".

  When it comes to validation it is vital that validation errors can be reported back to the "user". I'm considering the following:

  1. Defining an error namespace for my model objects. If validation fails, the error code should be attached to the model object via this namespace.

  2. Defining a binding namespace for my model objects. Here an application specific coupeling could be made between, for example, JSF valueRef and the model. This makes it possible to relate a model field with an application field.

  Kind Regards,

  Rune Bjørnstad.

  Posted by: brylex on November 05, 2003 at 01:36 AM

This work is licensed under a Creative Commons License.