William C. Wake's Blog

Testing Archives

Extreme Test Makeover

The idea is that you bring your code and tests, ready to run. We'll have a number of people who are experts in unit testing and acceptance testing, to help you improve and extend your tests. Some people who are planning to help include Ward Cunningham, Ron Jeffries, Jim Shore, Janet Gregory, Charlie Poole, and others.

Our ideal is that you bring production code and tests, so you can walk away with concrete, ready-to-go improvements. But you can bring whatever you want that's interesting enough to test.

The sessions will be 90 minutes long, including five minutes up-front for experts to talk, and ten minutes at the end to record lessons learned. Some sessions will be intimate, you and the expert and perhaps a couple people looking over your shoulder. For others, we plan to have a projector and microphone so many others can look on.

Brian has set up a yahoo group: test-makeover to let people connect with the artists. There will be a signup sheet at the conference too.

I hope you'll join us!

Brief review - Fit for Developing Software

[My bias disclosure - I know both Rick & Ward, I was a reviewer, and I've written for their publisher myself. This review is substantially as posted on the agile-testing group.]

Fit (see http://fit.c2.com) is a testing framework that Ward Cunningham developed. A test author writes tests as tables in a document that can be converted to HTML (e.g., Word, Excel, text editor, etc.). The programmers develop fixtures that connect to the system under test. Fit mediates the tests and the fixtures to run the tests, and captures the results. It colors in the document using red/yellow/green to show what happened.

The book is in two halves. The first half is targeted at test authors: from the user perspective, how does Fit work? It covers the basics of tables, fixtures, error handling, and so on. Then it goes into an extended example (several chapters) following a team developing rental software. The first half closes with advice about designing better tests.

The second half is targeted at programmers. While programmers should really read and understand the first half of the book, test authors will probably at most skim this half. This half starts by explaining how to implement various types of fixtures. Then it continues the earlier rental software example by showing the fixture code that would be developed. Finally, this half closes with some advanced topics: Fit's architecture, custom fixtures and runners, and model- based test generation.

The authors have done a good job explaining Fit from both the test- writing and programming perspectives. The text is clearly written, using plenty of examples, frequent breaks for Questions and Answers along the way, and exercises at many chapter ends.

This book is unique. While you can find information about Fit and fixtures on the web, what's on the web is much less readable than what this book provides. The book also gives you an extended example and helpful advice from two experts.

If you are considering Fit, or just want to understand its philosophy, this book provides the clearest explanation I've seen. For test authors, the first part of the book justifies the whole price. For programmers who need to understand how and why fixtures work, it's even more of a bargain.

Fit for Developing Software, by Rick Mugridge and Ward Cunningham, with a foreword by Brian Marick. Prentice Hall, 2005. ISBN 0-321-26934-9.

Fit code, part 8 of 8 - RowFixture

Data and Abstract Methods

The fixture holds three bits of data: an array containing the results of the query, a list of surplus items, and a list of missing items. From showing usages, I see that the list of surplus items is a list of domain objects; the missing list is a list of Parses.

The first abstract method is query(). It is responsible for producing an array of the "actual" results. The second abstract method is getTargetClass(). It returns the class object representing the type of the row. It's abstract for an interesting reason: the parent class ColumnFixture defines that method to return the type of the fixture itself. That would just lead to weird errors. By making it abstract, it forces the user to override it.

This is an interesting twist - usually my abstract methods are at the top of the hierarchy, and may get filled in along the way down. In this case, the method is becoming abstract in the middle.

In a sense, that happens because RowFixture and ColumnFixture have a slightly strained relationship. Maybe I'm just not getting why the latter is an example of the former; it feels like the inheritance is more for implementation than anything.

doRows() - The Overall Algorithm

Along the way, this method calls two overloaded list() methods: one for making a list of Parses, the other for making a list of objects. This parallel structure (methods for each of the two main data structures) continues across the class.

Method doRows() calls buildRows() to add in new rows for the surplus values. This method works by building up a "fake" head of a parse chain, then adding each item to the last one in the list. In the end, it throws away the head and returns the interesting part of the list, which gets attached to the end of the table. This seems like a little pattern worth remembering if I ever need to add rows to a fixture.

match() - The Heart

Since this algorithm looks complicated, I'm going to start by just looking around. First, what are the places that call it? By doRows() certainly, since that's how we got here. Then it's called recursively at two places inside match() itself. The good news is we don't have some sort of pair of mutually recursive methods.

Recursive algorithms have a base case and a recursion case. The recursive case here is just incrementing column, and passing along lists. The column is always incrementing, and the first if says that if we've exceeded the number of columns, we should just do a check on the lists. That makes it look like we'll always terminate: we either increment column, in which case we'll eventually stop, or we do something that doesn't recurse, which will stop as well. (Or rather, if it doesn't, it won't be because of the recursion here.)

The other thing to look at on these recursive calls is the lists. We know the column gets bigger - do the lists get smaller? One case passes on the originals, so we know it's no bigger. The other is trickier - I see things that seem to indicate that the lists will shrink (tests for 0 or 1 item), but it's not obvious that it must be so without a little digging.

So, from the top of match(): the first case we mentioned before - if we're past the number of columns, do a check() on the currents lists. (We'll come back to that method later.) The second case says that if the current column binding is null, move on to the next column. The third and final case is where the meat is: we're in a column in the middle, trying to match.

So, we build up two maps: one for the expected, one for the computed ("actual"). Each map has a list of items that have the given value at the chosen column. We pull out the keys in either list, and work our way through them. Here, there are four interesting cases:

1. The expected list is empty - we have a value found only in the computed list, so add it to the surplus list.
2. The computed list is empty - we have a value found only in the expected list, so add it to the missing list.
3. There is only one value in each list, and they have the same key (by how we got here) - check this row (actual vs. computed).
4. Finally, both lists have more than one item with the same key value - recurse, but only on the list of items with this same key value. (Now I can see that I'm recursing on a list that's no bigger. It could be the same size if all the keys are the same.)

eSort() and cSort()

Each routine produces a Map, from key values to a List of Objects (either domain objects or Parses). The bin() method takes care of putting items in the map. That method expands an Array into a List; the RowFixtureTest mentions a bug in that neighborhood and I suspect this is to address that.

The sort() methods handle exceptions and rows with no value in a particular cell.

Back to the Algorithm

check()

Leftovers and Learnings

The other big thing for me to wonder is how I'd have done a similar fixture. I think I'd have expected a simple set. The problem is, that's fine for the query() side, but not so good for the "expected" side: how would you get from those values to construct the objects to compare as sets? (Knowing the contents of an object's fields and return values from its methods doesn't tell you how to construct it.)

Another alternative would be to get the query values, and match each one up against the rows. The naive algorithm for this is a little slow (n^2). It might be a bit simpler. I suspect its report wouldn't be as nice.

The current algorithm is able to take advantage of partial matches - if enough data cells make it a unique match, it can then know it has the "right" element even though some of the fields/methods are wrong.

Closing Out...

Fit code, part 7 - ColumnFixture

doRows() - Capture the Header Row

If the cell ends in "()", it's a method call, and set via bindMethod(). Peeking inside there, it camel cases the name (so "shirt size()" becomes "shirtSize()") and uses the TypeAdapter.on() method to create the adapter.

Otherwise, the cell is assumed to be a field name, set via bindField(). That helper method also camel-cases the name and uses the "field" version of TypeAdapter.on().

Any problem in the header parsing marks the cell with an exception.

doRow() - Handling the Basics

Execute() is to be called before processing the first column that represents a method name: you can have a bunch of inputs, use execute() to make things happen, then check the outputs. If you don't override execute(), then you either have to know which column is first and make it kick things off (which is brittle), or you let each output column compute "from scratch". (Reading it here makes me wonder if I've been diligent about this in all my ColumnFixtures:)

If there are any unhandled exceptions, they're attached to the first leaf cell of the table.

doCell() - Per-Cell Handling

I'm going to write a test for it, but it took a couple minutes to figure out how to do so. I think what I'll do is create a table with a default value for x and y, leave x blank, and have execute() print the value of y. If it's called when x is processed, it'll print the default value rather than the one that was set.

(Pause)

OK - I'm back, and it does act like I expected from the code - execute() is called for a blank cell. I guess that makes sense to do if the blank cell were a call cell (like plus()); I'm not clear on the value for an input cell.

Back to doCell(): if the TypeAdapter is null, it ignores the cell. If it's a field, it parses the text and sets the field. If it's a method, it calls check().

check() - Calling execute()

What I've learned

• Don't forget about reset() and execute()
• Execute() is a little word in the face of blank input cells.

Fit code, part 6 - TypeAdapter

C# Fit

TypeAdapter

This gives the unification of fields and methods. In Fit, you can have a ColumnFixture with a field, and it has an implicit setter ("name") or getter ("name()"). Or you can have a method (also "name()"). For most purposes, we don't care what it is, we just want to treat it as a setter or getter.

The TypeAdapter has five fields:

• target - the fixture this adapter is "on" (set for a field or method, but not a type)
• fixture - the fixture this adapter is "on" (always set)
• field - the Field (only set for field references)
• method - the Method (only set for method references)
• type - the type, always set

Methods

The set() method does a field set. (Could we extend this to call "setter methods" with a signature like setFoo(Foo value)?)

The invoke() method assumes that a method is set, and calls it with no parameters.

The parse() method asks the fixture to parse the string according to type. In C#, parse() is something each type (even primitives) define. I'm sure that simplifies some of this code.

So let's say we have a ColumnFixture where phone() is of type PhoneNumber. How do we make that get parsed naturally? It looks like it works its way back to Fixture, which has a parse() method. So the ColumnFixture overrides it, and checks for an attempt to parse a class it knows about.

It seems like we could do some fanciness here, too, pushing an attempt to parse onto the domain classes. (So let Fixture.parse() take a look for "type.getMethod("parse")"; if we don't want that we could subclass and override to avoid it.)

Primitives and Their Classes

The last one is the only exception: Arrays. There, the parse() method tokenizes it by looking for commas. Like so many other places in Fit, it trims spaces. Each element of the array is given its own TypeAdapter. The toString() method puts the commas in when printing it out.

Reflection

Fit code, part 5 - ActionFixture

Fields

The class has three fields:

• cells - a Parse
• actor - a Fixture
• empty - an array of Class

Actor holds the object created by a start() action. Notice that it is static - that is what lets separate ActionFixture tables keep working with the same object without repeating start in every table.

Empty is the easiest - it's just an empty list so that things that want a list of argument types can have one. I marked it final, since it's never changed.

Methods: doCells()

The fixture invokes the method on itself by "getClass().getMethod()" - looking for the method on itself. This is a place where the Java version is nicer than the C# one. The C# version hard-coded that line to the equivalent of "ActionFixture.class.getMethod()". That meant that a subclass of ActionFixture would only have access to the four methods ("start" etc.) originally planned. The Java version lets you extend this vocabulary easily.

Another thing to notice is that the fixture calls getMethod() on cells.text(), not camel(cells.text()). That's a pity - my extended vocabulary has to be spelled exactly. (I don't think the rules for camel casing are consistent throughout. I'm probably getting hung up from Fitnesse experience - I think it may have slightly different rules.)

Methods: Actions

Enter() looks on the actor for a one-argument method it can use as a setter. It creates a TypeAdapter, which knows how to parse objects, passing it the cell text. Then it invokes the setter.

Press() invokes the named 0-argument method on the actor.

Check() assumes its 0-argument method is a getter, fetches the result, and passes it to Fixture's check() routine, which does the comparison and cell coloring.

Methods: method()

Next time, I'll take up TypeAdapter.

Fit code, part 4 - Fixture

Fixture: Fields and Two Helper Classes

There's a field counts that has counts of tests passed, failed, and exceptions/errors. The Counts class is just a data bag for these things. When a fixture calls wrong(), for example, the count is incremented.

The last field is args, which has the arguments from the first row of the fixture. The method getArgs() returns a String[] and lets a fixture use them. I don't think this is in the C# version yet but we definitely use that sort of thing there.

There's an internal class RunTime. It takes a snapshot of the current time. Right now, the only use of this is to put it in the summary, under the key "run elapsed time". Presumably some fixtures pull the RunTime object back out, and use toString() to display the elapsed time. But nothing in the standard distribution appears to use it directly. (Fit.Summary will display the elapsed time when it dumps the summary table.)

Starting Fixtures

Method getLinkedFixtureWithArgs() tries to load the fixture named in header.text(), then it sets up the arguments (for getArgs().

The method loadFixture() takes the name of the fixture, and attempts to "new up" the named fixture via reflection. Between the last method and this, I'm worried by what I don't see: what routine uses the camel method? That suggests a test: let's load "fit.ActionFixture", "fit.Action Fixture", and "fit.Action fixture" and see what happens. From what I understood going in, all three should be ok. From what I'm seeing here, I don't see what would make that work.

Why did I expect this? Because ColumnFixture does it for column names. It turns out that's not a good enough reason. The test shows that only "fit.ActionFixture" loads.

Up to interpretTables() again. It does getArgsForTable() - again. There's even a comment to that effect. I don't see why it should be necessary, though. Actually - it's all a little subtle, and I'd say the comment is misleading. The comment says, "// get them again for the new fixture object". But really, that's what we did in getLinkedFixtureWithArgs(). Now we're getting the arguments for the original fixture.

It works like this: when FileRunner starts, it runs doTables() on a new Fixture object. That's the object that tries to pull fixtures from tables and run them. When the first table is seen, its arguments are pulled out and given to the corresponding fixture. But then they're also copied back to the initial fixture as well. I imagine they're actually rarely needed there.

At any rate, interpretTables() then calls doTable(), which does a straightforward job of working its way into doCell(). Finally, it calls interpretFollowingTables().

InterpretFollowingTables()

All this work seems a little off - it seems the Fixture class is paying for interpretation that a particular table wants. I'm a long way off from looking at DoFixture, but if that's the table that should be first, it seems to me like it should pay for this complexity. I know I'm second-guessing here...

Check()

Up next...

Fit code, part 3 - Parse and Fixture

Parse

// This test shows offset isn't applied the way I expected
   public void testOffset() throws Exception {
        int offsetToData = 2;
        Parse p = new Parse(
                "xx
data
",
                Parse.tags, 
                0, 
                offsetToData);
        assertEquals("", p.leader);
    }

// This test shows cells with embedded tables "go away"
   public void testInnerTables() throws Exception {
        Parse p1 = new Parse(
"
stuff plus
");
        Parse p2 = new Parse(
"
stuff " + 
"
 inner
plus
");
        
        Parse cell = p1.at(0,0,0);
        assertFalse(cell.body.equals(""));
        assertEquals("stuff plus", cell.text());
        
        cell = p2.at(0,0,0);
        assertFalse(cell.body.equals(""));
        assertEquals("stuff inner plus", cell.text());
     }

FixtureTest

  assertEquals("     ", Fixture.escape("     "));

But there's a little more going on in Fixture...

Fixture

The core of this is: doTables() calls getLinkedFixtureWithArgs() and interpretTables(), which (eventually) calls doTable(), which calls doRows(), which calls doRow() once for each row, which calls doCells(), which calls doCell() once for each cell. In Fixture, doCell() calls ignore(), which marks the cell gray. ColumnFixture(), for example, overrides doCell() to do something interesting (like look for expected results).

I'm going to skip over how the first table gets loaded and interpreted - it looks interesting (i.e., tricky:) Instead, I'll peek down to a section labeled "Annotation". This area contains methods that can mark and color cells: right(), wrong(), info(), ignore(), error(), and exception(). I've seen these called in several of the standard fixtures before.

I see where exception() puts the stack trace into the cell. That gets SO ugly when something goes wrong. (It makes the cell huge, full of scary content useful only to programmers.) Maybe someday I'll take a whack at a more readable version.

Utilities

assertEquals("twoWords", Fixture.camel("two words"));
assertEquals("MiXedCAsE", Fixture.camel("MiXed cAsE"));
assertEquals("aFewWordsTogether", Fixture.camel("aFewWordsTogether"));
assertEquals(
    "acronymsLikeHTMLStillUppercase", 
    Fixture.camel("acronyms like HTML still uppercase"));

There's a parse() method that appears to handle only Strings, Dates, and ScientificDoubles. I know that C# works a little differently, since parse() is more integrated.

There's a check() method that looks too complicated to understand in 30 seconds. It uses a TypeAdapter, which is another class I want to look at soon.

Finally, there's a method to get the arguments from a Fixture. This is new - it used to be that the first row had the fixture only. (Rather, you had to parse any arguments out yourself.) Now that's built in, accessed via getArgs(), which returns a String array.

That's it for today. Next time, I want to dig into how fixtures get started (since this has changed some), and into the check() method.

This work is licensed under a Creative Commons License.