Skip to main content

Automated visual verification is hard

Posted by robogeek on December 21, 2005 at 4:47 PM PST

In the Quality Team we try to automate our testing as much as possible. This is easy for tests of the core library or other functionality where there's no GUI. But when you bring in a GUI like for AWT/Swing tests then the test complexity goes up dramatically, because for some scenarios you need to verify the graphics rendered correctly.

There's a general strategy we have in GUI testing we call "layered testing". There are always aspects to the GUI API's that we can test programmatically. E.g. if you click on a Button, you can programmatically test the ActionListener gets called, etc. We use that trick to avoid automating visual verification in most cases.

But I've been thinking about a really hard problem in this area for awhile. A few months ago I watched a video shot by Scoble (that Microsoft guy) about this ??sparkle?? product they're going to ship (they==Microsoft). I don't remember the name of the technology, but that doesn't matter. The capability is to change GUI components so they render as a pseudo-3D animation. In the video the team showed radiobuttons who would dance around if they were selected, or listboxes who'd been subclassed beyond recognition to shift their contents around a 3D space.

Looks cool ... but if you can't afford to test it, then how good will the result be?

That is -- what I've observed from 16 years in software development is there's always a limited budget for testing. In testing the GUI you can rely on manual testing, but it's labor intensive and your budget quickly runs out. The common quip in testing is "Testing is never finished, only abandoned" which is what happens when the budget runs out. If you can automate the testing then your budget ought to stretch further (except that automation doesn't come free, as it carries some overhead) and you can do more testing within your budget and time constraints.

Which gets me to dancing buttons.

Today buttons and other GUI components don't dance, they're rather still instead. But we can see the writing on the wall. CPU's are getting faster, memory more abundant, and more importantly graphics acceleration becoming relatively common. In a relatively short time it will be fairly common for GUI's to no longer be flat 2D still-lifes. Instead they'll be 2.5D dancing animations, and maybe they'll sing too with sound output.

I can just see the marketeers salivating over that coolness.

Anyway ... I'm wondering how we're going to test that capability when it becomes ubiquitous enough to worry about it.

With the flat 2D still-life style components we have today it's hard to automate visual verification. You can take a screen capture of the component and compare it against a known-good image. But there's a lot of caveats and overhead so in practice it's not that simple. But with dancing (and maybe singing) GUI components the complexity is amplified even more. It's not a still-life, but it's moving, and when you make the screen capture it's like trying to shoot a moving target.

What are we going to do when GUI's are dancing and singing away? How will we be able to automatically verify it's dancing the right way and not stumbling around?

That's what has been in the back of my mind since I saw Scoble's video.

Related Topics >>