16 November 2009

Keeping the cost of functional test down

It's essential for every developers and QAs in a team to have shared understanding of the environment against which functional tests are run. Only then you can talk about testing strategies. Why? More often than not, running functional tests against production like environment proves to be quite an expensive exercise. To Measure if it's valuable exercise or not, and to come up with strategies that minimise the risk of not having production like environment requires input from both developers and QAs.

In hypothetical scenario, let one of such differences between functional test environment and production environment be non-existence of application server. One might argue that it's slow and tedious to deploy application into application server every time which does slow down the continuous build. Another might argue that abstracting away from the environment that application server provides and enforces upon is simply too labour intensive. While QAs simply not happy with risk exposed from possibly not testing the integration point between application and application server, and anything that falls off the boundary of application. This sounds like a good place to get everyone talking. It might turn out that QAs are convinced that one or two integration tests are going to be ok given that the nature of integration is indifferent across the few integration points. However the discussion turns out to end, important thing is that everyone should have shared understanding at the end.

Whether by accident or common desire, there have been some efforts from open source communities to abstract away from existing servlet interface with relatively new Java API for RESTful web services. Most of the available implementations have an adaptor that bridges between servlet (i.e. an existing application server environment) and their own abstraction. This makes it a reasonable platform to write out-of container functional tests driven by HtmlUnitDriver with modified WebClient to intercept outgoing request to application.

public class OutOfContainerHtmlUnitDriver extends HtmlUnitDriver {
  private final InternalRequestDispatcher dispatcher;
  private final ResponseHandler responseHandler;
  private final List stubbedExternalResources;

  public OutOfContainerHtmlUnitDriver(InternalRequestDispatcher dispatcher, List stubbedExternalResources) {
    this.dispatcher = dispatcher;
    this.stubbedExternalResources = stubbedExternalResources;
    this.responseHandler = new ResponseHandler();
  protected WebClient modifyWebClient(WebClient client) {
    new WebConnectionWrapper(client) {
      public WebResponse getResponse(WebRequestSettings request) throws IOException {
        try {
          if (dispatcher.isInternalRequest(request.getUrl().toURI())) {
            StopWatch stopWatch = new StopWatch();
            Response response = dispatcher.dispatch(new HtmlUnitRequestAdaptee(request));
            return new WebResponseImpl(
          } else {
            for (StubbedExternalResource stubbedExternalResource : stubbedExternalResources) {
              if(stubbedExternalResource.supports(request)) {
                return stubbedExternalResource.handle(request);
            throw new RuntimeException(String.format("External resource '%s' is not stubbed.", request.getUrl()));
        } catch (URISyntaxException e) {
          throw new RuntimeException(e);

    return client;

What I've got above is an InternalRequestDispatcher that understands difference between internal request and external request. It also wraps around the application so that it can dispatch internal request to. ResponseHandler is responsible for conversion between HtmlUnit response and the application response.

08 November 2009

Functional Javascript test without DOM environment

When it comes to testing Javascript, it feels like the whole world is ready to use one or the other implementations of DOM environment with some sort of html fixture.

Why DOM environment? Yes most Javascript interact with DOM, let that be visual feedback, or complete state transition.

Testing outcome of visual feedback can be a subtle business especially when it's dynamic and best done by human eyes. However more often than not, it's driven by state transition (e.g. CSS class manipulation).

Principle of progressive enhancement tells me the current state of DOM should be complete and functional. Javascript should be used to enhance user experience. One of which is to achieve smooth transition between states of DOM. It's not hard to see the appeal of AHAH over AJAX / JSON approach given that such transition should be supported with or without Javascript.

Sometimes though it's natural to go with the later approach. Even then result of DOM modification from JSON should be a valid state transition. Good news is that you can go far with support of CSS without much DOM modification to provide the required visual feedback.

Quite often there's nothing special about having Javascript, as long as what's described so far holds true. There will be a thin layer of code that binds Javascript to DOM events and another layer of code that modifies DOM.

In my current project, I've been writing RhinoUnit based Javascript tests that are more functional. It's MVC styled Javascript with model being a thin wrapper around DOM and view being an entry point for an event prevention/delegation to control. Of course most of the visual presentation logic is in CSS rules. Within test, model is stubbed out with simple code that maintains its local states, and the interaction is driven through the view, and assertions are made against the internal state of stubbed out model.

I found the tests were reasonably high level that is flexible enough to support refactoring, and it supported user scenarios well.

Example of one test
function shouldUpdateModelGivenMakeIsSelected() {
  var make = {
    name : "make", 
    options : [
      {value : "", text : "Make (all)"}, 
      {value : "audi", text : "AUDI (25)", selected : true}, 
      {value : "ford", text : "FORD (15)"}]
  var model = {name : "model", options : [{value : "", text : "Model (any)"}]};

  searchForm.model.searchFilterDoms = [make, model];
  forRequest(searchForm.model.getUpdateFormUri() + "/make/audi.json").respondWith({
    make : [{value : "audi", text : "AUDI", count : 0}],
    model : [
      {value : "", text : "Model (any)", count : 0}, 
      {value : "a3", text : "A3", count : 15}, 
      {value : "a4", text : "A4", count : 10}]

  var makeFilter = searchForm.model.getSearchFilter(make);
  assert.that(makeFilter.isSelected(), eq(true));
  assert.that(makeFilter.isRestricted(), eq(false));
  assert.that(makeFilter.getValue(), eq("audi"));
  assert.that(join(makeFilter.getOptions(), "text"), eq("AUDI"));
  assert.that(join(makeFilter.getOptions(), "value"), eq("audi"));

  var modelFilter = searchForm.model.getSearchFilter(model);
  assert.that(modelFilter.isSelected(), eq(false));
  assert.that(modelFilter.isRestricted(), eq(false));
  assert.that(modelFilter.getValue(), eq(""));
  assert.that(join(modelFilter.getOptions(), "text"), eq("Model (any)|A3 (15)|A4 (10)"));
  assert.that(join(modelFilter.getOptions(), "value"), eq("|a3|a4"));