Model-Based Software Testing and Analysis with C#

If software testing is checking conformance to specifications or looking for discrepancies between expected behavior and actual behavior, then model-based testing is a promising way to do it. In my opinion, models may be the way to transform software testing from an art into a science, a strong reason to like them.

In the case of .NET, the NModel set of tools produced by Microsoft Research and freely distributed through CodePlex can be of great help when it comes to modeling. Based on the theory of abstract state machines, NModel has the advantage of using a programming language to express the behavior of the system under test. That is, no UML diagrams or fancy graphical editors. On the other hand, the programmatic approach makes NModel less attractive for non-programmers despite the fact that it does include a visualization tool as part of its suite.

The standard book on NModel is Model-based software testing and analysis with C# by Jacky et al. (the “NModel book”). In this article I try to make a brief overview of the book while emphasizing perceived deficiencies of NModel as a tool for modeling real-life systems.

A little bit of history

Yuri Gurevich is a theoretical computer scientist who’s developed the theory of evolving algebras. At Microsoft Research he’s lead a team that developed a specification language named AsmL, based on abstract state machines (AsmL comes from Abstract State Machine Language). Abstract state machines are state machines working with arbitrary data. As a result of this, ASMs are capable of expressing a much broader range of computations than FSMs (finite state machines) because the latter ones suffer from “state explosion” (unmanageable growth in number of states).

AsmL provides executable specifications and is embeddable into Word documents via an extension developed by the same team at Microsoft Research. I used AsmL and it is a great language: simple and elegant, it offers great constructs to express abstract concepts while maintaining a programmatic look and feel. I might say it’s the specification heaven for the computer programmer.

Despite its elegance, AsmL has a major drawback: it is neither a programming nor a modeling language. It’s never meant to be, it was supposed just to accompany specifications written in natural language and not to provide ways to construct real systems or the models thereof.

NModel fills the gap. Consisting in a set of tools and nothing else, leveraging the .NET to the fullest extent, anyone who can program .NET can program viable models in C# or any other .NET language (thus, the title of the “NModel book” is somewhat misleading; one can construct models in Visual Basic.NET or C++/CLI as easy as in C#). But NModel is not essentially different from AsmL; it’s founded upon the same sound theory of abstract state machines.

NModel: a brief description

NModel consists in a library and a set of tools. The library contains elements to specify the model. Such an element is the ActionAttribute attribute telling that a certain method represents a state transition. The tools have various purposes: graphic state explorer, test generator, test runner.

With NModel, the designer does not represent states explicitly. He represents data and changes upon the data and NModel computes the states automatically. This represents a great advantage over graphical modeling tools that start with states and end with data. With NModel, the designer models functionality directly, he doesn’t start from functionality to build diagrams only to arrive back at functionality. In order to facilitate state analysis, NModel ofers a graphical exploration tool, though.

The model, in NModel acceptance, is nothing else than a .NET custom library. The modeled elements are classes. The state transitions are methods and the state data consists in class fields. This permits to explore the model apart from NModel and, in fact, NModel itself encourages that. For example, NModel exhibits only one algorithm for state traversal but anyone can implement other algorithms with ease since NModel offers all that’s necessary to explore the state space in any desired manner.

Beside automatic state construction, NModel offers tools for model-based testing. Firstly, the designer writes stubs that act upon the system under test as dictated by the current state. Secondly, he uses the test generator that traverses the state space and, while traversing, acts upon the system under test via the above-mentioned stubs.

This is much more productive than writing test scenarios by hand because the tester can leverage the exploratory algorithms of NModel to generate a larger amount of scenarios in a shorter period of time while obtaining a better coverage of possible states and transitions.

The “NModel book”: a short overview

NModel comes accompanied by thorough documentation of the library but it lacks a tutorial explaining how to use everything. The “NModel book” fills this role.

The book is divided into five main parts:

1. Overview
2. Systems with Finite Models
3. Systems with Complex State
4. Advanced Topics
5. Appendices

Overview

This section of the NModel book introduces the reader into the topic of modeling. It explains the role of modeling in analysis and testing, it gives examples on how modeling does a better job at detecting some errors than traditional methods and it shows how modeling is useful in some cases harder to tackle by other means (such as design defects).

Systems with Finite Models

This section covers the issue of state machines with a small number of states, the ones that can be explored exhaustively. Topics like analysis, modeling, exploration, selection and testing are included.

Systems with Complex States

This section begins to reveal the power of abstract state machines and their advantage over finite state machines: the states of ASMs may contain complex data. The section covers modeling, analysis and testing systems with complex states.

Advanced Topics

As the name suggests, this section covers advanced topics: model composition, modeling objects (useful in conjunction with the OOP paradigm) as well as handling non-determinism.

Appendices

There are two kinds of appendices:

• a library reference –AND -
• a description of the tools included with NModel:
  • mpv, the Model Program Viewer
  • otg, the Offline Test Generator
  • ct, the Conformance Tester

NModel: friendly criticism

The first criticism of NModel is acknowledged by the autors themselves: the tool does not have other traversal algorithms other than the postman tour algorithm. However, a programmer can create new traversal algorithms with what the NModel library offers.

The second criticism of NModel consists in the lack of expressing parallelism. NModel offers the composition mechanism (which is, in fact, a Cartesian product over the state spaces adorned with some rudimentary “synchronism”) but the mechanism is so awkward that the authors themselves use it solely for state selection and not much else.

The third criticism of NModel consists in the lack of expressing structural composition: to be able to construct the whole from its parts. For example, provided we have a sub-model A and a sub-model B, there’s no way to produce a model C composed of A and B working together. Needless to say, C should be at a higher level of abstraction than the sheer composition A x B.

OBS: NModel does have a mechanism for state abstraction, i.e. for grouping a (potentially infinite) number of states into a single one. But for an abstraction based of structural composition as explained above, there is no support. That is, if one wants the model C from above, he must write it from scratch by hand.

The fourth criticism of NModel consists in the lack of expressing structural decomposition: to be able to break the whole into its parts. For example, provided we have a model A, there’s no way to break it into sub-models B and C that, when “put together”, do the same thing as A does. Needless to say, B and C should be at a lower level of abstraction than a sheer division of A’s states into two separate groups.

OBS: NModel does have a mechanism for state transition refinement via features and composition. Yet, such refinement applies to isolated states and transitions. It doesn’t work per entire groups of states, let alone per entire models. Most importantly, such refinement is not automatic with the exception of exploiting action matching via identical action names (the basic “synchronism” mechanism mentioned above). While yielding some relief in terms of decomposition, state transition refinement does not preclude the need for more elaborate ways to express structural decomposition as it happens in traditional top-down design methodologies.

The fifth criticism of NModel consists in the poor support it offers for the well-known categories of testing. The software testing industry has long established several distinct kinds of testing: parameter, limits, boundary, functional, stress, robustness, etc. While the test designer can implement them via test strategies, there is not much more support beyond that.

The sixth criticism of NModel consists in the poor support for repro scenarios. Testing with NModel follows two paradigms: predefined testing, quite reduced in scope and applicable to models with small number of states, and on-the-fly testing, much more powerful and customizable via test strategies. By far, the best option is on-the-fly testing in terms of both coverage and chances to find defects. Yet, when it comes to reproducing faulty scenarios, the support offered by NModel is sparse and cumbersome to use.

Conclusions

Model-based testing is a promising method to bring rigor and precision into the realm of test design. Despite requiring a higher level of expertise and more effort upfront, using models pays off due to a higher productivity in terms of test case generation and feature coverage.

NModel is a free suite of tools provided by Microsoft Research that can be used for model-based analysis and testing in the .NET ecosystem. While the NModel library is well documented by a help system provided with the tools, it is the “NModel book” the piece of work which contains a gentle yet thorough introduction into the world of modeling in general and NModel usage in particular.