Package edu.utep.cs.utjml.rat

Contains the source code to generate test cases randomly for Java classes.

See:
Description

Interface Summary

Constants	An interface to define constants.
Denotable	An interface to uniformly represent various kinds of Java values and objects that can be part of a test case.
RatListener	The listener interface for receiving test case generation events.

Class Summary

ArrayValue	A class to represent array values.
ArrayValueCreator	A class to create array values.
BooleanValue	A class to represent boolean values.
ClassValueCreator	A class to create class values.
IntValue	A class to represent int values.
Main	Provides the entry point to the JML/JUnit-based random test case generator for Java classes.
MethodCall	A class to represent method calls.
MethodInfo	A common information needed from JMethodDeclarationType and CMethod in this class.
RatAdapter	An abstract adapter class for receiving test case generation events, aka RAT events.
RatClassLoader	A special class loader to take into account of rat-specific options such as -C (--classpath).
RatMessages
RatOptions	This class is automatically generated from RatOptions.opt and contains member fields corresponding to command-line options.
RatTestResult	A class for collecting the results of executing test cases.
RatTestRunner	A JML/JUnit test runner class.
RatTestRunner.JmlResultPrinter
StringValueCreator	A class to create string values.
TestCase	A class to represent test cases.
TestCaseCreator	A class to create a set of test cases for a given method.
TestClassGenerator	A class for generating JML/JUnit test driver classes.
TestClassGenerator.NameGenerator	A class for generating unique names for test methods.
TestDataClassGenerator	A class for generating JML/JUnit test data classes.
Utils	A utility class that provides various static methods.
Value	An abstract class to denote various values.
ValueCreator	A class to create various values.

Exception Summary

Main.RatClassNotFoundException

Unchecked version of ClassNotFoundException.

Package edu.utep.cs.utjml.rat Description

Contains the source code to generate test cases randomly for Java classes. The test case generation tool, called rat, implemented in this package takes a Java source code file and produces JUnit test class source code files to test the public methods (and constructors) of the class defined in the input file. Given an input file, say X.java, two output files are generated: a test oracle class file and a test data class file, named X_JML_Test.java and X_JML_TestData.java, respectively.

The test oracle class implements a test oracle for the class to be tested. For a given test case, a test oracle determines the test result (i.e., test success or test failure). The generated oracle class contains one test oracle method for each method to be tested. The test oracle method determines test results by observing the execution of the method under test. In general, if the execution of the method with the given test case doesn't result in a runtime assertion violation, then the test succeeds; otherwise, the test fails. That is to say, runtime assertion check is used as a test oracle. The test data class provides a set of test cases for each method to be tested. The test cases are generated randomly, and each test case is implemented as a separate JUnit test method.

Basic Use

The rat tool provides both a command-line interface and a programming interface. The simplest way to use the rat tool is to run the main class, Main, on the command-line by also specifying the Java class to test, as shown in the example below.

java edu.utep.cs.utjml.rat.Main T.java

This will produce both the test oracle file, T_JML_Test.java, and the test data file, T_JML_TestData.java, in the current working directory. The tool accepts many command-line options including one that specifies the directory in which to produce the output test files; refer to its help message with the option --help.

To perform the actual testing, simply compile both output files with a Java compiler (e.g., javac) and run the test data class with a Java interpreter (e.g., java).

javac T_JML_Test.java T_JML_TestData.java
java T_JML_TestData

Remember that the test data class is a JUnit test class, so one can run it with any JUnit test runner classes, such as text-, swing-, or awt-based test runner.

Test Coverage

By default, the rat tool generates a fixed number of random test cases for each method to be tested. It doesn't use any test coverage criteria in generating test data, and thus doesn't check for duplicate or redundant test cases. However, if there exists a bytecode file of the target class, produced with jmlc or utjmlc, the tool prunes the so-called meaningless test cases from the set of generated test cases; a meaningless test case is one that doesn't satisfy the precondition of the method, and such test case is inadequate to testing the method.

The rat tool can be directed to generate a minimal number of test cases that collectively satisfy a certain test coverage criterion. The only coverage criterion currently supported by the tool is called a postcondition condition coverage. This coverage criterion is similar to the condition coverage except that it uses the postcondition of the method. With this criterion, the tool tries to generate a (minimal) set of test cases that covers all possible combinations of values of the postcondition's "atomic boolean expressions". For example, if the postcondition is of the form x && y (where x and y are boolean variables), the combinations of x-y values to cover are:

(true, true),
(true, false),
(false, true), and
(false, false).

The tool tries to generate a set of test cases that collectively cover all the above possibilities. However, it is frequently impossible to achieve a 100% coverage, because of mutual constraints among the variables appearing in the postcondition.

Two steps are required to let the tool use the postcondition condition coverage:

1. compile the target class with utjmlc with the option -Xtgen, e.g.,
   java edu.utep.cs.utjml.compiler.Main -Xtgen T.java
   
   
2. run the rat tool with the command-line options -Xc or -Xlc, e.g.,
   java edu.utep.cs.utjml.rat.Main -Xc T.java

The -Xtgen option of utjmlc lets the compiler generates bytecode that produces test coverage information at runtime. This information may be used by testing tools such as rat. Both options of the rat tool (i.e., -Xc or -Xlc) enables the postcondition condition coverage. However, they differ in the amount of test coverage information that the tool shows to the user; the first shows only a summary of coverage information for each method, while the second also shows the coverage information of each individual test case. A sample output is shown below. (The command rat is an alias for java edu.utep.cs.utjml.rat.Main.)

> utjmlc -Xtgen Sample.java
> rat -Xc Sample.java
** class Sample **
== method doSomething ==
No. of atomic exprs: 1
No. of attempts: 37
No. of meaningless: 13
No. of test cases: 2
Coverage: 2/2 = 1.0
== method doSomethingElse ==
No. of atomic exprs: 2
No. of attempts: 61
No. of meaningless: 31
No. of test cases: 2
Coverage: 2/4 = 0.5

Programming Interface

The main class Main also provides APIs, such as Main.compile(String[]) and Main.compile(String[], edu.utep.cs.utjml.rat.RatListener, java.io.OutputStream), to allow for other Java code to call the tgen tool directly. The class implements the Observer design pattern to let the caller observe the progress of test case generation, including coverage information. To observe the test case generation process, one needs to create an RatListener object, and then register it with the edu.utep.cs.utjml.rat.RatListener#addRatListener method or pass it as an argument to the compile method. (see an example below).

import edu.utep.cs.utjml.rat.*;

public class SampleUsage {

  public static void main(String [] args) {

    // create an observer to display the progress of test case generation
    RatListener listener = new RatAdapter() {
      public void startClass(String name) {
        System.out.println("** class " + name + " **");
      }
      public void startMethod(String name) {
        System.out.println("== method " + name + " ==");
      }
      public void testcase(String coverage, boolean added, TestCase tc) {
        System.out.println("  " + (added ? "+" : "-") + coverage);
      }
      public void coverage(int noexprs, int attempt, int mncount, int nocases) {
        System.out.println("No. of atomic exprs: " + noexprs);
        System.out.println("No. of attempts: " + attempt);     
        System.out.println("No. of meaningless: " + mncount);
        System.out.println("No. of test cases: " + nocases);
        System.out.println("Coverage: " 
          + nocases + "/" + (int) Math.pow(2, noexprs)
          + " = " + nocases / Math.pow(2, noexprs));
      }
    };

    // compile Foo.java, directing error messages to System.err.
    Main.compile(new String[] {"Foo.java"}, listener, System.err);
  }
}