The commercial
Parasoft Jtest Java testing tool won numerous awards and has
been adopted by thousands of development teams worldwide. Its
clients include IBM, HP, and over 10,000 companies worldwide. On 26
March 2002, Parasoft Co.
announced the commercial
Jtest
4.5 Java testing tool, which has been popularly used in
industry. Jtest 4.5 allows the user to specify method-call
lengths of generated tests from one up to three. When setting
method-call lengths as three, the test generation time is long and
the number of generated tests is large for relatively large
programs. On January 2004, we proposed a novel definition of
redundant tests based on method inputs (different from
traditional definitions of redundant tests based on structural
coverage) and published our preliminary results (in a technical
report) on detecting a high percentage (about 90%) of redundant
tests among tests generated by Jtest
4.5 and later published the more comprehensive results in our
ASE 04 paper; these identified redundant tests increase the
testing time without increasing the ability to detect faults or
increase confidence. Parasoft
announced Jtest
5.0 on 3 February 2004 and Jtest
5.1 on 28 June 2004; neither of these new versions allows the
user to configure the method-call lengths for generating a large
number of test inputs any more. We expect that the new techniques in
our ASE 04 paper can be ultimately incorporated by the existing test
generation tools to improve testing performance greatly, benefiting
software practitioners.
Updates: Mar 17, 2005: Parasoft Jtest 5.1 won
Software Development Magazine's 15th
Annual
Productivity Award.
Nov 5, 2004: Upon Parasoft's invitation, Tao
Xie visited Parasoft and gave a presentation on "Strategic
Automated Software Testing in the Absence of Specifications".
Sept 16, 2004: We were notified by Parasoft
Co. that Parasoft Jtest 6.0 (internal version, not released
yet) has fixed the test redundancy issue identified by us and added
back the option to generate long method-call sequences.]
Tao Xie, Darko Marinov, and David Notkin. Rostra: A Framework
for Detecting Redundant Object-Oriented Unit Tests. In
Proceedings of the 19th IEEE International Conference on
Automated Software Engineering (ASE
2004), Linz, Austria, pp. 196-205, September 2004. [PDF][BibTex] ©2004 IEEE
Within our knowledge, our ASE 03 paper is the
first proposed approach of using inferred program semantic
properties (other than structural coverage) for helping test
generation and test selection for inspection. Automated test
generation tools are able to produce and execute a large number of
test inputs that extensively exercise the unit under test. However,
without a priori specifications, developers need to manually verify
the outputs of these test executions, which is generally
impractical. To reduce this cost, we developed unit test selection
techniques to select a valuable subset of automatically generated
test inputs. Then developers can verify their outputs, equip them
with test oracles, and put them into the existing test suite. In
particular, we feed dynamically inferred program behaviors in the
form of specifications (e.g., the ones inferred by Daikon)
to a specification-based test generation tool (e.g., Parasoft
Jtest) so that some benefits of specification-based testing
can be achieved without requiring specifications. On 2 March 2004, Agitar Software Inc. announced
the
commercial Agitator
testing tool for Java. The tool automatically generates initial
tests, infers Daikon-invariant-like
observations,
lets developers confirm these observations to assertions, and
generates more tests to violate these inferred&confirmed
observations.
Updates: July 17, 2006: Agitar people published
an ISSTA
06
paper on Agitator.
Feb
7, 2006: Agitar CTO Alberto Savoia
gave a guest lecture at
Tao Xie's CSC
591T course at North Carolina State University.
April
7, 2005: Agitar CTO Alberto Savoia gave a guest
lecture at UC Berkeley on Unit/Developer Testing, describing
Agitator's philosophy.
Mar
17, 2005: Agitar’s Agitator and Dashboard 2.0 won
Software Development Magazine's 15th
Annual Jolt Product Excellence Award.
Nov
17, 2004: Kent Beck gave a presentation on "Developer
Testing" at Developer
Testing Forum. Listen
to
it!
Aug 19, 2004: Agitar CTO Alberto Savoia presented
“Beyond
JUnit:
The Future of Developer Testing for Java” at JavaOne 2004. He
attracted a standing-room only crowd of more than 600 interested
developers.
Aug
2, 2004: Kent
Beck, the creator of JUnit and pioneer of eXtreme programming
joined Agitar, becoming the first Agitar
Software
Fellow.
July 23, 2004: Upon Agitar's invitation, Tao
Xie visited Agitar and gave a presentation on "Automated
Software Testing with Inferred Program Properties".
June
29, 2004: Agitar’s Agitator won 2004
Duke’s
Choice Award at JavaOne Developer Conference.
Dec 6, 2003: The tools in our ASE 03 paper
were highlighted in the testdriven.com
newsletter
#1.
Tao Xie and David Notkin. Tool-Assisted Unit Test Selection
Based on Operational Violations. In Proceedings of the
18th IEEE International Conference on Automated Software
Engineering (ASE
2003), Montreal, Canada, pp. 40-48, Oct. 2003. [PDF][BibTex][Slides]
(Nominated
for
the Best Paper Award) ©2003 IEEE.
An extended version to appear in a special issue of Automated
Software Engineering Journal. [PDF][BibTex]
Generating
tests
for an object-oriented program typically involves two tasks:
generating relevant receiver-object states and generating relevant
method arguments. Existing specification-based test generation tools
such as Korat and TestEra
rely on specifications such as class invariants to determine whether
their directly constructed receiver-object states are valid.
However, specifications often don't exist in practice. In our TACAS
05 paper, we have developed new techniques and a supporting tool,
called Symstra, for indirectly constructing valid and relevant
receiver-object states through method sequences, without requiring
any specifications. To address the issues of state explosion and
relevant-argument generation, we have defined symbolic states of
object-oriented programs (produced by symbolic execution) and novel
comparisons of symbolic states. Experimental results showed that the
tool generates tests that achieve higher structural coverage faster
than the existing tools. The Automated
Software Engineering group at NASA Ames Research Center
implemented a modified version of our Symstra techniques on top of Java PathFinder,
a model checker and test-input generator for Java, and found the
techniques to be effective in Java Pathfinder’s test-input
generation. The Foundations
of Software Engineering group at Microsoft Research has also
implemented the Symstra techniques in their test-generation
tools for C#.
Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A
Framework for Generating Object-Oriented Unit Tests using Symbolic
Execution.
In Proceedings
of
the 11th International Conference on Tools and Algorithms for the
Construction and Analysis of Systems
(TACAS
2005),
Edinburgh,
U.K., pp. 365-381, April 2005. [PDF][BibTex]
How
does our research work impact software industry?
How