Practical Tools for Testing Concurrent Programs
Poster
In our experience with developing production programs in Java,
unit testing has proven effective in assuring the reliability of code with a single thread of control. Unfortunately, unit testing has proved much less effective in assuring the reliability of code with multiple threads of control, often simply because the JUnit testing framework silently ignores failures in auxiliary threads. Java libraries and user programs frequently make assumptions about the threading context in which they execute, but these assumptions are rarely enforced by the actual code and typically only appear in program documentation. Since thread scheduling is non-deterministic, a unit test can succeed on one run and fail on the next, or repeatedly succeed on one platform and occasionally fail on another.
To improve test-driven development for concurrent programs, we are developing
(i) an extension of the JUnit framework, actively supporting the developer by treating tests that could silently ignore failures in auxiliary threads as test errors;
(ii) a lightweight annotation language, which can be used to specify and check the threading properties of both existing and new code; and
(iii) a testing framework that can execute unit tests according to a specified set of recorded or generated schedules, elevating the unit testing of concurrent programs to a rigorous, deterministic process.
- 28 Views
-
Like (1)
This project is kindly supported by
ATP
TEXAS
Practical Tools for Testing Concurrent Programs
Concurrency-Aware Unit Testing Framework
Current Frameworks JUnit, TestNG Are Broken!
► JUnit, TestNG, etc. were not designed with concurrency in mind → The criteria for success and failure are incorrect for concurrent programs → The design promotes writing tests that succeed by default even when they should fail ► Writing good concurrent unit tests is as hard as writing good concurrent programs ► Problem 1: Exceptions are not automatically detected and considered a test failure. → The program fails in an auxiliary thread, but the unit test nonetheless succeeds, even long after the other thread has failed.
Primary Thread spawns Lengthy calculation.... Auxiliary Thread Failure! Success!
Corky Cartwright, Mathias Ricken {cork|mgricken}@rice.edu
Unpredictable Behavior of Concurrent Unit Tests
Unit Tests Are Effective...
► Help find and prevent bugs in sequential code ► Tests must pass before code changes are committed → Fewer bugs enter the repository ► Tests exhibiting fixed bugs are added to test suite → Prevent old bugs from reappearing ► Current tools: JUnit, TestNG, Ant
Inserting Delays and Scheduled Replay
► Custom class loader rewrites class files as they are loaded ► Modified class files will run on any Java VM → Portable and open-source ► Bytecode can be dynamically compiled by an embedded JIT compiler → Faster than interpretation ► When an atomic block ends, a delay may be inserted, forcing the program into a different schedule
Thread T1 Atomic Block Thread T2 Atomic Block Atomic Block Atomic Block Atomic Block
► Same technique can be used for record and reply by schedule ► At the end of an atomic block, all threads are put on hald, and only the scheduled one resumes ► Difficult to implemenent a Java scheduler in the same Java VM → Scheduler itself may create and destroy values, influencing garbage collection and schedule
Thread T1 Thread T2 Thread T3 Currently executing Atomic Block Atomic Block Atomic Block Atomic Block Threads T1, T2, T3 could resume Schedule dictates that T3 resumes
► Problem 2: The test does not have to wait for all auxiliary threads to terminate. → The program fails in an auxiliary thread, but the primary thread completes the unit test so quickly, that failure goes unnoticed.
Primary Thread spawns Success! Auxiliary Thread Failure!
... But Not for Concurrent Programs
► Thread switching is non-deterministic and machine-specific ► Success of a unit test does not imply correct behavior under all possible schedules ► Most programs are concurrent → GUI: separate thread for display → Multi-core: programs must be concurrent to benefit ► Current tools not effective or easy to use on large projects
Problem: Ignored Exception!
Problem: Late Exception!
► Solution: Use a default exception handler
► Solution: Wait for all spawned auxiliary threads
Figure: Random delays inserted between atomic blocks
Figure: Transfer of control at the end of an atomic block
Lightweight Checking of Concurrency Invariants Using Java Annotations
Java Annotations to Specify Invariants DrJava Case Study
► DrJava recently exhibited a number of concurrency-related problems → We believe we have diligently fixed most problems by manually ► To test the utility and to make future development of DrJava easier, we annotated a current version and several old versions of DrJava ► The most recent version had a lower percentage of thread checks failing → Had the annotations not been done after the fact, they probably would have identified even more problems → Still identified several potential bugs in the current version
45000 40000 35000 30000 25000 10.00 20000 8.00
Unit Tests
Could Not Run Unit Test Failures Unit Tests Passed 90 36 610 0 0 881
Practical Tools for Concurrency
(1) A concurrency-aware extension of JUnit (2) Lightweight dynamic and static checking of concurrency invariants using Java annotations (3) Execution with short delays inserted at critical places to test different execution schedules (4) Recording of program execution and replaying according to a predetermined schedule
► Declare which classes and methods should not be or may only be executed by certain threads ► Simple to introduce into existing large projects → Just annotate parts of the projects, no need for a complete rewrite ► Dynamic checks to maintain invariants are automatically inserted → Some checks are also performed statically
20.00 18.00 16.00 14.00 12.00
1000 900 800 700 600 500 400 300 200 100 0
Mar’04 Sep’06
A Way to Specify API Threading Discipline
► Maintaining a threading discipline is problematic for current APIs → APIs are often extensible, so API designers do not have full control → Invariants are written as comments or white papers at best ► Annotations record violations: They are not just comments anymore! ► Utility can apply annotations to existing libraries using external XML specification files → No need to access source code or even recompile → XML files written for a library can be reused and openly shared → We have started writing a specification file for the Java Swing API
Augmenting Type Systems Using Annotations
► By introducing annotations describing permitted threading behavior, we have in effect augmented Java’s type system. → Subtyping problems: C <: D according to Java, but not by our checking ► Formulation of the utility’s subtyping relation based on Featherweight Java ► Annotations can be used to augment Java’s type system to get additional guarantees; material for future research.
15000 10000 5000 0
6.00 4.00 2.00 0.00
Mar’04 Sep’06 965 4161 18.83 3796 30616 11.03
Violations Passed Checks % Violations
Thread Checks and Failures
Readers
Add Comment