Randomizing the order of the tests

One of JUnit best practices say that one should not assume the order in which tests in a test case are run. It is assumed that method will be executed in arbitrary order.
Therefore the having dependencies between the tests is discouraged.

JUnit uses reflection to find the test cases and it is not guaranteed that the order will be the same in different VM(s). But, you do not change your vm so often so the order of the test remains usually persistent across the lifetime of the project.

I have seen the tests that people do make some assumptions about the order and one test "prepares" the data or environment for the next. And the result is that test may fail if executed separately.

The dependencies might not be intentional at all, and just be the result of the way the test were ordered and no one realizes that they exist.

In the project I am right now we wanted to ensure that there are no dependencies between the tests and the solution was to intentionally make the order unpredictable. The problem was to find a way to randomize the test cases.

In android (2.2) source code when you look at
android.test.InstrumentationTestRunner.onCreate(Bundle)
you will see that
android.test.suitebuilder.TestSuiteBuilder.build()
is using something called a test grouping that is constructed with
android.test.suitebuilder.TestGrouping.SORT_BY_FULLY_QUALIFIED_NAME
and later adds a test(s) from a testCases list.

I am not sure in which order is this list populated but I am guessing the order can be predicted. As a result we will get a almost constant sequence of the test(s) unless some new test(s) are introduced.

So looking at the method android.test.InstrumentationTestRunner.onCreate(Bundle)
showed that the set of test is created inside the method using a local instance of TestSuiteBuilder and is assigned to the AndroidTestRunner instance. A class instance but the getter getAndroidTestRunner() everytime return new instance of the runner.

Recently I became more comfortable (& skilled) in java reflection and at first thought I expected to be able to get to that field with reflection. The problem was that onCreate just starts the test just before it completes. So you might not be able to randomize or you will randomize when the test are executed.

What is more with reflection you will have an ugly code, with lot of exception catching and without compiler checks so error prone and hard to maintain. Consequently, is very fragile and if the field name will change it will fail badly. Last but not least is a common belief the reflection is very slow and even though I had not measured it I share the same opinion.

The reflection might be used between onCreate and onStart but I figured out that we can save the runner used in onCreate and in onStart or start shuffle the test(s).

The result is below

 package net.retsat1.starlab.testrandomizer;  
 import java.util.Collections;  
 import java.util.List;  
 import junit.framework.TestCase;  
 import android.test.AndroidTestRunner;  
 import android.test.InstrumentationTestRunner;  
 /**  
  * A test runner that each time runs the set of test executes them in random order.  
  * This might be useful to eliminate the dependencies between the test(s)  
  *   
  * @author marek.defecinski  
  */  
 public class RandomizingTestRunner extends InstrumentationTestRunner {  
      private AndroidTestRunner androidTestRunner;  
      /**  
       * We hijack this method to make some modifications of the order of tests  
       */  
      @Override  
      public void onStart() {  
           List<TestCase> testCases = androidTestRunner.getTestCases();  
           Collections.shuffle(testCases);  
           super.start();  
      }  
      /**  
       * When the {@link AndroidTestRunner android runner} is created in {@link InstrumentationTestRunner#onCreate(android.os.Bundle)} we save it here.  
       * The same method at the end has a start method which we override to randomize the order of the test  
       */  
      @Override  
      protected AndroidTestRunner getAndroidTestRunner() {  
           androidTestRunner = super.getAndroidTestRunner();  
           return androidTestRunner;  
      }  
 }  

Hope someone will find it useful.