Multiple tests with minitest

I have an app with some specs written into minitest. As usual, I start them using rake.

Because some times I got a random results, my specs can pass one time, and fail an other time.

In this case, I can keep the sequence number and replay it later, after fixing. Because I have this kind of tests (with a random result), I generally run rake many time, just to be sure that the app is okay.

I would like to 开发者_StackOverflow社区know if there is a nice way to perform multiple rake tests (100 times for example), and stop them if there any failure or any error?

---

I think you should think again about your test, not about the test call. A test with a random result looks wrong for me.

What's the random factor in your test? Can you write a mock-element for the random factor and repeat the test with different values for the mock-element. So you get a "complete" test.

---

I created a dummy test with random result to simulate your situation:

#store it as file 'testcase.rb'
gem 'test-unit'
require 'test/unit'

class X < Test::Unit::TestCase
  def test_1
    num = rand(10) 
    assert_true( num < 5, "Value is #{num}")
  end
end 

The following task calls the test 10 times and stops after the first failure:

TEST_REPETION = 10
task :test do
  TEST_REPETION.times{
    stdout = `ruby testcase.rb`

    if stdout =~ /\d+\) Failure/
      puts "Failure occured"
      puts stdout
      exit
    else
      puts 'Tests ok'
    end
  }
end

For real usage I would adapt some parts:

• Instead puts 'Tests ok' define a counter to see how often the test was succussfull
• Instead puts stdoutyou may store the result in a result file?