Comments on: New Rails Isolation Testing

By: Ben

Ben — Tue, 08 Sep 2009 05:51:20 +0000

@Ted, this isn’t about parallelization. It’s just that some tests modify global state which can be hard/impossible to undo without restarting the VM. This isolation allows those things to occur, be examined, and then be completely discarded at the OS level.

By: Ted

Ted — Thu, 30 Jul 2009 11:48:34 +0000

Wouldn’t this be susceptible to race conditions at the database level? If one test has an assert_difference and expects X records to be created, but simultaneously another test runs that creates records in the same table, you’d have a false positive. Or is each test inherently wrapped in a transaction (in which case parallelizing them would seem to provide a negligible benefit)? Thanks.

By: Chad Woolley

Chad Woolley — Wed, 01 Jul 2009 17:59:59 +0000

What about debugging? Will this mess with call stacks, breakpoints or stepping through test code or environment code during tests? That is, do most current debuggers (ruby-debug, rubymine, etc) handle forking OK?

By: wycats

wycats — Wed, 01 Jul 2009 17:41:09 +0000

@dan interestingly, the Merb tests were much more primitive than what we’ve done here. In this case, the parent literally doesn’t know that the tests are being run in forks; everything that happens in the child (in terms of collecting stats) is simply replayed in the parent. In Merb, we massively subverted the test runner ;)

By: wycats

wycats — Wed, 01 Jul 2009 17:39:13 +0000

@chad on the contrary! Because we’re isolating just the running of the test inside the fork, it means we can do far more invasive things inside the test itself, but not have to start up the full environment each time. Unfortunately, this approach doesn’t work on Windows, so Windows tests would indeed be slower, but that should impact your scenario, which is development-time test.

In effect, you get roughly the same speed as running the tests in a single process (with the extra overhead of forking, which is quite small), but all the benefits of running tests in isolation.

By: Dan Mayer

Dan Mayer — Wed, 01 Jul 2009 17:38:13 +0000

I know that Merb does forking for tests in one of their suites as well as that caused us to have to support test isolation for Devver. It wasn’t that hard of a feature but it sure through me for a loop trying to dig in and see why the tests were behaving so different locally or on devver. Kind of nice to keep things isolated though.

By: Avdi

Avdi — Wed, 01 Jul 2009 16:54:44 +0000

I’ve done something similar with forking to isolate tests. I’m curious how you handles the communication back to the parent process?

By: Chad Woolley

Chad Woolley — Wed, 01 Jul 2009 16:28:56 +0000

Does this slow things down any more? One of the biggest headaches we have is slow individual test startup time. In a Rails project that has many plugins/gems/dependencies and a complex environment, running a *single* test can take up to 15+ seconds – all environment initialization.

This completely kills the TDD red-green-refactor flow of a pair.

There are several ways people try to get around this. Some people use tools that keep the environment spun up (e.g. spec server or autotest) – this is a pain, because you always have situations where tests are failing falsely due to stale state, and you have to manually restart. Other people go to great lengths to lazily initialize and defer everything in their environment, so less stuff gets loaded during most tests. This also takes a lot of careful thought and work.

Does your approach of new process creation for every test exacerbate this problem? What other thoughts do you have on this topic?

Thanks,
– Chad

By: Yehuda Katz

Yehuda Katz — Wed, 01 Jul 2009 16:12:55 +0000

It just means that Carl and I have started actively using edge for projects, and don’t want to see “Starting Rails 2.3″ every time we boot our apps ;)

The best way to gauge how close we are to a public beta is to keep watching these posts (and posts we’re going to start doing on the RoR blog).

By: lolcatz

lolcatz — Wed, 01 Jul 2009 10:46:17 +0000

thanks for a nice write-up!

i also noticed that you bumped rails version to 3.0pre.
does it mean that something like beta will be out in the nearest months?