Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

On Rails Testing

One of the things that has both pleasantly surprised and frustrated me over the past six months is the state of Rails’ internal tests. While the tests can sometimes cover the most obscure, important bugs, they can sometimes be heavily mock-based or very coupled to internal implementation.

In large part, this is because of the requirement that patches come with tests, so the test suite has accumulated a lot of knowledge over time, but isn’t always the clearest when it comes time for a major refactor. Just to be clear, without the test suite, I don’t think the work we’ve been doing would have been possible, so I’m not complaining much.

However, I recently became aware that Sam Ruby, one of the authors of Agile Web Development on Rails, has released a test suite that tests each step in the depot application, as well as each of the advanced chapters.

Last week, Carl and I started digging into the Rails initializer, and the tests in the initializer (railties) are more mock-based and less reliable than the tests in ActionPack (which we’ve been working with so far). They’re pretty reasonable unit tests for individual components, but getting all of the tests to pass did not result in an (even close) bootable Rails app. Thanks to Sam’s tests, we were able to work through getting a booting Rails app by the end of last week.

Over the past week or two, Carl and I have been running the tests a few times a day, and while they definitely take a while to get through, they’ve added a new sense of confidence to the work we’re doing.

11 Responses to “On Rails Testing”

So, Yehuda, would you say that the heavy use of mocks are a detriment to working with the Rails test suite? Or is it a plus overall? (Maybe you know that I’m a bit of an anti-mocking zealot.)

I’ve realized recently that although mocks have their place (in unit tests), they need to be complemented by higher levels tests to gain real confidence (especially during refactoring). It also helps if the mocking framework allows a way to do checks on mocks (so they can’t have an entirely different interface than the object they are supposed to be replacing). Mocha sort of has this with it’s #responds_like method, but it’s not a full solution. I’m not sure what capabilities other mocking libraries offer.

@Francis, I’m just learning about mocks and stubs and, because I value your opinion, I’m wondering if you can provide a link or two about why you’re an anti-mocking zealot?

I just used Mocha to mock out a call to a 3rd party REST API call for my tests. Surely this is an ok use case for mocks, right? So where do they go wrong?

Mocking out a call to a REST API is a pretty good use-case for mocks. My general rule is “Don’t mock anything you own” and more strictly “Don’t mock anything happening inside your own process”. There are people who use mocks to good effect, but they’re an extremely sharp tool and my experience is that the people who use them well are very skilled programmers with an extremely honed testing philosophy.

I’m glad I’m not the only one struggling with mocks. I find them frustrating to use sometimes. Mocha, for example, doesn’t handle stubbing and mocking class methods very well, which causes me to mock / stub much deeper than I expect to.

Gabe: Well, if you’ve got a free hour, you could watch the “Testing Heresies” talk I gave at the last RubyConf. A lot of it is devoted to the subject of mocking:

In a nutshell, I’d say that I think it is system integrity that matters in your application, not how layers interact. I find that mocking is a way to lock down layer interaction, but that doesn’t do you any good if the overall system doesn’t work, and in fact is a detriment when you have to do serious refactoring and have to spend extra time dragging your mocks along with you.

Yehuda: “Don’t mock anything you don’t own” is a great rule of thumb, I’d say.

Arg, “Don’t mock anything you own”, even.

Be wary before knocking the mocking. When such things SAS, web services, legacy systems, payment gateways exist … it is possible to see a means of designing and testing interactions across such boundaries. And the reason why mocking and stubbing are considered useful solutions to such recurring problems.

Don’t knock the mock!

Isn’t rr the answer to this dilemma, the need for both a simple, atomic test of behavior (mock) and also an integrated black box test (no mock)? Doesn’t the proxying feature have some bearing on this?

Thanks for the replies, Francis and Yehuda. Francis, I’ll give your talk a view when I’ve got a spare hour. And yeah, Yehuda, I also like your rule of thumb; thanks for sharing that.

Yehuda, are you still working on a new Rails 3 book via Manning? The book they’re selling with that title is a Merb book and there haven’t been any pre-release updates in months. Just wondering if you’ve dropped the idea of completing the Rails 3 book.

Leave a Reply