Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Ruby Require Order Problems

Bundler has inadvertantly exposed a number of require order issues in existing gems. I figured I’d take the opportunity to talk about them. There are basically two kinds of gem ordering issues:

Missing Requires

Imagine a gem that uses nokogiri, but never requires it. Instead, it assumes that something that is required before it will do the requiring. This happens relatively often with Rails plugins (which were previously able to assume that they were loaded at a particular time and place).

A well-known example of this problem is gems that didn’t require “yaml” because Rubygems required it. When Rubygems removed its require in 1.3.6, a number of gems broke. In general, bundler assumes that gems require what they need. If gems do so, this class of ordering problems is eliminated.

Constant Definition as API

This is where a gem checked for defined?(SomeConstant) to decide whether to activate some functionality.

This is a bit tricky. Essentially, the gem is saying that the API for using it is “require some other gem before me”. Personally, I consider that to be a problematic API, because it’s very implicit, and can be hard to track down exactly who did what require.

A better solution is to provide an explicit hook for people to activate the optional functionality. For instance, Haml
provides: Haml.init_rails(binding) which you can run after activating Haml.

This is slightly more manual than some would like, but the API of “make sure you require the optional dependencies before me” is also manual, and more error-prone.

Even if Bundler “respected require order”, which we plan to do (in some way) in 0.10, it’s still up to the user of the gem to ensure that they listed the optional gem above the required gem. This is not ideal.

A workaround that works great in Bundler 0.9 is to simply require the order-dependent gems above Bundler.require. We do this in Rails, so that gems can test for the existence of the Rails constant to decide whether to add optional Rails dependencies.

require "rails/all" 
Bundler.require(:default, Rails.env)

In the case of shoulda and mocha, a better solution could be:

# Gemfile 
group :test do 
  gem "shoulda" 
  gem "mocha" 
  # possible other gems where order doesn't matter 
end
 
# application.rb 
Bundler.require(:default) 
Bundler.require(Rails.env) unless Rails.env.test? 
 
# test_helper.rb 
# Since the order matters, require these gems manually, in the right order 
require "shoulda" 
Bundler.require(:test)

In my opinion, if you have gems that specifically depend on other gems, it is appropriate to manually require them first before automatically requiring things using Bundler.require. You should treat Bundler.require as a shortcut
for listing out a stack of requires.

Possible solutions?

One long-term solution, if we get gem metadata in Rubygems 1.4 (or optional dependencies down the line), would be to specify that a gem has optional dependencies on another gem and specify a file to run both gems are
available.

For instance, the Haml gem could say:

# gemspec 
s.integrates_with "rails", "~> 3.0.0.beta2", "haml/rails" 
 
# haml/rails.rb 
require "haml" 
require "rails" 
Haml.init_rails(binding)

Bundler would handle this in Bundler.require (or possibly Bundler.setup).

If this feature existed in Rubygems, it would work via gem activation. If the Haml gem was activated after the Rails gem, it would require “haml/rails” immediately.

If the Haml gem was activated otherwise, it wouldn’t do anything until the Rails gem was activated. When the Rails gem was activated, it would require the file.

15 Responses to “Ruby Require Order Problems”

Checking constant definition is a way of stating an implicit, optional dependency. It’s not an error.

Using normal Ruby requires instead of Bundler.require obviates this class of ordering problems as well.

In your test helper:


require 'shoulda'
require 'mocha'

In an app initializer:


require 'haml/rails'

Relying on Bundler.require for convenience will turn into relying on it to be smart and handle these easily programmer-solvable issues with complex engineering. That deserves a dash of vinegar, not sugar.

Jeremy Kemper: Well put.

“Just use Bundler” is not a satisfactory answer to these types of questions:

https://rails.lighthouseapp.com/projects/8994/tickets/3685-actionpack-235-gem-declares-incompatibility-with-rack-110#ticket-3685-9

For once, I disagree ;)

I don’t see anything wrong with load order being important for gems. CarrierWave has adapters for half a dozen ORMs, and they’re all required implicitely (if the relevant constant is defined). I’ve never had a single person complain about this or file even a single issue about it.

If I can’t rely on load order, there’ll have to be an extra step of requiring the right orm extension. That destroys a lot of the “works-out-of-the-box” feeling, and gives me a significant disadvantage compared to other solution which haven’t bothered to implement ORM agnosticism. Essentially you’re discouraging one good practice (agnosticism) in favour of encouraging another (no load order). The problem is that having significant load order isn’t really broken in the real world. Lots of gems have it, lots of people use it, no one ever complains. The only software that *does* care, is Bundler.

You’re saying bundler is going to respect gem ordering, I really hope you do implement that. Just from a purely philosophical perspective, because all code is executable in Ruby, ignoring load order seems like a recipe for pain and disaster. If Bundler were a tool for, say, Python, where you can be explicit about what to require from where, then yes, load order wouldn’t matter. But that’s not how Ruby works. Ignoring load order means that we create the potential for subtle bugs. If two gems add the same core method for example, which one wins? If load order is respected, the answer is simple: the last one wins. If it isn’t, the answer is: who knows?

So if you’re going to preserve load order anyway, and it’s an established practice that’s worked well for many people in creating a seamless experience for agnostic gems, then why is using constants as an API a problem?

@jeremy @jonas I don’t actually disagree with either of you. However, the implicit API of “make sure you require foo before you require me” is hard to document well and people *do* make mistakes.

@jeremy Our recommended solution to people with problems using Bundler.require *is* to do explicit requires. They just didn’t expect the implicit constant dependency and weren’t sure what exactly was happening.

@jonas Anecdotally, dealing with people who file bundler issues, it’s clear to me that many Ruby programmers do *not* realize that they have to require gems like your carrierwave dependencies before carrierwave. They may have accidentally done the right thing in the past, or your error messages might be very helpful.

In any event, my proposal is not specifically related to bundler; it’s providing an automated way to provide the sort of glue code you’re providing via implicit defined? checked in an explicit way in the .gemspec. It doesn’t reduce the out of the box feeling, and in fact could reduce these kinds of errors.

Oh, and @jeremy, I’m fully ok with requiring haml/rails in an initializer. That’s what I meant when I said “This is slightly more manual than some would like, but the API of “make sure you require the optional dependencies before me” is also manual, and more error-prone”. The problem is that the Haml folks are not ok with this approach.

Thankfully, Rails plugins can (even in 3.0) rely on the Rails constant being present in Rails apps. This is not true about other combinations.

Being able to declare optional dependencies would be nice. When I have questions about a gem that I can’t answer by looking at the documentation, one of the first places I look is the gemspec. This is often not helpful, so I then dig through the code.

I am eying the s.integrates_with “rails”, “~> 3.0.0.beta2″, “haml/rails” code with excitement.

The problem I have with the manual solution is that it causes widespread problems.

This is not a simple problem to understand. And the pain hits when you have to figure out what order library X expects itself to be loaded in relation to your other gems. Did they check the constant? Did they put it in an init.rb file? Is it in the lib/.rb file or nestled somewhere in the hierarchy?

Yeah, that’s not hard. But it’s 10-15 minutes where I have to investigate what your particular tastes are. In the case of haml there are >10K downloads of the gem for the 3.0 BETA release alone.

Some people know what to do, some people don’t, and others will get it right accidentally. But it’s a dependency issue. A very decent solution has been proposed. We need to be poking at it’s deficiencies, not brushing this off as “not a problem”.

This is a dependency resolution problem which really should be taken care of in the gemspec.

I agree that implicit optional dependencies are confusing, error prone, and tricky to document. It seems to me that the best way to handle that is via a special integration oriented file, e.g: “require ‘haml/rails’”. Why not stop there?

Once you have that nice (easily documented) integration oriented library to load, it seems to me that going the extra step (with implicit rubygems “integrates_with”) behavior is a step backwards. Then we would have implicit dependencies again, code that is run in certain circumstances but not in others. Yes, the load order is fixed, but otherwise it just makes our lives a little bit difficult.

To use a rails gem plugin via bundler, all you’d need is:

gem “haml”, “~>12.34.56″, :require => “haml/rails”
gem “rspec-rails”, “~>12.34.56″, :require => “spec/rails”

and so on.

Also, in reply to Jonas (et al), I *have* run into issues with implicit load orders. Many times over the last three years. Most recently with will_paginate in a bundler controlled project, but the problem absolutely pre-dates bundler. And in almost every circumstance, the documentation was unclear as to what I did wrong, and I had to go searching through the source code for the gem in question until I found the “if defined?(ActionController)” that had caused me the pain. I would far rather just “require ‘will_paginate/rails’”. I don’t believe I ever filed a bug report or submitted a patch for a project bit me with implicit integration, but Yehuda’s post has emboldened me, and I will from now on.

@Nicholas

I’m beginning to see it.

require “/rails”

I like this way, possibly a little bit more than integrates_with. (though integrates with would be interesting for the graph data it would generate)

I am also liking your instinct to report/patch the way there.

@nicholas @collin This would mean that as part of the process of installing Haml for Rails would be “make sure you require ‘haml/rails’ in your application.rb”. People seem generally unhappy with adding additional steps to installing plugins.

@wycats I understood that the concept was

# Gemfile
gem “haml”, :require => “haml/rails”

# haml/rails.rb
require “rails”
require “haml”
require “haml/railtie” # or however the integration works

# haml/sinatra.rb
require “sinatra”
require “haml”
require “haml/sinatra_plugin” # or whatever

I’m okay with the “extra” step if I can keep it stuffed in one place(Gemfile).

For my knowledge I would like to know if shoulda depends of mocha or it’s the other way around ?

Leave a Reply

Archives

Categories

Meta