Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Gem Versioning and Bundler: Doing it Right

Recently, an upgrade to Rake (from version 0.8.7 to version 0.9.0) has re-raised the issue of dependencies and versioning in the Ruby community. I wanted to take the opportunity to reiterate some of the things I talked about back when working on Bundler 1.0. First, I’ll lay out some basic rules for the road, and then go into some detail about the rationale.

Basic Versioning Rules for Apps

  1. After bundling, always check your Gemfile.lock into version control. If you do this, you do not need to specify exact versions of gems in your Gemfile. Bundler will take care of ensuring that all systems use the same versions.
  2. After updating your Gemfile, always run bundle install first. This will conservatively update your Gemfile.lock. This means that except for the gem that you changed in your Gemfile, no other gem will change.
  3. If a conservative update is impossible, bundler will prompt you to run bundle update [somegem]. This will update the gem and any necessary dependencies. It will not update unrelated gems.
  4. If you want to fully re-resolve all of your dependencies, run bundle update. This will re-resolve all dependencies from scratch.
  5. When running an executable, ALWAYS use bundle exec [command]. Quoting from the bundler documentation: In some cases, running executables without bundle exec may work, if the executable happens to be installed in your system and does not pull in any gems that conflict with your bundle. However, this is unreliable and is the source of considerable pain. Even if it looks like it works, it may not work in the future or on another machine. See below, “Executables” for more information and advanced usage.
  6. Remember that you can always go back to your old Gemfile.lock by using git checkout Gemfile.lock or the equivalent command in your version control.

Executables

When you install a gem to the system, Rubygems creates wrappers for every executable that the gem makes available. When you run an executable from the command line without bundle exec, this wrapper invokes Rubygems, which then uses the normal Rubygems activation mechanism to invoke the gem’s executable. This has changed in the past several months, but Rubygems will invoke the latest version of the gem installed in your system, even if your Gemfile.lock specifies a different version. In addition, it will activate the latest (compatible) installed version of dependencies of that gem, even if a different version is specified in your Gemfile.lock.

This means that invoking executables as normal system executables bypasses bundler’s locked dependencies. In many cases, this will not pose a problem, because developers of your app tend to have the right version of the system-installed executable. For a long time, the Rake gem was a good example of this phenomenon, as most Gemfile.locks declared Rake 0.8.7, and virtually all Ruby developers had Rake 0.8.7 installed in their system.

As a result, users fell into the unfortunate belief that running system executables was compatible with bundler’s locked dependencies. To work around some of the remaining cases, people often advocate the use of rvm gemsets. Combined with manually setting up application-specific gemsets, this can make sure that the “system executables” as provided via the gemset remain compatible with the Gemfile.lock.

Unfortunately, this kludge (and others) sufficiently reduced the pain that most people ignored the advice of the bundler documentation to always use bundle exec when running executables tied to gems in the application’s Gemfile.lock.

It’s worth noting that typing in rake foo (or anyexecutable foo) in the presence of a Gemfile.lock, and expecting it to execute in the bundler sandbox doesn’t make any sense, since you’re not invoking Bundler. Bundler’s sandbox relies on its ability to be present at the very beginning of the Ruby process, and to therefore have the ability to ensure that the versions of all loaded libraries will reflect the ones listed in the Gemfile.lock. By running a system executable, you are executing Ruby code before Bundler can modify the load path and replace the normal Rubygems loading mechanism, allowing arbitrary unmanaged gems to get loaded into memory. Once that happens, all bets are off.

bundle install –binstubs

In order to alleviate some of the noise of bundle exec, Bundler 1.0 ships with a --binstubs flag, which will create a bin directory containing each of the executables that the application’s gems expose. Running bin/cucumber, for instance, is the equivalent of running bundle exec cucumber.

The generated bin directory should contain portable versions of the executables, so it should be safe to add them to version control.

The rails Command

The only exception to the above rules is the rails executable. As of Rails 3.0, the executable simply looks for a script/rails file, and execs that file. The generated script/rails loads the local boot environment, which invokes bundler immediately:

# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
 
APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rails/commands'

The generated boot.rb is very simple:

require 'rubygems'
 
# Set up gems listed in the Gemfile.
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
 
require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])

In short, the rails executable does a bunch of work to guarantee that your application’s bootstrapping logic runs before any dependencies are loaded, and it uses Kernel#exec to purge the current process of any already-loaded gems. This is not something that general-purpose gems can or should do, and it’s an open question whether making the rails executable work without bundle exec sufficiently muddies the waters with regard to other gems as to render the benefit not worth it.

37 Responses to “Gem Versioning and Bundler: Doing it Right”

Thanks Yehuda,

Nicely written, I was unaware of “bundle install –binstubs” this will come in really handy.

I think that a closer integration of bundler and RVM would be a really neat solution to issues like this. I’ll be chatin’ on #rvm.

Ooh.. I like the –binstubs option. Never noticed it before.

Wouldn’t the most elegant solution be to modify RubyGems so it becomes “bundler-aware”? Perhaps the ultimate solution would be to have an integrated bundler/rubygems that replaces both.

In our case the rake-0.9.0 release also exposed another issue: gems have back doors they can use to install other gems. In our case, a gem we depended on was installing rake with no version constraint during the native extensions installation process. Bundler reported that rake-0.8.7 was being installed, said nothing about 0.9 (because it didn’t know about it), and then `gem list rake` showed us both 0.8.7 and 0.9.0. Once we narrowed it down we reported it and the gem maintainer released new versions right away, but this might not be the only gem that does this. I’ve reported this to the rubygems tracker (http://rubyforge.org/tracker/index.php?func=detail&aid=29236&group_id=126&atid=575) but, until it is addressed, it’s something for everybody to keep their eyes on.

Thanks for the excellent post and reiterating the importance of using `bundle exec`! :)

@Jan, @Thomas, both those ideas sound promising. Can’t speak for either project but I personally would love to see patches and forks exploring those ideas.

Aside from ‘bundle install –binstubs’, could ‘bundle exec -list’, which would return a list of the executables that the application’s gems expose, help clarify the issue?

so with binstubs in effect, could rvm set the PATH to use those automatically?

DGM, even if rvm couldn’t, no reason you can’t yourself. Interesting approach.

DGM – just add “export PATH=./bin:$PATH” to your .rvmrc. Works great!

So why isn’t `gem “rake”` included in default Rails Gemfile? You do need to include it to lock rake to 0.8.7 or whatever?

I always use ‘bundle exec bash’ to enter a bundled shell when working in a project folder. Works great :)

I love the idea of a bundler-aware RubyGems.

Thanks for figuring it out, the final fix is to use `.rvmrc`:

rvm use 1.9.2@test –create;
export PATH=./bin:$PATH;
alias bundle=”bundle –binstubs”;

Today I will update RVM documentation to give that hints.

Generated .rvmrc support for adding ./bin to path has been released in latest RVM (rvm get head). For example in your project you can run say,

rvm –rvmrc –create use 1.9.2@test

Then look in the generated .rvmrc file and uncomment the bundler section if you use bundler.

Yehuda, I suggest you add a /etc/bundlerc and/or ~/.bundlerc where you can set binstubs as default as a feature to bundler.

Making Rubygems bundler-aware would be bad for both Rubygems and Bundler because it would constrain both moving forward. I think a better solution is for Rubygems to expose the extension points that Bundler needs, and for Bundler to use them.

Good ! Thanks for your post!

that’s what I use:

alias b=’bundle exec’

and then just:

b what_ever_you_need_here

simple and fast :)

Great post and interesting follow-up discussions. I’m loving it!

I wrote a script today that you can stick in your ~/bin and it will find the right rake for you. It’s safer than “export PATH=./bin:$PATH” that was recommended earlier, because doesn’t work on dirs you haven’t specified, and it’s more reliable too, because it works from your project’s subdirs, whereas adding ./bin to the path will only work from the project root dir.

https://github.com/JoshCheek/burake

Brian, nice tip! Great for inspecting production environment without using `bundle exec` every time! \m/

This is a fantastic post. I just realized I don’t have to manually specify versions for every single gem in my Gemfile. Goddamn, that will save me a ton of time.

I always wondered why bundler couldn’t do something like this:

exec ‘bundle’, ‘exec’, $0, *ARGV unless ENV['BUNDLE_GEMFILE']

Would save users needing to know to wrap certain ruby executables in “bundle exec”.

rvm after_cd hook http://blockgiven.tumblr.com/post/6111172456/bundler-binstubs

I have found that creating an alias for rake solves my 80/20 rule pain. alias ber=’bundle exec rake’

So we should have :

“rake foo” should be replaced as “bundle exec rake foo”

Use https://github.com/gma/bundler-exec it will automatically use bundle exec whenever there is a Gemfile in the directory -> no more worries / no strange binstubs

for a gem app.

a solution is not use wrappers:

gem install –no-wrappers foo
gem update –no-wrappers foo

or you can write ‘gem: –no-wrappers’ into ~/.gemrc

and in bin/foo.rb

file = File.symlink?(__FILE__) ? File.readlink(__FILE__) : __FILE__

ENV["BUNDLE_GEMFILE"] = File.join(File.dirname(File.absolute_path(file)), ‘../Gemfile’)

.rvmrc support for adding ./bin to path is very awesome!

I’m new to Ruby on Rails and was a bit perplexed when I got nginx and unicorn working locally on my development machine, but couldn’t get it working on my Linode. Everything was set up the same, same Linux distro, used RVM to install the same ruby version, used nginx stable ppa. The only difference was I have only just got the Linode, but I set up my development machine 2 or 3 months ago.

So on both servers I blindly ran gem update, bundle update, bundle install, anything that seemed to do stuff. Didn’t have a clue what the implications were, but ran them to see if it would sort it. Net result, I broke my development server, it was now getting the same error. I didn’t understand what the error was, even after running “bundle exec” I was nowhere further forward.

Luckily I stumbled across this article, so I:
1. Uncommented unicorn from /Gemfile
2. bundle install
3. bundle exec unicorn -c /var/www/lukearmstrong.co.uk/config/unicorn.rb

And it worked.

If I may, I’ll leave the error message here so that it may be picked up by search engines:
/usr/local/rvm/gems/ruby-1.9.2-p180/gems/bundler-1.0.15/lib/bundler/runtime.rb:31:in `block in setup’: You have already activated rack 1.3.0, but your Gemfile requires rack 1.2.3. Consider using bundle exec. (Gem::LoadError)

Great post!
But what about autotest? Specifically autotest-growl.
We don’t include it in our Gemfile because it’s Mac only.

Here’s my solution. A bit less complicated than Josh Cheek’s, and a bit more robust than Ivan Storck’s:

Add this to your .bashrc:

rake() { if [ -e ./Gemfile.lock ]; then bundle exec rake “$@”; else /usr/bin/env rake “$@”; fi; }

I was running into strange problems with rake db:migrate (nesting too deep errors) and while I found the answer elsewhere this really explains the issue.

Here’s something I tinkered with a few days ago to try to elegantly solve this problem:

https://github.com/jjb/bshell

(which is just a fancier way of doing: bundle exec zsh)

I haven’t had time to look at the bundler source and consider my solution’s thoroughness — feedback welcome.

John

Great info!

I disagree about always using bundle exec though, except where we shouldn’t. Instead we should make our scripts bundler-aware just like rails is, and scripts that aren’t aware, we can just use a wrapper.

Very good article.
but in my case, there is a lot of ruby crontabs that execute ruby exclusively bundle context..
should make some better solution. this is too error prone and complicate.

Leave a Reply

Archives

Categories

Meta