Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.
Gem Versioning and Bundler: Doing it Right
May 30th, 2011
Recently, an upgrade to Rake (from version 0.8.7 to version 0.9.0) has re-raised the issue of dependencies and versioning in the Ruby community. I wanted to take the opportunity to reiterate some of the things I talked about back when working on Bundler 1.0. First, I’ll lay out some basic rules for the road, and then go into some detail about the rationale.
Basic Versioning Rules for Apps
- After bundling, always check your
Gemfile.lockinto version control. If you do this, you do not need to specify exact versions of gems in your Gemfile. Bundler will take care of ensuring that all systems use the same versions.
- After updating your
Gemfile, always run
bundle installfirst. This will conservatively update your
Gemfile.lock. This means that except for the gem that you changed in your
Gemfile, no other gem will change.
- If a conservative update is impossible, bundler will prompt you to run
bundle update [somegem]. This will update the gem and any necessary dependencies. It will not update unrelated gems.
- If you want to fully re-resolve all of your dependencies, run
bundle update. This will re-resolve all dependencies from scratch.
- When running an executable, ALWAYS use
bundle exec [command]. Quoting from the bundler documentation: In some cases, running executables without bundle exec may work, if the executable happens to be installed in your system and does not pull in any gems that conflict with your bundle. However, this is unreliable and is the source of considerable pain. Even if it looks like it works, it may not work in the future or on another machine. See below, “Executables” for more information and advanced usage.
- Remember that you can always go back to your old Gemfile.lock by using
git checkout Gemfile.lockor the equivalent command in your version control.
When you install a gem to the system, Rubygems creates wrappers for every executable that the gem makes available. When you run an executable from the command line without bundle exec, this wrapper invokes Rubygems, which then uses the normal Rubygems activation mechanism to invoke the gem’s executable. This has changed in the past several months, but Rubygems will invoke the latest version of the gem installed in your system, even if your
Gemfile.lock specifies a different version. In addition, it will activate the latest (compatible) installed version of dependencies of that gem, even if a different version is specified in your
This means that invoking executables as normal system executables bypasses bundler’s locked dependencies. In many cases, this will not pose a problem, because developers of your app tend to have the right version of the system-installed executable. For a long time, the Rake gem was a good example of this phenomenon, as most
Rake 0.8.7, and virtually all Ruby developers had
Rake 0.8.7 installed in their system.
As a result, users fell into the unfortunate belief that running system executables was compatible with bundler’s locked dependencies. To work around some of the remaining cases, people often advocate the use of rvm gemsets. Combined with manually setting up application-specific gemsets, this can make sure that the “system executables” as provided via the gemset remain compatible with the
Unfortunately, this kludge (and others) sufficiently reduced the pain that most people ignored the advice of the bundler documentation to always use
bundle exec when running executables tied to gems in the application’s
It’s worth noting that typing in
rake foo (or
anyexecutable foo) in the presence of a
Gemfile.lock, and expecting it to execute in the bundler sandbox doesn’t make any sense, since you’re not invoking Bundler. Bundler’s sandbox relies on its ability to be present at the very beginning of the Ruby process, and to therefore have the ability to ensure that the versions of all loaded libraries will reflect the ones listed in the
Gemfile.lock. By running a system executable, you are executing Ruby code before Bundler can modify the load path and replace the normal Rubygems loading mechanism, allowing arbitrary unmanaged gems to get loaded into memory. Once that happens, all bets are off.
bundle install –binstubs
In order to alleviate some of the noise of
bundle exec, Bundler 1.0 ships with a
--binstubs flag, which will create a
bin directory containing each of the executables that the application’s gems expose. Running
bin/cucumber, for instance, is the equivalent of running
bundle exec cucumber.
bin directory should contain portable versions of the executables, so it should be safe to add them to version control.
The only exception to the above rules is the
rails executable. As of Rails 3.0, the executable simply looks for a
script/rails file, and
execs that file. The generated
script/rails loads the local boot environment, which invokes bundler immediately:
# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application. APP_PATH = File.expand_path('../../config/application', __FILE__) require File.expand_path('../../config/boot', __FILE__) require 'rails/commands'
boot.rb is very simple:
require 'rubygems' # Set up gems listed in the Gemfile. ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__) require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])
In short, the
rails executable does a bunch of work to guarantee that your application’s bootstrapping logic runs before any dependencies are loaded, and it uses
Kernel#exec to purge the current process of any already-loaded gems. This is not something that general-purpose gems can or should do, and it’s an open question whether making the
rails executable work without
bundle exec sufficiently muddies the waters with regard to other gems as to render the benefit not worth it.