Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Bundler’ Category

Gem Versioning and Bundler: Doing it Right

Recently, an upgrade to Rake (from version 0.8.7 to version 0.9.0) has re-raised the issue of dependencies and versioning in the Ruby community. I wanted to take the opportunity to reiterate some of the things I talked about back when working on Bundler 1.0. First, I’ll lay out some basic rules for the road, and then go into some detail about the rationale.

Basic Versioning Rules for Apps

  1. After bundling, always check your Gemfile.lock into version control. If you do this, you do not need to specify exact versions of gems in your Gemfile. Bundler will take care of ensuring that all systems use the same versions.
  2. After updating your Gemfile, always run bundle install first. This will conservatively update your Gemfile.lock. This means that except for the gem that you changed in your Gemfile, no other gem will change.
  3. If a conservative update is impossible, bundler will prompt you to run bundle update [somegem]. This will update the gem and any necessary dependencies. It will not update unrelated gems.
  4. If you want to fully re-resolve all of your dependencies, run bundle update. This will re-resolve all dependencies from scratch.
  5. When running an executable, ALWAYS use bundle exec [command]. Quoting from the bundler documentation: In some cases, running executables without bundle exec may work, if the executable happens to be installed in your system and does not pull in any gems that conflict with your bundle. However, this is unreliable and is the source of considerable pain. Even if it looks like it works, it may not work in the future or on another machine. See below, “Executables” for more information and advanced usage.
  6. Remember that you can always go back to your old Gemfile.lock by using git checkout Gemfile.lock or the equivalent command in your version control.

Executables

When you install a gem to the system, Rubygems creates wrappers for every executable that the gem makes available. When you run an executable from the command line without bundle exec, this wrapper invokes Rubygems, which then uses the normal Rubygems activation mechanism to invoke the gem’s executable. This has changed in the past several months, but Rubygems will invoke the latest version of the gem installed in your system, even if your Gemfile.lock specifies a different version. In addition, it will activate the latest (compatible) installed version of dependencies of that gem, even if a different version is specified in your Gemfile.lock.

This means that invoking executables as normal system executables bypasses bundler’s locked dependencies. In many cases, this will not pose a problem, because developers of your app tend to have the right version of the system-installed executable. For a long time, the Rake gem was a good example of this phenomenon, as most Gemfile.locks declared Rake 0.8.7, and virtually all Ruby developers had Rake 0.8.7 installed in their system.

As a result, users fell into the unfortunate belief that running system executables was compatible with bundler’s locked dependencies. To work around some of the remaining cases, people often advocate the use of rvm gemsets. Combined with manually setting up application-specific gemsets, this can make sure that the “system executables” as provided via the gemset remain compatible with the Gemfile.lock.

Unfortunately, this kludge (and others) sufficiently reduced the pain that most people ignored the advice of the bundler documentation to always use bundle exec when running executables tied to gems in the application’s Gemfile.lock.

It’s worth noting that typing in rake foo (or anyexecutable foo) in the presence of a Gemfile.lock, and expecting it to execute in the bundler sandbox doesn’t make any sense, since you’re not invoking Bundler. Bundler’s sandbox relies on its ability to be present at the very beginning of the Ruby process, and to therefore have the ability to ensure that the versions of all loaded libraries will reflect the ones listed in the Gemfile.lock. By running a system executable, you are executing Ruby code before Bundler can modify the load path and replace the normal Rubygems loading mechanism, allowing arbitrary unmanaged gems to get loaded into memory. Once that happens, all bets are off.

bundle install –binstubs

In order to alleviate some of the noise of bundle exec, Bundler 1.0 ships with a --binstubs flag, which will create a bin directory containing each of the executables that the application’s gems expose. Running bin/cucumber, for instance, is the equivalent of running bundle exec cucumber.

The generated bin directory should contain portable versions of the executables, so it should be safe to add them to version control.

The rails Command

The only exception to the above rules is the rails executable. As of Rails 3.0, the executable simply looks for a script/rails file, and execs that file. The generated script/rails loads the local boot environment, which invokes bundler immediately:

# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
 
APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rails/commands'

The generated boot.rb is very simple:

require 'rubygems'
 
# Set up gems listed in the Gemfile.
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
 
require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])

In short, the rails executable does a bunch of work to guarantee that your application’s bootstrapping logic runs before any dependencies are loaded, and it uses Kernel#exec to purge the current process of any already-loaded gems. This is not something that general-purpose gems can or should do, and it’s an open question whether making the rails executable work without bundle exec sufficiently muddies the waters with regard to other gems as to render the benefit not worth it.

Bundler: As Simple as What You Did Before

One thing we hear a lot by people who start to use bundler is that the workflow is more complicated than it used to be when they first start. Here’s one (anonymized) example: “Trying out Bundler to package my gems. Still of the opinion its over-complicating a relatively simple concept, I just want to install gems.”

Bundler has a lot of advanced features, and it’s definitely possible to model fairly complex workflows. However, we designed the simple case to be extremely simple, and to usually be even less work than what you did before. The problem often comes when trying to handle a slightly off-the-path problem, and using a much more complex solution than you need to. This can make *everything* much more complicated than it needs to be.

In this post, I’ll walk through the bundler happy path, and show some design decisions we made to keep things moving as smoothly as before, but with far fewer snags and problems than the approach you were using before. I should be clear that there are probably bugs in some cases when using a number of advanced features together, and we should fix those bugs. However, they don’t reflect core design decisions of bundler.

In the Beginning

When you create a Rails application for the first time, the Rails installer creates a Gemfile for you. You’ll note that you don’t actually have to run bundle install before starting the Rails server.

$ gem install rails
Successfully installed activesupport-3.0.0
Successfully installed builder-2.1.2
Successfully installed i18n-0.4.1
Successfully installed activemodel-3.0.0
Successfully installed rack-1.2.1
Successfully installed rack-test-0.5.6
Successfully installed rack-mount-0.6.13
Successfully installed tzinfo-0.3.23
Successfully installed abstract-1.0.0
Successfully installed erubis-2.6.6
Successfully installed actionpack-3.0.0
Successfully installed arel-1.0.1
Successfully installed activerecord-3.0.0
Successfully installed activeresource-3.0.0
Successfully installed mime-types-1.16
Successfully installed polyglot-0.3.1
Successfully installed treetop-1.4.8
Successfully installed mail-2.2.6.1
Successfully installed actionmailer-3.0.0
Successfully installed thor-0.14.2
Successfully installed railties-3.0.0
Successfully installed rails-3.0.0
22 gems installed
$ rails s
=> Booting WEBrick
=> Rails 3.0.0 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2010-09-30 12:25:51] INFO  WEBrick 1.3.1
[2010-09-30 12:25:51] INFO  ruby 1.9.2 (2010-08-18) [x86_64-darwin10.4.0]
[2010-09-30 12:25:51] INFO  WEBrick::HTTPServer#start: pid=26380 port=3000

If you take a look at your directory, you’ll see that Bundler noticed that you didn’t yet have a Gemfile.lock, saw that you had all the gems that you needed already in your system, and created a Gemfile.lock for you. If you didn’t have a necessary gem, the Rails server would error out. For instance, if you were missing Erubis, you’d get Could not find RubyGem erubis (~> 2.6.6). In this case, you’d run bundle install to install the gems.

As you can see, while bundler is providing some added features for you. If you take your application to a new development or production machine with the Gemfile.lock, you will always use exactly the same gems. Additionally, bundler will resolve all your dependencies at once, completely eliminating the problem exemplified by this thread on the Rails core mailing list. But you didn’t have to run bundle install or any bundle command unless you were missing some needed gems. This makes the core experience of bundler the same as “I just want to install gems”, while adding some new benefits.

Adding Gems

When you add a gem that is already installed on your system, you just add it to the Gemfile and start the server. Bundler will automatically pick it up and add it to the Gemfile.lock. Here again, you don’t need to run bundle install. To install the gem you added, you can run bundle install or you can even just use gem install to install the missing gems.

Once again, while bundler is handling a lot for you behind the scenes (ensuring compatibility of gems, tracking dependencies across machines), if you have all the gems that you need (and specified in your Gemfile) already installed on your system, no additional commands are required. If you don’t, a simple bundle install will get you up to date.

Deployment

After developing your application, you will probably need to deploy it. With bundler, if you are deploying your application with capistrano, you just add require "bundler/capistrano" to your deploy.rb. This will automatically install the gems in your Gemfile, using deployment-friendly settings. That’s it!

Before bundler, you had two options:

  • Most commonly, “make sure that all the gems I need are on the remote system”. This could involve including the gem install in the capistrano task, or even sshing into the remote system to install the needed gems. This solution would work, but would often leave no trail about what exactly happened. This was especially problematic for applications with a lot of dependencies, or applications with sporadic maintenance
  • Using the GemInstaller gem, which allowed you to list the gems you were using, and then ran gem install on each of the gems. Heroku used a similar approach, with a .gems manifest. While this solved the problem of top-level dependencies, it did not lock in dependencies of dependencies. This caused extremely common problems with rack and activesupport, and a long tail of other issues. One particularly egregious example is DreamHost, which directs users to use a brittle patch to Rails

In both of these cases, there was no guarantee that the gems you were using in development (including dependencies of dependencies like Rack and Active Support) remained the same after a deploy. One recent example that I saw was that carrierwave added an updated dependency on the newest activesupport, which depends on Ruby 1.8.7. Without making any change to his .gems manifest, Heroku failed to deploy his application on their Ruby 1.8.6 stack.

With bundler, this sort of problem can never happen, because you’re always running exactly the same versions of third-party code in production as you are in development and staging. This should increase your confidence in deployments, especially when you come back to an application months after it was originally developed.

Tweaked Gems

Some times, you will need to make small changes to one of your dependencies, and deploy your application with those changes. Of course, in some cases you can make the tweaks via monkey-patching, but in other cases, the only real solution is to change the gem itself.

With bundler, you can simply fork the gem on Github, make your changes, and change the gem line in your Gemfile to include a reference to the git repository, like this:

# change this
gem "devise"
 
# to this
gem "devise", :git => "git://github.com/plataformatec/devise.git"

This behaves exactly the same as a gem version, but uses your tweaks from Github. It participates in dependency resolution just like a regular gem, and can be installed with bundle install just like a regular gem. Once you’ve added a git repository, you run bundle install to install it (since the normal gem command can’t install git repositories).

You can use :branch, :tag, or :ref flags to specify a particular branch, tag or revision. By default, :git gems use the master branch. If you use master or a :branch, you can update the gem by running bundle update gem_name. This will check out the latest revision of the branch in question.

Conclusion

Using bundler for the sorts of things that you would have handled manually before should be easier than before. Bundler will handle almost all of the automatable record-keeping for you. While doing so, it offers the incredible guarantee that you will always run the same versions of third-party code.

There are a lot of advanced features of Bundler, and you can learn a lot more about them at the bundler web site.

Archives

Categories

Meta