Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Bundler: As Simple as What You Did Before

One thing we hear a lot by people who start to use bundler is that the workflow is more complicated than it used to be when they first start. Here’s one (anonymized) example: “Trying out Bundler to package my gems. Still of the opinion its over-complicating a relatively simple concept, I just want to install gems.”

Bundler has a lot of advanced features, and it’s definitely possible to model fairly complex workflows. However, we designed the simple case to be extremely simple, and to usually be even less work than what you did before. The problem often comes when trying to handle a slightly off-the-path problem, and using a much more complex solution than you need to. This can make *everything* much more complicated than it needs to be.

In this post, I’ll walk through the bundler happy path, and show some design decisions we made to keep things moving as smoothly as before, but with far fewer snags and problems than the approach you were using before. I should be clear that there are probably bugs in some cases when using a number of advanced features together, and we should fix those bugs. However, they don’t reflect core design decisions of bundler.

In the Beginning

When you create a Rails application for the first time, the Rails installer creates a Gemfile for you. You’ll note that you don’t actually have to run bundle install before starting the Rails server.

$ gem install rails
Successfully installed activesupport-3.0.0
Successfully installed builder-2.1.2
Successfully installed i18n-0.4.1
Successfully installed activemodel-3.0.0
Successfully installed rack-1.2.1
Successfully installed rack-test-0.5.6
Successfully installed rack-mount-0.6.13
Successfully installed tzinfo-0.3.23
Successfully installed abstract-1.0.0
Successfully installed erubis-2.6.6
Successfully installed actionpack-3.0.0
Successfully installed arel-1.0.1
Successfully installed activerecord-3.0.0
Successfully installed activeresource-3.0.0
Successfully installed mime-types-1.16
Successfully installed polyglot-0.3.1
Successfully installed treetop-1.4.8
Successfully installed mail-2.2.6.1
Successfully installed actionmailer-3.0.0
Successfully installed thor-0.14.2
Successfully installed railties-3.0.0
Successfully installed rails-3.0.0
22 gems installed
$ rails s
=> Booting WEBrick
=> Rails 3.0.0 application starting in development on http://0.0.0.0:3000
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2010-09-30 12:25:51] INFO  WEBrick 1.3.1
[2010-09-30 12:25:51] INFO  ruby 1.9.2 (2010-08-18) [x86_64-darwin10.4.0]
[2010-09-30 12:25:51] INFO  WEBrick::HTTPServer#start: pid=26380 port=3000

If you take a look at your directory, you’ll see that Bundler noticed that you didn’t yet have a Gemfile.lock, saw that you had all the gems that you needed already in your system, and created a Gemfile.lock for you. If you didn’t have a necessary gem, the Rails server would error out. For instance, if you were missing Erubis, you’d get Could not find RubyGem erubis (~> 2.6.6). In this case, you’d run bundle install to install the gems.

As you can see, while bundler is providing some added features for you. If you take your application to a new development or production machine with the Gemfile.lock, you will always use exactly the same gems. Additionally, bundler will resolve all your dependencies at once, completely eliminating the problem exemplified by this thread on the Rails core mailing list. But you didn’t have to run bundle install or any bundle command unless you were missing some needed gems. This makes the core experience of bundler the same as “I just want to install gems”, while adding some new benefits.

Adding Gems

When you add a gem that is already installed on your system, you just add it to the Gemfile and start the server. Bundler will automatically pick it up and add it to the Gemfile.lock. Here again, you don’t need to run bundle install. To install the gem you added, you can run bundle install or you can even just use gem install to install the missing gems.

Once again, while bundler is handling a lot for you behind the scenes (ensuring compatibility of gems, tracking dependencies across machines), if you have all the gems that you need (and specified in your Gemfile) already installed on your system, no additional commands are required. If you don’t, a simple bundle install will get you up to date.

Deployment

After developing your application, you will probably need to deploy it. With bundler, if you are deploying your application with capistrano, you just add require "bundler/capistrano" to your deploy.rb. This will automatically install the gems in your Gemfile, using deployment-friendly settings. That’s it!

Before bundler, you had two options:

  • Most commonly, “make sure that all the gems I need are on the remote system”. This could involve including the gem install in the capistrano task, or even sshing into the remote system to install the needed gems. This solution would work, but would often leave no trail about what exactly happened. This was especially problematic for applications with a lot of dependencies, or applications with sporadic maintenance
  • Using the GemInstaller gem, which allowed you to list the gems you were using, and then ran gem install on each of the gems. Heroku used a similar approach, with a .gems manifest. While this solved the problem of top-level dependencies, it did not lock in dependencies of dependencies. This caused extremely common problems with rack and activesupport, and a long tail of other issues. One particularly egregious example is DreamHost, which directs users to use a brittle patch to Rails

In both of these cases, there was no guarantee that the gems you were using in development (including dependencies of dependencies like Rack and Active Support) remained the same after a deploy. One recent example that I saw was that carrierwave added an updated dependency on the newest activesupport, which depends on Ruby 1.8.7. Without making any change to his .gems manifest, Heroku failed to deploy his application on their Ruby 1.8.6 stack.

With bundler, this sort of problem can never happen, because you’re always running exactly the same versions of third-party code in production as you are in development and staging. This should increase your confidence in deployments, especially when you come back to an application months after it was originally developed.

Tweaked Gems

Some times, you will need to make small changes to one of your dependencies, and deploy your application with those changes. Of course, in some cases you can make the tweaks via monkey-patching, but in other cases, the only real solution is to change the gem itself.

With bundler, you can simply fork the gem on Github, make your changes, and change the gem line in your Gemfile to include a reference to the git repository, like this:

# change this
gem "devise"
 
# to this
gem "devise", :git => "git://github.com/plataformatec/devise.git"

This behaves exactly the same as a gem version, but uses your tweaks from Github. It participates in dependency resolution just like a regular gem, and can be installed with bundle install just like a regular gem. Once you’ve added a git repository, you run bundle install to install it (since the normal gem command can’t install git repositories).

You can use :branch, :tag, or :ref flags to specify a particular branch, tag or revision. By default, :git gems use the master branch. If you use master or a :branch, you can update the gem by running bundle update gem_name. This will check out the latest revision of the branch in question.

Conclusion

Using bundler for the sorts of things that you would have handled manually before should be easier than before. Bundler will handle almost all of the automatable record-keeping for you. While doing so, it offers the incredible guarantee that you will always run the same versions of third-party code.

There are a lot of advanced features of Bundler, and you can learn a lot more about them at the bundler web site.

17 Responses to “Bundler: As Simple as What You Did Before”

The only downside I’ve had from Bundler is the speed it takes to resolve the dependencies. But I’ve seen this is more Rubygem’s shortcoming than Bundler’s. In addition it looks like Rubygems should be getting a much needed feature push soon to resolve this issue.

Other than that I’ve been a big fan of Bundler. To the point I’ve been going back and adding it into older projects.

I don’t really understand people saying that bundler over-complicating smth. Have they ever developed app bigger then hello world and deployed it? In my case there was lots of problems with gems, especially in active development stages. With bundler everything just work. No more pain about gems versions or crazy bugs after deployment etc. I’m realy very grateful to all you guys who developed such a great tool as bundler.

Bundler is one of the best parts of my workflow these days. It saves time and it gives me confidence that I can open up a project from a few months ago, type `bundle install` and I’m ready to roll. It was not very long ago that things were a lot more complicated and painful than that.

From my experience I’ve found most people that have had issues with Bundler (especially performance related ones) are often running (or referring to) an outdated version.

I don’t get all the back-handed comments from everybody about bundler. The only issue I’ve ever have with Bundler is when I was trying to run an app as root (something I shouldn’t be doing anyway, so there). Other than that, Bundler has simplified my gem dependencies and remove the complexity I had previously with any project using over about 5 gems.

Certainly makes it easy to run edge or backported Rails without having to put it in vendor and bloat the project transport size…

I am a huge fan of Bundler and I’ve been using it since its early stages.
The only thing I wanted to yell about is “Fetching Rubygems…” and goes via internet wire each time, to get the list of gems.
Can’t it be cached and with some –flag force it to fetch the list of gems??

I don’t understand how people can say it’s more complicated, sure it was a long and winding road to get to bundler 1.0 but now we are here I’d say it’s far superior and simpler than what came before it.

I’m surprised. Bundler made dealing with gems so much simpler. It’s so much easier to for a gem and fix a bug while using it a Rails project, deploy with the fix, push it upstream and after some time have the gem released. So much simple that I’ve never done it before. I love Bundler.

To those doubting the usefulness of Bundler, try building an app to be spun up on demand via cloud virtual machines. Without Bundler, it’s very difficult to get the right version of every dependency, and even when you do you would need to create a new base image every time you upgrade a gem. Bundler also helps development on any size team; when I add a new gem I no longer need to tell everyone what to install (and help some install it), I just tell everyone to run bundle install.

Regardless of _everyone_ grokking it, Bundler is a necessary step for Ruby’s evolution if it’s going to continue being useful. Thanks for all the hard work Yehuda!

We recently started using Bundler, and for us, the tough part was getting everything setup. I spent the better part of 2 nights digging into dependency issues and version conflicts before finally getting back to stable.

Now that we’re back to a good state, I expect keeping environments in sync should be a non issue.

I feel good to work with the bundle, see how simple it is to treat all project dependencies.

@Yehuda, are you using RVM’s custom gemsets? It seems this post is intimating that Bundler obviates that need: just install your gems to system (or RVM’s @global) and let Bundler sort them out.

With custom gemsets I find I’m often running ‘bundle install’. Perhaps I’m doing it wrong(tm)?

I had my issues with bundler in the beginning and have been using it from 0.8x. It was painful to use and I was quite frustrated. I will say right now that they’ve done a wonderful job with it, and if you’re not using it, you really need to be. There are still a few bugs that need to be worked out (gems with :git repos specified in test groupings, I’m looking at you…) but overall it’s a good experience.

Bundler has been hit and miss with me. An issue I ran into today with a project is it kept insisting I needed to install acts-as-taggable-on. It was in the Gemfile, it was already installed, but it insisted it was missing. Pure frustration ensued.

Seems like a clean way to do this, but I’m having trouble with this use case: an existing project where we’ve frozen the rails gems and applied patches manually that did not get rolled into 2.3.8 (little things like missing constants from const_get, json date parsing etc.) Further there is a soap4r gem installed which has patches in place in its gemdir and there is no actual source for this (it was probably checked out from git and patched.) What’s the easiest way to get bundler to cram all of these into vendor/bundle so I can get on with ‘getting things done’ and not messing with deploying patched gems? Check in the soap4r gem back to a git source I suppose? Is there any way to tell bundler to just source the gem (copy it) from the local filesystem?

Also where/how does bundler keep its list of gems installed in vendor/bundle? Because when I just copied the files (and the specifications, cached files etc.) from gemdir to vendor/bundle it didn’t automatically pick up those additions from the filesystem.

ps. I did try gem ‘soap4r’, ’1.5.8.2′, path: ‘gemdir/soapr4r-1.5.8.2′ in the Gemfile and this resulted in:

Using soap4r (1.5.8.2) from source at ~/.rvm/gems/ruby-1.9.1-p378/gems/soap4r-1.5.8.2 ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:374:in `gets’: Is a directory – ~/.rvm/gems/ruby-1.9.1-p378/gems/soap4r-1.5.8.2/bin/.svn (Errno::EISDIR)
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:374:in `block in shebang’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:374:in `open’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:374:in `shebang’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:399:in `app_script_text’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:328:in `block in generate_bin_script’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:327:in `open’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:327:in `generate_bin_script’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:304:in `block in generate_bin’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:297:in `each’
from ~/.rvm/rubies/ruby-1.9.1-p378/lib/ruby/site_ruby/1.9.1/rubygems/installer.rb:297:in `generate_bin’
from ~/.rvm/gems/ruby-1.9.1-p378/gems/bundler-1.0.2/lib/bundler/source.rb:380:in `generate_bin’
from ~/.rvm/gems/ruby-1.9.1-p378/gems/bundler-1.0.2/lib/bundler/source.rb:397:in `install’
from ~/.rvm/gems/ruby-1.9.1-p378/gems/bundler-1.0.2/lib/bundler/installer.rb:55:in `block in run’

Well now that I’ve committed and then reinstalled the gem from a git repo it’s installed it under vendor/bundle/…/bundler/gems. But now when I launch the app, Rails says that required gems are missing although they are installed in the system gemenv. Whoa. Am I missing something here? I only want to bundle/freeze SOME of the gems–the ones we have to patch to get to work, and have all of the others pulled normally from the system gemenv. Is Bundler just not designed to do this? I wish I had known several hours ago.

I’ve used Bundler on a number of projects and I think it really streamlines the gem install and upgrade process. I’ve had a few minor issues but usually to do with the gems themselves not being packaged properly.

Well done. I personally think you guys have done a great job on this.

Leave a Reply

Archives

Categories

Meta