Using the New Gem Bundler Today

November 3rd, 2009

As you might have heard, Carl and I released a new project that allows you to bundle your gems (both pure-ruby and native) with your application. Before I get into the process for using the bundler today, I’d like to go into the design goals of the project.

The bundler should allow the specification of all dependencies in a separate place from the application itself. In other words, it should be possible to determine the dependencies for an application without needing to start up the application.
The bundler should have a built-in dependency resolving mechanism, so it can determine the required gems for an entire set of dependencies.
Once the dependencies are resolved, it should be possible to get the application up and running on a new system without needing to check Rubyforge (or gemcutter) again. This is especially important for compiled gems (it should be possible to get the list of required gems once and compile on remote systems as desired).
Above all else, the bundler should provide a reproducible installation of Ruby applications. New gem releases or down remote servers should not be able to impact the successful installation of an application. In most cases, git clone; gem bundle should be all that is needed to get an application on a new system and up and running.
Finally, the bundler should not assume anything about Rails applications. While it should work flawlessly in the context of a Rails application, this should be because a Rails application is a Ruby application.

Using the Bundler Today

To use the gem bundler today in a non-Rails application, follow the following steps:

gem install bundler
Create a Gemfile in the root of your application
Add dependencies to your Gemfile. See below for more details on the sorts of things you can do in the Gemfile. At the simplest level, gem "gem_name", "version" will add a dependency of the gem and version to your application
At the root, run gem bundle. The bundler should tell you that it is resolving dependencies, then downloading and installing the gems.
Add vendor/gems/gems, vendor/gems/specifications, vendor/gems/doc, and vendor/gems/environment.rb to your .gitignore file
Inside your application, require vendor/gems/environment.rb to add the bundled dependencies to your load paths.
Use Bundler.require_env :optional_environment to actually require the files.
After committing, run gem bundle in a fresh clone to re-expand the gems. Since you left the vendor/gems/cache in your source control, new machines will be guaranteed to use the same files as the original machine, requiring no remote dependencies

The bundler will also install binaries into the app’s bin directory. You can, therefore, run bin/rackup for instance, which will ensure that the local bundle, rather than the system, is used. You can also run gem exec rackup, which runs any command using the local bundle. This allows things like gem exec ruby -e "puts Nokogiri::VERSION" or the even more adventurous gem exec bash, which will open a new shell in the context of the bundle.

Gemfile

You can do any of the following in the Gemfile:

gem "name", "version": version may be a strict version or a version requirement like >= 1.0.6. The version is optional.
gem "name", "version", :require_as => "file": the require_as allows you to specify which file should be required when the require_env is called. By default, it is the gem’s name
gem "name", "version", :only => :testing: The environment name can be anything. It is used later in your require_env call. You may specify either :only, or :except constraints
gem "name", "version", :git => "git://github.com/wycats/thor": Specify a git repository to be used to satisfy the dependency. You must use a hard dependency (“1.0.6″) rather than a soft dependency (“>= 1.0.6″). If a .gemspec is found in the repository, it is used for further dependency lookup. If the repository has multiple .gemspecs, each directory will a .gemspec will be considered a gem.
gem "name", "version", :git => "git://github.com/wycats/thor", :branch => "experimental": Further specify a branch, tag, or ref to use. All of :branch, :tag, and :ref are valid options
gem "name", "version", :vendored_at => "vendor/nokogiri": In the next version of bundler, this option will be changing to :path. This specifies that the dependency can be found in the local file system, rather than remotely. It is resolved relative to the location of the Gemfile
clear_sources: Empties the list of gem sources to search inside of.
source "http://gems.github.com": Adds a gem source to the list of available gem sources.
bundle_path "vendor/my_gems": Changes the default location of bundled gems from vendor/gems
bin_path "my_executables": Changes the default location of the installed executables
disable_system_gems: Without this command, both bundled gems and system gems will be used. You can therefore have things like ruby-debug in your system and use it. However, it also means that you may be using something in development mode that is installed on your system but not available in production. For this reason, it is best to disable_system_gems
disable_rubygems: This completely disables rubygems, reducing startup times considerably. However, it often doesn’t work if libraries you are using depend on features of Rubygems. In this mode, the bundler shims the features of Rubygems that we know people are using, but it’s possible that someone is using a feature we’re unaware of. You are free to try disable_rubygems first, then remove it if it doesn’t work. Note that Rails 2.3 cannot be made to work in this mode
only :environment { gem "rails" }: You can use only or except in block mode to specify a number of gems at once

Bundler process

When you run gem bundle, a few things happen. First, the bundler attempts to resolve your list of dependencies against the gems you have already bundled. If they don’t resolve, the metadata for each specified source is fetched and the gems are downloaded. Next (either way), the bundler checks to see whether the downloaded gems are expanded. For any gem that is not yet expanded, the bundler expands it. Finally, the bundler creates the environment.rb file with the new settings. This means that running gem bundler over and over again will be extremely fast, because after the first time, all gems are downloaded and expanded. If you change settings, like disable_rubygems, running gem bundle again will simply regenerate the environment.rb.

Rails 2.3

To get this working with Rails 2.3, you need to create a preinitializer.rb and insert the following:

require "#{File.dirname(__FILE__)}/../vendor/bundler_gems/environment"
 
class Rails::Boot
  def run
    load_initializer
    extend_environment
    Rails::Initializer.run(:set_load_path)
  end
 
  def extend_environment
    Rails::Initializer.class_eval do
      old_load = instance_method(:load_environment)
      define_method(:load_environment) do
        Bundler.require_env RAILS_ENV
        old_load.bind(self).call
      end
    end
  end
end

It’s a bit ugly, but you can copy and paste that code and forget it. Astute readers will notice that we’re using vendor/bundler_gems/environment.rb. This is because Rails 2.3 attaches special, irrevocable meaning to vendor/gems. As a result, make sure to do the following in your Gemfile: bundle_path "vendor/bundler_gems".

Gemcutter uses this setup and it’s working great for them.

Bundler 0.7

We’re going to be releasing Bundler 0.7 tomorrow. It has some new features:

List outdated gems by passing --outdated-gems. Bundler conservatively does not update your gems simply because a new version came out that satisfies the requirement. This is so that you can be sure that the versions running on your local machine will make it safely to production. This will allow you to check for outdated gems so you can decide whether to update your gems with –update. Hat tip to manfred, who submitted this patch as part of his submission to the Rumble at Ruby en Rails
Specify the build requirements for gems in a YAML file that you specify with --build-options. The file looks something like this:
```
mysql:
  config: /path/to/mysql_config
```
This is equivalent to –with-mysql-config=/path/to/mysql_config
Specify :bundle => false to indicate that you want to use system gems for a particular dependency. This will ensure that it gets resolved correctly during dependency resolution but does not need to be included in the bundle
Support for multiple bundles containing multiple platforms. This is especially useful for people moving back and forth between Ruby 1.8 and 1.9 and don’t want to constantly have to nuke and rebundle
Deprecate :vendored_at and replace with :path

A new directory DSL method in the Gemfile:

directory "vendor/rails" do
  gem "activesupport", :path =&gt; "activesupport" # :path is optional if it's identical to the gem name
                                                # the version is optional if it can be determined from
                                                # a gemspec
end

You can do the same with the git DSL method

git "git://github.com/rails/rails.git" do # :branch, :tag, or :ref work here
  gem "activesupport", :path =&gt; "activesupport" # same rules as directory, except that the files are
                                                # first downloaded from git.
end

Fix some bugs in resolving prerelease dependencies

This entry was posted on Tuesday, November 3rd, 2009 at 12:50 am and is filed under Rails 3, Ruby, Ruby on Rails. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

34 Responses to “Using the New Gem Bundler Today”

Tom Ward, Posted November 3, 2009, 6:16 am

I found that the preinitializer.rb you’ve described doesn’t work under passenger. Passenger doesn’t seem to load config/boot.rb, so the Rails::Boot constant doesn’t exist, and even if it did, the boot wouldn’t be run. I’ve taken to hijacking rails initializer in environment.rb instead. I’ve also written a rails template to add bundler to a project, described here: http://bit.ly/4xCxm7

Hope this is helpful to some people and thanks for the hard work on bundler

Dean Strelau, Posted November 3, 2009, 8:21 am

Thank you! Just last night I was fighting with trying to get this working in a Rails 2.3 app without much luck. Your preinitializer file fixed it right up.

Mikkel Malmberg, Posted November 3, 2009, 8:50 am

I’ve already used it and it works great. It’s the first bundling solution I’ve tried that just – you know – works.

So thanks a bunch!

Hongli Lai, Posted November 3, 2009, 9:32 am

@Tom Ward: Phusion Passenger does load boot.rb and preinitializer.rb. Or to be more exact:
- When the “smart-lv2″ spawn method is used (the default), it loads preinitializer.rb, then environment.rb. environment.rb is responsible for loading boot.rb.
- When the “smart” spawn method is used, it first loads the Rails framework, then preinitializer.rb, then environment.rb. In this case the preinitializer hack that Yehuda posted won’t work.
- When the “conservative” spawn method is used, it loads environment.rb. environment.rb loads boot.rb, which in turn loads preinitializer.rb.

Tom Ward, Posted November 3, 2009, 11:12 am

Hongli Lai – Sorry, I didn’t investigate the issue extensively. While I can see that passenger does load boot.rb, for the preinitializer hack to work, doesn’t it need to load boot.rb before preinitializer.rb? In which case only the conservative spawn method will work.

Thanks for explaining the differences between the three though, I didn’t realise they’d change the load order.

James H., Posted November 3, 2009, 11:32 am

Fantastic work, Carlhuda! This definitely scratches a few itches I’ve been having.

Tim Elliott, Posted November 3, 2009, 12:36 pm

I’ve been adding the bin directory to my .gitignore file as well.

The hash-bang line at the beginning of the bin scripts depends on the environment in which they are installed.

Richie Vos, Posted November 3, 2009, 1:36 pm

I personally added vendor/gems/cache to my gitignore as well. When deploying I:
1. deploy the code
2. symlink shared/vendor/gems as release/vendor/gems
3. run gem bundle

This makes my deploy really fast, since it doesn’t have to redownload gems it already has, and doesn’t have to pull that gem sources/specs down.

Since I can specify a version in my Gemfile, the only thing I’d see being worried about is that what if my gems somehow disappear when I’m deploying, but I’m ok with that rare issue being an issue.

Contact me if anyone’s interested in a gist of my cap file.

Richie Vos, Posted November 3, 2009, 1:45 pm

Also, wanted to just say thanks for making this. I’m pretty happy with it so far. It works orders of magnitude better than rake gems:install.

I’m using it under passenger with a completely different config that’s working well.

I have the gems in vendor/gems

In environment.rb I have:
# turn off rails yelling at me about missing gemspecs
Rails::VendorGemSourceIndex.silence_spec_warnings = true

My preinitializer.rb:
require “#{File.dirname(__FILE__)}/../vendor/gems/environment”

I’ll have to reread this to figure out why the more complicated version’s needed, but this version “works for me”.

One weird thing is ‘gem bundle’ doesn’t work for me when I use the ruby enterprise edition gem command. To get around this I just run gem bundle with the normal ruby 1.8.7 gem command. I’m thinking it’s something weird with my server config though, not a bundler issue.

Jack Dempsey, Posted November 3, 2009, 3:19 pm

Thanks for the writeup Yehuda. Documentation/discussion like this does a great job in helping all of us move forward.

Also, nice template Tom. I’ve included it as a built-in option in beet–this should let people experiment with new Rails apps in an even easier fashion. For instance, as of the 0.4.0 release, you can now run this command to generate an application based on your template and set it up with git (and of course any other recipes you desire):

$ beet -g bundler_test_app -m bundler -r rails/git

Feel free to take a look at the project here: http://jackdempsey.github.com/beet/

Dean Strelau, Posted November 3, 2009, 9:37 pm

So it turns out things aren’t quite all rosey yet. I’m trying to use the bundler for a 2.3 app and getting this error:

ActionView::TemplateError (undefined method `content_for’ for #)

Other ActionView methods seem to be similarly missing. Full backtrace and Gemfile are at http://gist.github.com/225591 if thinks they may have any ideas. Thanks!

Jeroen Zwartepoorte, Posted November 6, 2009, 5:13 am

How to get rails 2-3-stable branch included in your app:

only :none do
git ‘git@github.com:jpzwarte/rails.git’, :branch => ’2-3-stable’ do
%w(activesupport activerecord actionpack actionmailer activeresource).each { |lib| gem lib, ’2.3.4′, :path => lib }
gem ‘rails’, ’2.3.4′, :path => ‘railties’
end
end

My clone has .gemspec files which bundler needs to work.

Matthew Todd, Posted November 9, 2009, 7:23 am

FYI, it seems gemcutter have moved the Rails::Boot monkeypatch from config/preinitializer.rb to config/boot.rb:

http://github.com/qrush/gemcutter/commit/1708305422738a074ec6c57f21ffb713496cd46b

Doing the same in my own app fixed the Passenger trouble I (like @Tom Ward) was having.

Aaron Gibralter, Posted November 18, 2009, 4:48 pm

@Tom Ward @Matthew Todd … @everyone

Ruby only lets you do something like

class SomeModule::SomeKlass; … end;

if the constant SomeModule is already defined. The problem Tom described above can easily be solved by changing it to:

module Rails
class Boot
…
end
end

http://gist.github.com/238294

Hey, it’s a monkey patch anyway, so it’s not supposed to be elegant…

Richie Vos, Posted November 28, 2009, 2:08 am

I had someone ask me, so here’s my capistrano script for deploying bundler: http://gist.github.com/244420

cap bundler:install is a 1-time install to get bundler downloaded.

cap bundler:bundle_new_release is what I have hooked into deploy:after_update_code to actually pull down the new code.

There is a gotcha with this approach. Since it’s sharing the same gems between releases, if your deploy fails, the roll-back may leave you with the wrong gems. All it would take is a on rollback to fix that, but I’ll leave that to those more concerned with rollback working :).

Donovan Bray, Posted December 7, 2009, 1:09 pm

Richie, you should add “–cached” to your bundle command, otherwise you will find the versions of dependencies creeping forward ignoring what was previously cached.

http://gist.github.com/250979

Scott Motte, Posted December 14, 2009, 11:41 am

Richie thanks for posting that for everyone. It’s been working great for me in production.

Manuel Morales, Posted December 14, 2009, 12:50 pm

It seems that I’m getting a loading order problem. Looks like calendar_date_select and faker gems require Rails to be loaded first:

/home/manuel/scrum/teamtrick/vendor/bundler_gems/gems/calendar_date_select-1.15/lib/calendar_date_select.rb:6:NameError: uninitialized constant ActionView::Helpers::FormHelper

The Aaron Gibralter fix (http://gist.github.com/242240/) didn’t work for me.

Any clue?

Gary S. Weaver, Posted January 4, 2010, 1:48 pm

Great post, and thanks for your work on Bundler! Some thoughts on how it fills the gap for Ruby/Rails that Maven/Ivy filled for Java here:
http://stufftohelpyouout.blogspot.com/2010/01/im-huge-fan-of-bundler.html

As stated in that post, I think the two huge features in Bundler are the easy ability to scope dependencies only for use in Rake testing (unit/functional, etc.) and the ability to get dependencies easily from Git, which is why I think that Github should start thinking about the surge in traffic that will start coming their way in the next few years.

What I’m hoping that will evolve though is a somewhat DRY-er Gemfile. Right now it can be fairly succinct, but I’ve already seen examples of usage for scoping dependencies by others that could have been expressed much more simply and clearly. Coming from the Java and Maven2 side, simplicity was key in its adoption. If there were a set of Bundler recipes readily available both in git of Bundler/linked in its README and in other central documentation, I think it might help a lot. This post goes a long way in that regard, though!

Richie Vos, Posted January 12, 2010, 2:35 pm

In case anyone comes across this. The new version of bundler has changed the pathing to make it cross ruby version compatible. Make sure to update your deploy files and bundle installs accordingly.

I’ve updated the gist at http://gist.github.com/244420 accordingly.

@donovan, I originally was using bundler on a project where I wasn’t committing the gem cache, so I left the –cached out. However, I think that’s a bad idea in the long-run, and have started committing the cache, and doing exactly what you said (adding –cached). The gist has been updated with that as well.

The one thing that sucks about that is my git repos keep growing whenever I update gems, but maybe git sub-module’ing gems is the solution to that.

@scott, np

Steven Holloway, Posted January 20, 2010, 4:47 pm

On our last project we used gem collector (by MIke Williams of Cogent Consulting) which is a similar idea to bundler but goes no where near as far.
We started a new project yesterday and bundler was my my 2nd step.
It went in without a hitch.
The setup is intuitive and we we had our gems sorted in less than 30 minutes.
Now reading about the build options feature just makes me more excited.
Awesome work Yehuda. As we say in Oz, Your a legend.
:-) Steven

David, Posted January 23, 2010, 6:27 pm

Where do I save the preinitializer.rb?

Dalibor Nasevic, Posted January 30, 2010, 5:09 am

@David: config/preinitializer.rb

Sketchy, Posted February 4, 2010, 10:20 am

“gem bundle” doesn’t work on Mac with -v 0.9.1? Anyone else seen this?

iain, Posted February 9, 2010, 4:13 am

@Sketchy The new command is “bundle” (without the “gem” part).

I was also not am amused by the second big change in bundler. Every minor version release seems to be completely different. Very confusing…

DGM, Posted February 11, 2010, 9:53 pm

clear_sources doesn’t work any more either. Is there any up to date docs on this?

wycats, Posted February 12, 2010, 12:44 pm

clear_sources is now the default. If you don’t want gemcutter, don’t include it :)

DGM, Posted February 15, 2010, 10:46 am

Too many tutorials out there with obsolete syntax… :)

Does this preinitializer still work with 2.3?

Mohammed, Posted February 15, 2010, 11:28 am

Any chance of getting an updated tutorial for the latest version of Bundler to work with Rails 2.3.4?

Thanks

roger, Posted June 11, 2010, 4:48 pm

Are those ampersands in the post expected?

Fred, Posted July 15, 2010, 6:41 pm

@Hongli Lai: Does the smart spawning method work at all with Bundler? Or are they mutually exclusive? I can’t seem to get smart spawning mode to work with bundler, whereas smart-lv2 and conservative works perfectly.

Please see: http://code.google.com/p/phusion-passenger/issues/detail?id=512

Paul Jensen, Posted August 25, 2010, 5:22 am

@Hongli Lai Thanks, you’re pointer on the smart spawning method worked.

B7, Posted June 4, 2011, 8:43 am

Thanks for the info. The HTML entities are visible on this page:

gem “name”, “version”: version may be a strict version or a version requirement like >= 1.0.6. The version is optional.

Cumpanasu FLorin, Posted October 31, 2011, 7:41 am

Is there any preconfigured IDE with Rails-MySQL (eg Netbeans) ?

Using the New Gem Bundler Today

Using the Bundler Today

Gemfile

Bundler process

Rails 2.3

Bundler 0.7

34 Responses to “Using the New Gem Bundler Today”

Leave a Reply

Archives

Categories

Meta