Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Bundler 0.9: Heading Toward 1.0

Over the past two years, Carl and I have been working on-again off-again on the problem of dependency resolution. Fabien Franzen implemented the first partial solution for Merb, with thor merb:gem:install.

When we started working on Rails, we knew we wanted to finally crack the nut, making it possible for Rails itself to have some of its own dependencies, and solving some persistent, long-term problems with gem plugins.

Earlier this year, we released bundler, our first real attempt to solve this problem. Among other things, it shipped with the first true dependency resolver for Rubygems. It also modeled the entire dependency process on top of Rubygems, allowing us to support git repositories in the same dependency resolution process as Rubygems themselves.

Over the next few months, we refined the bundler quite a bit. We’re proud of the fact that individuals, web shops, and deployment companies have adopted bundler, and Gemfile has become a standard way of expressing gem dependencies for an app.

During this time, we’ve also received quite a bit of feedback. As we approach the release of Bundler 1.0, which we hope to ship along with Rails 3.0 final, we took the opportunity to take a look at all the feedback people have sent so far.

Having done so, we’re proud to announce Bundler 0.9 with radically improved workflows that fit our needs and the needs of those who have contributed feedback and patches.

The Cache

Before Bundler 0.9, Bundler installed gems and git repositories to a local application cache. When we looked at the deployment workflow, we found that while people liked the ability to cache their .gem files in their application, they had a lot of trouble at deployment time.

Essentially, people (and we!) wanted the ability to expand the gems to a central cache that they could reuse across deployments. People tried various symlinking approaches, but it became clear to us Bundler 1.0 needed to support this workflow natively.

On a related note, people who developed a number of applications on a single machine wanted to be able to reuse a single system cache across their application, and not need to connect to remotes so often when they already had all the gems they needed on their systems.

Finally, Rails itself grated against the default Bundler workflow. Most people installing a new Rails app run gem install rails, then rails my_app, and then want to go into their application and start working. But with Bundler 0.5 to 0.8, they needed to hit the remotes again even though, by definition, the system gem cache already had all the gems they needed.

Runtime

The fully-packaged application works fantastically for deployment, enabling a repeatable, reliable development to staging to production workflow. However, the need to explicitly bundle the application after added a new dependency in the very early stages of an application’s life cycle feels more like a compile step than a lean mean agile machine.

In Bundler 0.9, we’re adding the ability to run your application against gems already on your system, with the same reliability that you’ve grown to love from our dependency resolver.

Once you’ve gotten your application working, you can lock down the dependencies, so coworkers or production will use exactly the same environment that worked for you.

This allows you to use your common system gem repository across multiple apps, with a fully resolved virtual environment for each application based on the app’s dependencies specified in the Gemfile.

Locking

A lot of people (including me) love the idea of storing the .gem files in the application to create a single deployable unit with no network dependencies.

Storing the .gem files in the repository also provides a record of the fully resolved dependencies for your application, so that a new released gem between testing and deployment cannot change the environment. In Bundler 0.9, we provide a mechanism for you to save that record without having to also store the .gem files.

This won’t give you a single, dependency-less deployment package, but it will save you from unexpected environment changes.

New Commands

When we started Bundler, we had just one command: gem bundle. Over the following months, the command took on a life of its own, with a slew of flags representing both options and verbs. Bundler 1.0 will have a small, streamlined list of commands, each with its own set of flags:

bundle pack

Take all .gem files needed for the application and place them into the local application. By default, bundler places them in vendor/gems.

bundle lock

Resolve all dependencies in the Gemfile, and store a record of that process. In the future, use that record, preventing any changes in the environment or in the remote sources from changing the gems used by the application.

bundle install

Install the bundle specified by the Gemfile to the cache. By default, Bundler installs the gems (and git repositories) into system gems. You can override this to install into another location, or into your local application, just like in Bundler 0.8 and before.

After cloning a new repository for the first time, you will probably want to run this command.

If you packed gems, Bundler will use the gems from your application. If you packed and locked gems, Bundler will install the gems immediately without updating the remote sources.

bundle check

Check to see whether your cache has all the required dependencies. If it does not, Bundler will print out the list of missing gems.

bundle exec …

Run the specified command using the local environment. For instance, if you have Rails 2.3 and 3.0 installed, if you run bundle exec rails . in a Rails 2.3 application, Bundler will run the rails command using Rails 2.3 (and other gems in the current environment).

Environments

In Bundler 0.9, as in Bundler 0.8 and before, the Gemfile specifies an app-specific environment. New in Bundler 0.9, you may store the .gem files and unpacked gems in a central location, as well as in the application itself.

Because Bundler ships with a dependency resolver, you do not need to specify named environments and switch between them. Instead, the environments are virtual, based on the gems you specify as application dependencies in the Gemfile, and you still get all the benefits of a shared, system-wide cache.

18 Responses to “Bundler 0.9: Heading Toward 1.0”

It sounds like you’ve brought together all the best parts of maven and improved them.

So, how do these changes effect people currently using this with Rails 2.3.x?

I really like the way the system works, and I’ve got it running on multiple deployments, each with their own gems. I’d hate to think I need to put them all in a common area, since I might as well just install them on the system again.

Could you post a new official “HOW-TO” for using it with 2.3.x? I think a lot of people would appreciate a page with that information.

Nice. The “compile step”, which seems to be a very appropriate name to me, always was that one thing that held me from using bundler.

Will it be possible to have the Gemfile as the only bundler-related file an a rails app (when using system gems)?

Two questions:

* Did the source move? The previous repository doesn’t have any commits since early January http://github.com/wycats/bundler

* Will the instructions on http://yehudakatz.com/2009/12/31/spinning-up-a-new-rails-app/ still work? It appears that “directory” has changed to “path”, but it still doesn’t store the Rails source locally. Maybe another directive is needed?

To answer my own question, the source is now here: http://github.com/carlhuda/bundler

@JGeiger check out his previous post “Using the new gem bundler today” http://yehudakatz.com/2009/11/03/using-the-new-gem-bundler-today/

Doesn’t account for the new stuff here’s talking about here, but those aren’t released yet.

Im noticing latest on http://github.com/carlhuda/bundler is broken on 1.9.2 head when it comes to remote sources and introduces a dependency on latest git (for the –recursive option) – already reported on github but thought it is worth mentioning.

Bundler really should be called isolator … its totally not just about bundling anymore :)

It seems there are a number of issues with bundler and Windows in the Issue Tracker on yehudas github. Also, bundler and JRuby. Recall that JRuby can’t use native extensions. Hopefully these platforms will be checked before final release.

“You can override this to install into another location, or into your local application, just like in Bundler 0.8 and before.”

Can someone elaborate on how to make that happen?

You may want to update http://weblog.rubyonrails.org/2010/1/1/getting-a-new-app-running-on-edge to reflect this change. Maybe your blog post as well :)

Bundler is now broken for me with rails 2.3.5. No dependancies load at all or load in the wrong order (not sure since I can’t get my app running). Can be duplicated by creating a fresh app and adding your hoptoad key in initializers. Boom.

active support is now broken for rails 2.3.x, as a LoadError is now thrown instead of a Gem::LoadError.

also, how do I go about supporting simultaneous development with multiple ruby interpreters? I can’t lock multiple complete environments any more… ‘ruby-debug’ for ruby and ‘ruby-debug’ for jruby are actually different gems that unfortunately have the same name. I CAN NOT cache them to the same place. Bundler pre 0.8 would never properly select -java gems so I ended up rolling my own environment jail before 0.8 introduced its’ own… i’m sad to see this dropped -entirely-.

I got this error in a brand new app containing just this Gemfile:

“”"
source :gemcutter

gem “rails”, :git => “git://github.com/rails/rails.git”
“”"

$ bundle install
Fetching git://github.com/rails/rails.git

Fetching source index from http://gemcutter.org
Resolving dependencies
Installing abstract (1.0.0) from system gems
Installing actionmailer (3.0.0.beta) from git://github.com/rails/rails.git (at master) fatal: destination directory ‘/home/marco/.bundle/bundler/gems/rails-16a5e918a06649ffac24fd5873b875daf66212ad-master’ already exists.
fatal: Not a git repository
An error has occurred in git. Cannot complete bundling.

I have the same problem as Marco Lazzeri. But not with the rails gem, but with will_paginate. On my local development machine it works perfectly. But on the production server it always fails with the message that the directory already exists – event if I delete the dir upfront.

It seems that the gems are not loading in proper order because I keep getting `load_missing_constant’: uninitialized constant Handsoap (NameError)

has anyone else received this type of error?

I just attempted to update my already-running-on-Rails-3-pre app to the Rails 3 beta and having major headaches with the current version of bundler. `bundle install` seems to install to ~/.bundle NOT the system. This took 20 minutes of digging around to figure out what happened since bundle install doesn’t tell you where it’s installing. Also, if I’m using Passenger, I need to install to system gems, or at least the boot process needs to be able to find these gems.

I wish that you guys would keep a concise changelog and focus your efforts on the documentation. It seems like you’re pushing changes every day, and randomly pushing things up to gemcutter (you pushed 0.9.5 today, but 0.9.6 was tagged in the repo?). The documentation is okay, but thin, and most the details are elaborated in blog posts (both from you guys and other people in the community) which go out of date extremely fast. Googling turns up mostly outdated information which is just a wild goose chase. I realize this is not a 1.0.0 release and you have good reasons for the changes you are making, but these breaking changes are very hard to track down. How hard would it be to just keep a canonical source of information? It will save thousands of wasted hours for those of us edge riders who don’t have time to follow Rails development day in and day out, but still want to get on Rails 3 early.

This may help people who run into the same issues as Marco and Maximilian. I was having the same issues with

Not a git repository
An error has occurred in git. Cannot complete bundling.

It turns out I had an old version of bundler installed(0.9.4) along with 0.9.6. I assumed, when I installed the rails beta, that I did not have it installed at all and could safely ignore

Due to a rubygems bug, you must uninstall all older versions of bundler for 0.9 to work

Once I cleared out bundler from my gems completely and then installed 0.9.6, it started working properly.

If you are like me and upgraded a rails 3.0.pre app to rails 3.0.beta:

Make sure you reinstall, or copy from a fresh ‘rails .’ generated app, your boot.rb and environment.rb files, as well as script/console, script/dbconsole, and bin/rake. I had lots of issues until I manually updated these files. After that, bundler seemed to play nice.

I agree that the documentation for bundler is woefully inadequate and inconsistent, and there’s much outdated information on blogs about using it with Rails 3.0. Rubygems isn’t perfect, but I konw exactly how it works, and how to use it with my apps.

Leave a Reply

Archives

Categories

Meta