Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Awesome bundling in Merb 1.0 final

Since we released RC1, people have had persistent issues with bundling Merb gems. It turns out that the most common reason, by far, is conflicts between gems bundled in your app with gems available in your system.

When we first set up bundling, we thought it would be convenient to be able to bundle only some of the dependencies for your application. As it turns out, that blunts the reason for bundling in the first place: hassle-free, write-once-run-anywhere applications.

Through the course of four RCs, we’ve tried a number of things, none of which were really effective. The only full solution that guarantees that your bundled app runs correctly on development boxes and production boxes is to bundle everything in your app, and completely ignore your system gem repository.

While it is easy enough to mix and match, the number of reports of failure, combined with failures by the core team itself, leads us to believe that this is a *bad* idea, especially given some of the current assumptions made by Rubygems (which causes it to activate the latest version of a gem, which might conflict with a different dependency).

A simple example of a failure:

  • merb-haml 0.9.12 relies on merb-core 0.9.12
  • Imagine you have merb-core 0.9.13 installed in your system
  • running the merb binary loads the latest merb from all your repositories.
  • this will load merb-core 0.9.13
  • a dependency on merb-haml 0.9.12 will try to activate merb-core 0.9.12, which will conflict with the already loaded 0.9.13.

This is just one example of many issues that people had. The solution is to rely entirely on bundled gems, and remove system gems from bundled binaries.

The side-effect is that you will need to bundle gems like mongrel, rake, etc. But it’s clear to me that even bundling mongrel and rake is a good idea, as different versions of mongrel or rake can have subtle differences that lead to infuriating bugs. I personally have experienced some of these bugs in my day, and being able to package up an entire working app would have been a godsend!

RC5, which should be the final release before 1.0 final is released at RubyConf, will include bundle-everything by default, as well as more support for webrat, JRuby, and Windows.


14 Responses to “Awesome bundling in Merb 1.0 final”

I think it’s fine for this to be default, but I would like to be able to go the dangerous route as well.

Maybe it would be useful to know where each gem is loaded from (when it prints out ‘dependency webrat …’).

what we need is a little github rubygems look-alike that dls all the relevant dependencies automatically;

whenever I’ve (in the past month or so) had to deploy a new merb stack (either on a production box) or just as a new development box it has usually taken ~2 hours regardless of whether it was gem install, sake task, thor task, w/w-out capistrano…. I think rubygems needs to be expanded on/replaced to stay current with the social code sharing that projects such as github/merb have brought about

my 0.02

@nathan it should be possible to go the dangerous route. However, if something makes the core team pull their hair out with frustration, it’s an extremely unrecommended course.

@feydr we’ve tracked down a lot of the issues that are frustrating you and I think the path I outlined in this post will actually improve things. We’re also planning on doing nightly builds, which should mean you don’t need to use github to install gems.

I disagree with you on this topic.

Basically the whole dependencies thing is for your to “lock” your gem with dependencies that you know it will work.

For example, merb-haml 0.9.12 should have stated it depends on merb-core = 0.9.12

Or at least, said: merb-core ~> 0.9.12

In the first way, it will only install or trigger merb-core 0.9.12 installation and activation.

In the second way, you state “this gem is safe to be run against merb-core 0.9.12 before 0.10.0″

I believe that package dependency is been neglected by gem developers, IMHO is their fault, not the package system.

@luis the problem is that it’s impossible to be sure what the combination of dependency requirements in your local and system gems will produce at runtime. Adding system gems can cause merb to suddenly load a system gem instead of a local gem and vice versa. I used one example to illustrate what had gone wrong, but it’s just one example.

Bundling everything prevents this confusion.

I agree on the unpredictable situations topic, so bundling/freezing is mandatory.

The only thing around that are gems that require compilation of extensions like Mongrel, ParseTree or even nokogiri.

Bundling all in OSX for your application deployment will fail when deployed in a Linux box.

Or I’m missing something?

@luis We bundle our gems in a similar fashion now and simply run the bundler as part of the deployment process. We don’t actually check our gems into our git repo, we just have a way to say “go from no gem environment to everything I need.”

@luis merb.thor includes a task for recompiling binary gems automatically in each environment. .gitignore simply ignores the .bundle, .so, makefile, et al.

@atmos I’ve been talking with halorgium about this and while I think it still has some problems vs. checking everything in (specifically wrt consistency and predictability) I think it makes sense for us to offer that as an option.

@luis && @atmos

Using remote gem servers seems like a recipe for trouble when deploying.

First of all, you now have external server dependencies. If the gem server goes down, you can’t deploy. Having gems in a git repository means that your gems will always be available. Even if you create an internal gem server, that’s that much more complexity to the entire system. Honestly, the last thing you want to happen during a deploy is it all going wrong because gems are not available for install. Imagine you have to deploy a fix really quick because of a bug that got live and deploying explodes… That will be one bad day :). Another advantage with bundling your gems in the git repo is that git is fast. You will be able to deploy MUCH faster since you are only copying over the delta from the last deploy and everything is compressed.

As mentioned earlier, with compiled extensions, you are not bundling the compiled bits, only the source and the extensions will be compiled locally on the target systems.

Forgive my very noob question to merb, but isn’t this just similar to the assembly signing issue commonly found using statically typed languages? We always make sure the builds are fully encapsulated (apart from system dependencies) for third party libraries, etc to avoid conflicts like these and so they will work in any environment.
Am I understanding the deployment difficulties discussed here? I want to get into merb so want to know the best path for deployment.

@atmos && @wycats:

And what about deployment servers you don’t have compilers or enough privileges to compile?

Sorry to be so stubborn or negative, but what happens when these gems requires dependencies that are not under control by the user? All those scenarios high likely will happen.

@carllerche: excellent good point about cannot locate a gem server from deployment one.

I’m still not sold on the idea or the implementation but I think I should not be tried to convince here, time will tell about the right way to do it and this solution could evolve with time.

Thank you!

@luis if you don’t have privileges to compile, nothing stops you from compiling on your development box and shipping the compiled binary, but that honestly seems like a recipe for disaster (except for Windows, where it’s necessary)

Leave a Reply