This week saw a flurry of back-and-forth about the future of the web platform. In the “web sucks” camp were Sachin Agarwal of Posterous (The Web Sucks. Browsers need to innovate) and Joe Hewitt (Sachin summarized some of his tweets at @joehewitt agrees with me).

Chris Blizzard responded with a few narrow examples of what Firefox has been doing lately (geolocation, orientation, WebGL).

On Hacker News, most of the comments took Sachin to task for his argument that standards don’t matter, pointing people back to the “bad old days” of the browser wars.

In my opinion, both camps kind of miss the point.

Sachin made some very pointed factual claims which are just complete, pure hokum. Close to the top of his post, he says:

Web applications don’t have threading, GPU acceleration, drag and drop, copy and paste of rich media, true offline access, or persistence. Are you kidding me?

Really? Are you kidding me. In fact, browsers have implemented, and the WHAT-WG has been busily standardizing, all of these things for quite a number of years now. Threading? Check. GPU acceleration? Check and check. Drag and drop? Check. Offline access and persistence? Check and check.

Almost universally, these features, which exist in shipping browsers today, were based on experiments conducted years ago by browsers (notably Firefox and Safari, more recently Chrome), and did not wait for approval from the standards bodies to begin implementing.

In fact, large swaths of HTML5 are based on proposals from browsers, who have either already implemented the feature (CSS-based animations and transitions) or who then build the feature without waiting for final approval. And this is transparently obvious to anyone, since HTML5 has not not yet been approved, and some parts are the subject of internal W3C debate, yet the browsers are implementing the parts they like and innovating in other areas.

You can see this explicitly in the features prefixed with -moz or -webkit, because they’re going ahead and implementing features that have not gotten consensus yet.

In 2010, the WHAT-WG is a functioning place to bring a proposal for further review once you’re conducting some experiments or have a working implementation. Nobody is sitting on their hands waiting for final approval of HTML5. Don’t confuse the fact that people are submitting their ideas to the standards bodies with the misguided idea that only approved standards are being implemented.

And in fact, this is basically never how it’s worked. Apple implemented the <canvas> tag and <input type="search" /> in 2004 (here’s a 2004 rant from the editor of the then-brand-new HTML5 spec). Opera and Apple worked on the <video> tag in 2007. The new CSS flexible box model is based on XUL, which Firefox has had for over a decade. HTML5 drag and drop support is based on implementations shipped by the IE team over a decade ago. Firefox has been extending JavaScript for almost its entire history (most recently with JavaScript 1.8).

CSS transition, transforms and animations were built by Apple for the iPhone before they were a spec, and only later ported to the desktop. Firefox did the initial experimentation on WebGL in 2006, back when they were calling it Canvas 3d (here‘s a working add-on by Mozilla developer Vladimir Vukićević).

Google Gears shipped what would become the Application Cache, Web Storage, and Web Workers, proving out the concepts. It also shipped geolocation, which is now part of the HTML5 spec.

@font-face originally shipped in Internet Explorer 5.5. Apple has added new mobile events (touchstart, touchmove, touchend, onorientationchange) with accompanying JavaScript APIs (Apple developer account needed) without any spec activity that I can discern. Firefox, at the same time, added support for hardware accelerometers.

Even Microsoft bucked the standards bodies by shipping its own implementation of the W3C’s “cross-origin-resource-sharing” spec called Cross domain request, and shipped it in IE8.

It’s perfectly understandable that people who haven’t been following along for the past few years might have missed all of this. But the fact remains that browser vendors have been moving at a very rapid clip, implementing all kinds of interesting features, and then sharing them with other browsers for feedback, and, one hopes, consensus.

Some of these implementations suck (I’ve heard some people say not-nice things about WebGL and drag-and-drop, for instance). But that’s quite a bit different from saying that browsers are sitting on their hands waiting for the W3C or WHAT-WG to tell them what to do.

In the beginning, Rubygems made a decision to allow multiple versions of individual gems in the system repository of gems. This allowed people to use whatever versions of gems they needed for individual scripts, without having to partition the gems for specific purposes.

This was a nice starting place. Being able to just install whatever gems into one place and have scripts just work, without having to partition your gems into buckets made Rubygems an extremely pleasant tool to work with. In my opinion, it was a good decision.

As the ecosystem has matured, and more people build gems with dependencies (especially loose dependencies, like “rack”, “>= 1.0″), this original sin has introduced the dreaded activation error:

can't activate rspec-rails (= 1.3.2, runtime) for [],
already activated rspec-rails-2.0.0.beta.6

This error occurs because of the linear way that gems are “activated”. For instance, consider the following scenario:

# On your system:
thin (1.2.7)
- daemons (>= 1.0.9)
- eventmachine (>= 0.12.6)
- rack (>= 1.0.0)

actionpack (2.3.5)
- activesupport (= 2.3.5)
- rack (~> 1.0.0)

rack (1.0.0)
rack (1.1.0)
activesupport (2.3.5)
daemons (1.0.9)
eventmachine (0.12.6)

Quickly glancing at these dependencies, you can see that these two gems are “compatible”. The gems have the rack dependency in common, and the >= 1.0.0 is compatible with ~> 1.0.0.

However, there are two ways to require these gems. First, you could require actionpack first.

require "action_pack"
# will result in "activating" ActiveSupport 2.3.5 and Rack 1.0.0
 
require "thin"
# will notice that Rack 1.0.0 is already activated, and satisfies
# the >= 1.0.0 requirement, so just activates daemons and eventmachine

Second, you could require thin first.

require "thin"
# will result in "activating" Rack 1.1.0, Daemons 1.0.9, 
# and EventMachine 0.12.6
 
require "action_pack"
# will notice that Rack 1.1.0 is already activated, but that
# it is incompatible with ~> 1.0.0. Therefore, it will emit:
# ---
# can't activate rack (~> 1.0.0, runtime) for ["actionpack-2.3.5"], 
# already activated rack-1.1.0 for ["thin-1.2.7"]

In this case, because thin pulled in Rack 1.1, it becomes impossible to load in actionpack, despite the fact that a potentially valid combination exists.

This problem is fundamental to the approach of supporting different versions of the same gem in the system and activating gems linearly. In other words, because no single entity ever has the opportunity to examine the entire list of dependencies, the onus is on the user to make sure that the gems requires are ordered correctly. Sometimes, it means that the user must explicitly require the right version of a child dependency just to make sure that the right versions are loaded.

There are two possible solutions to this problem.

Multiple Named Environments

One solution to this problem is to catch potential conflicts at installation time and ask the user to manage multiple named environments, each with a fully consistent, non-conflicting view of the world.

The best way to implement this looks something like this (I’ll use a fictitious gemenv command to illustrate):

$ gemenv install thin
- Installing daemons (1.0.10)
- Installing eventmachine (0.12.10)
- Installing rack (1.1.0)
- Installing thin (1.2.7)

$ gemenv install actionpack -v 2.3.5
- Uninstalling rack (1.1.0)
- Installing rack (1.0.1)
- Installing actionpack (2.3.5)

$ gemenv install rack -v 1.1.0
- rack (1.1.0) conflicts with actionpack (2.3.5)
- if you want rack (1.1.0) and actionpack (2.3.5)
  you will need a new environment.

$ gemenv create rails3
$ gemenv install actionpack -v 3.0.0.beta3
- Installing abstract (1.0.0)
- Installing builder (2.1.2)
- Installing i18n (0.3.7)
- Installing memcache-client (1.8.2)
- Installing tzinfo (0.3.20)
- Installing activesupport (3.0.0.beta3)
- Installing activemodel (3.0.0.beta3)
- Installing erubis (2.6.5)
- Installing rack (1.1.0)
- Installing rack-mount (0.6.3)
- Installing rack-test (0.5.3)
- Installing actionpack (3.0.0.beta3)

$ gemenv use default
$ ruby -e "puts Rack::VERSION"
1.0.1
$ gemenv use rails3
$ ruby -e "puts Rack::VERSION"
1.1.0

Essentially, the single entity with full knowledge of all dependencies is the installer, and the user creates as many environments as he or she needs for the various non-conflicting sets of gems in use.

This works nicely, because it guarantees that once using an environment, all gems available are compatible. Note that the above command is fictitious, but it bears similarity to rip.

Virtual, Anonymous Environments

Another solution, the one bundler uses, is to allow multiple, conflicting versions of gems to exist in the system repository of packages, but to ensure a lack of conflicts by resolving the dependencies used by individual applications.

First, install conflicting gems. In this case, Rails 2.3 and Rails 3.0 require different, incompatible versions of Rack (1.0.x and 1.1.x).

$ gem install rails
... output ...
$ gem install rails -v 3.0.0.beta3
... output ...

Next, specify which version of Rails to use in your application’s Gemfile:

gem "rails", "2.3.5"

When running script/server in an app using Bundler, Bundler can determine that Rails needs Rack 1.0.x, and pulls in Rack 1.0.1 from the system. If you had specified gem "rails", "3.0.0.beta3", bundler would have pulled in Rack 1.1.0 from the system.

In essence, we have the same kind of isolation as the fictitious command above, but instead of manually managing named environments, Bundler creates virtual isolated environments based on the list of gems used in your application.

Why Did We Use Virtual, Anonymous Environments?

When considering the tradeoffs between these two solutions, we realized that applications already typically have a list (executable or not) of its dependencies. The gem install command already works great for installing dependencies, and introducing a dependency resolution step there feels awkward and out of place.

Additionally, as an application evolves, it’s natural to continue updating its list of gems, keeping a record of the changes as you go. You could keep a separate named environment for each application, but you’d probably want to keep a list of the dependencies in the application anyway so that it’s possible to get up and running on a second machine.

In short, since a list of an application’s dependencies makes sense anyway, why burden the end-user with the need to maintain separate named environments and manually install gems.

And once we’re doing this, why not build a toolchain around installing gems and keeping records of not only the top-level installed gems, but the exact versions of all gems used at a given time? Why not build in support for “edge” gems on your local machine or in git repositories? Why not create workflows for sharing your application with other developers and deploying your application?

As application and gem developers ourselves, we wanted a tool that managed application’s dependencies across the lifecycle of an application, in the context of an application.

That said, there is absolutely room for experimentation in this space. Tools that enforce consistency at install time might be extremely appropriate for managing the gems that you use in scripts, but which you don’t share with other machines. Context matters.

Postscript

We commonly hear something to the effect of “but why add all this complexity? Without all these features, bundler could be so much smaller!”. The truth is that the bundler code itself is under 2,000 lines of code, and hasn’t grown a whole lot in the past few major revisions.

The new rpg tool, recently released by the venerable Ryan Tomayko, is also in that range. The original rip (currently in the “classic” branch) was also in that ballpark. The new rip (the current master branch), may well demonstrate that a fully-featured dependency management system for Ruby can be written in many fewer lines of code. If so, I’m excited to see the abstractions that they use to make it so.

But the bottom line is that all of the new package management solutions have done a good job of packing features into a small number of new lines of code, bundler included. By starting with good abstractions, it’s often possible to add rich new features without having to add a whole lot of new code (or even by deleting code!). We’ve certainly found that in our journey with bundler.

Bundler has inadvertantly exposed a number of require order issues in existing gems. I figured I’d take the opportunity to talk about them. There are basically two kinds of gem ordering issues:

Missing Requires

Imagine a gem that uses nokogiri, but never requires it. Instead, it assumes that something that is required before it will do the requiring. This happens relatively often with Rails plugins (which were previously able to assume that they were loaded at a particular time and place).

A well-known example of this problem is gems that didn’t require “yaml” because Rubygems required it. When Rubygems removed its require in 1.3.6, a number of gems broke. In general, bundler assumes that gems require what they need. If gems do so, this class of ordering problems is eliminated.

Constant Definition as API

This is where a gem checked for defined?(SomeConstant) to decide whether to activate some functionality.

This is a bit tricky. Essentially, the gem is saying that the API for using it is “require some other gem before me”. Personally, I consider that to be a problematic API, because it’s very implicit, and can be hard to track down exactly who did what require.

A better solution is to provide an explicit hook for people to activate the optional functionality. For instance, Haml
provides: Haml.init_rails(binding) which you can run after activating Haml.

This is slightly more manual than some would like, but the API of “make sure you require the optional dependencies before me” is also manual, and more error-prone.

Even if Bundler “respected require order”, which we plan to do (in some way) in 0.10, it’s still up to the user of the gem to ensure that they listed the optional gem above the required gem. This is not ideal.

A workaround that works great in Bundler 0.9 is to simply require the order-dependent gems above Bundler.require. We do this in Rails, so that gems can test for the existence of the Rails constant to decide whether to add optional Rails dependencies.

require "rails/all" 
Bundler.require(:default, Rails.env)

In the case of shoulda and mocha, a better solution could be:

# Gemfile 
group :test do 
  gem "shoulda" 
  gem "mocha" 
  # possible other gems where order doesn't matter 
end
 
# application.rb 
Bundler.require(:default) 
Bundler.require(Rails.env) unless Rails.env.test? 
 
# test_helper.rb 
# Since the order matters, require these gems manually, in the right order 
require "shoulda" 
Bundler.require(:test)

In my opinion, if you have gems that specifically depend on other gems, it is appropriate to manually require them first before automatically requiring things using Bundler.require. You should treat Bundler.require as a shortcut
for listing out a stack of requires.

Possible solutions?

One long-term solution, if we get gem metadata in Rubygems 1.4 (or optional dependencies down the line), would be to specify that a gem has optional dependencies on another gem and specify a file to run both gems are
available.

For instance, the Haml gem could say:

# gemspec 
s.integrates_with "rails", "~> 3.0.0.beta2", "haml/rails" 
 
# haml/rails.rb 
require "haml" 
require "rails" 
Haml.init_rails(binding)

Bundler would handle this in Bundler.require (or possibly Bundler.setup).

If this feature existed in Rubygems, it would work via gem activation. If the Haml gem was activated after the Rails gem, it would require “haml/rails” immediately.

If the Haml gem was activated otherwise, it wouldn’t do anything until the Rails gem was activated. When the Rails gem was activated, it would require the file.

When you clone a repository containing a Unix tool (or download a tarball), there’s a standard way to install it. This is expected to work without any other dependencies, on all machines where the tool is supported.

$ autoconf
$ ./configure
$ make
$ sudo make install

This provides a standard way to download, build and install Unix tools. In Ruby, we have a similar (little-known) standard:

$ gem build gem_name.gemspec
$ gem install gem_name-version.gem

If you opt-into this convention, not only will it simplify the install process for your users, but it will make it possible for bundler (and other future automated tools) to build and install your gem (including binaries, proper load path handling and compilation of C extensions) from a local path or git repository.

What to Do

Create a .gemspec in the root of your repository and check it in.

Feel free to use dynamic code in here. When your gem is built, Rubygems will run that code and create a static representation. This means it’s fine to pull your gem’s version or other shared details out of your library itself. Do not, however, use other libraries or dependencies.

You can also use Dir[] in your .gemspec to get a list of files (and remove files you don’t want with -; see the example below).

Here’s bundler’s .gemspec:

# -*- encoding: utf-8 -*-
lib = File.expand_path('../lib/', __FILE__)
$:.unshift lib unless $:.include?(lib)
 
require 'bundler/version'
 
Gem::Specification.new do |s|
  s.name        = "bundler"
  s.version     = Bundler::VERSION
  s.platform    = Gem::Platform::RUBY
  s.authors     = ["Carl Lerche", "Yehuda Katz", "André Arko"]
  s.email       = ["carlhuda@engineyard.com"]
  s.homepage    = "http://github.com/carlhuda/bundler"
  s.summary     = "The best way to manage your application's dependencies"
  s.description = "Bundler manages an application's dependencies through its entire life, across many machines, systematically and repeatably"
 
  s.required_rubygems_version = ">= 1.3.6"
  s.rubyforge_project         = "bundler"
 
  s.add_development_dependency "rspec"
 
  s.files        = Dir.glob("{bin,lib}/**/*") + %w(LICENSE README.md ROADMAP.md CHANGELOG.md)
  s.executables  = ['bundle']
  s.require_path = 'lib'
end

If you didn’t already know this, the DSL for gem specifications is already pretty clean and straight-forward, there is no need to generate your gemspec using alternative tools.

Your gemspec should run standalone, ideally with no additional dependencies. You can assume its __FILE__ is located in the root of your project.

When it comes time to build your gem, use gem build.

$ gem build bundler.gemspec

This will spit out a .gem file properly named with a static version of the gem specification inside, having resolved things like Dir.glob calls and the version you might have pulled in from your library.

Next, you can push your gem to Rubygems.org quickly and painlessly:

$ gem push bundler-0.9.15.gem

If you’ve already provided credentials, you’ve now published your gem. If not, you will be asked for your credentials (once per machine).

You can easily automate this process using Rake:

$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
require "bundler/version"
 
task :build do
  system "gem build bundler.gemspec"
end
 
task :release => :build do
  system "gem push bundler-#{Bunder::VERSION}"
end

Using tools that are built into Ruby and Rubygems creates a more streamlined, conventional experience for all involved. Instead of trying to figure out what command to run to create a gem, expect to be able to run gem build mygem.gemspec.

A nice side-effect of this is that those who check in valid .gemspec files can take advantage of tools like bundler that allow git repositories to stand in for gems. By using the gem build convention, bundler is able to generate binaries and compile C extensions from local paths or git repositories in a conventional, repeatable way.

Try it. You’ll like it.