Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Other’ Category

Here’s to the Next 3 Years

I officially joined Engine Yard on January 1, 2008, about a week before we announced our Series A funding, becoming its twenty-second employee. I was Engine Yard’s very first “Engineering” hire, and I would spend the next year working on Ezra’s Merb project, finally releasing Merb 1.0 at MerbCamp that October.

When I joined Engine Yard, I had already been working on the jQuery project for a couple years, having started visualjquery.com at a time in jQuery’s history before jQuery had version numbers. I learned a lot from John Resig about building and motivating open source communities, and I did my first professional work writing jQuery in Action.

The Merb project was my first full-time open source work, and I met some of my earliest open source friends while working on it. When I first met Carl, he was working on a rewrite of the Merb router, the one part of Merb I was still scared to work on. I met Andy Delcambre, who now works on Engine Yard’s AppCloud, after he volunteered to help with Merb.

I met Loren Segal, well-known for the YARD documentation tool, while working to give Merb documentation I could be proud of. And I became the first Github user outside of Chris, PJ and Tom in order to get the Merb project on git. If you’re paying attention, that made me Github user nil.id, and was the source of some hilarious bugs.

In Hampton Catlin’s first Ruby survey in 2008, about one in eight respondents said that they preferred Merb, and by the end of the year, the Rails and Merb teams were coming to blows. During this same time, Engine Yard had taken on a Series B round of funding (this time for $15 million), and had grown from a small company of a couple dozen to a company pushing 100. We had Sales and Marketing departments. We also had an Engineering department, which had spent a large part of the year on our first fully automated product, Engine Yard Solo.

On a personal level, I started getting heavily involved in Open Source other than Merb, especially DataMapper and Rubinius. I started speaking at conferences, starting with a talk on DataMapper at MountainWest RubyConf in 2008. Engine Yard also threw its first Hackfest at that event, an event we would repeat in 2009 and again in 2010. This event, and the work our great community marketing, Open Source and support teams would do over the next several years elevated Engine Yard’s reputation as a team of experts that would keep your app running no matter what.

As Engine Yard grew, a lot of people asked us why we were supporting Merb when so much of our business depended on the success of the Rails community. By late 2008, it was probably the most common question I got when speaking at conferences. In truth, the best I could tell was that Engine Yard was worried about the future of Rails, and wanted to make sure that Ruby remained successful. What we didn’t realize at the time was that the one full-time employee (me) that Engine Yard had on Merb was more than the total number of worldwide full-time developers on Rails. In short, Rails was a fully volunteer project, and that had something to do with some of the issues that drove people to Merb in the first place.

At the end of 2008, Tom Mornini, the CTO of Engine Yard, recognized that if we could join forces (and this was definitely a big if), we could help shore up Rails into the future. I remember the first time Tom asked me about it in the stairwell of our South Park office. I thought he was off the wall to even suggest it. After all, it didn’t seem like the teams could get along at all. But the Merb team had adopted so much of the Rails philosophy that at its core, the only question that remained was how aggressive the Rails project would get to clean things up.

In the years since Ezra first announced Merb, Rails had gotten thread-safe, Rack support, and some improvements to the internals. After hashing out the details (in some very contentious conversations), we decided to give it a try, announcing we would merge the projects on December 23, 2008. That announcement quite literally changed the trajectory of my life. In the almost two years since that announcement, I have worked harder than I have ever worked before, and am more proud of the work I have done than I have ever been before.

Perhaps more importantly, I have worked with some of the best open source developers in the business without whom Rails 3 would have been a pale shadow of itself. Including me, Rails 3 brought on eight new committers. The Rails 3 project did more than improve the quality of the codebase: it reinvigorated the community of developers actively contributing. And by moving toward a model that allowed Rails to have more dependencies, Rails became the center of a thriving Ruby community, where anyone can reuse the code that makes Rails tick for their non-Rails projects.

In the summer of 2009, José Valim became a Google Summer of Code student dedicated to rewriting the Rails generators so that users of DataMapper, RSpec, and Haml could use the standard Rails generators. A lot of people thought it was too optimistic to do in a summer, but José finished the job well before the end of the summer, giving him time to continue his contributions.

Around the same time, Mikel Lindsaar decided to write a ground-up mail implementation, releasing the imaginatively named mail gem in October 2009. This project, which involved writing a parser that could parse the gigabytes of mail in the Trec and Enron email databases, strained credulity, and would enable us to give ActionMailer the facelift it needed for Rails 3.0.

Santiago Pastorino showed up out of nowhere, with a level of pluck that is rarely seen in the Ruby community. Despite a noticeable language barrier, Santiago went from a warning fixer to a Rails committer. This month, he became an official member of the Rails core team. While working on Rails, I learned a lot from a lot of people, but Santiago’s raw determination and skill should stand out as an example to the Ruby community.

Everyone knows Aaron Patterson, and I was just as surprised as everyone else when he started the long, dedicated task of improving the performance of the Arel library. He already has quite the reputation in the Ruby community, and even has commit access to Ruby itself, so I’m personally gratified to work with him on Rails. If you haven’t yet watched it, watch his Worst Ideas Ever presentation from RubyConf last year. Be prepared to laugh.

Xavier Noria has been a one-man documentation machine, helping to grow and maintain the Rails guides, and constantly reminds everyone that documentation is just as essential to the codebase as tests. Without him, Rails 3 might have been great internally, but nobody would know it. Documentation folks often get little love in open source projects, but to me, the work Xavier does is key to our success.

More recently, I’ve been extremely impressed by Piotr Sarnacki’s work (for Ruby Summer of Code) on finally delivering on the promise of truly mountable engines. His work is already merged into Rails master, and will ship with Rails 3.1. As with the rest of the guys I’ve talked about, he took on a pretty complicated and entangled task, and managed to get more done in less time than people expected. Keep an eye on him!

In short, the Rails 3 project was about more than code; it was about reshaping the Rails community. I’m proud to have been a part of growing the Rails ecosystem, and humbled by the number of people who have worked so hard for so little.

Now that we released Rails 3, I have had an opportunity to think about what to do next. I could, of course, continue full-time work on Rails for Engine Yard, which has been a fantastic place to do Open Source for the past several years. I am still extremely interested in open web technologies, and want to do my part to help the open web succeed. Having robust, mature server side frameworks that are easy for anyone to use is certainly one large piece of the puzzle, and I’ve really enjoyed helping to bring the core of Rails up to date with modern web technologies.

But as Avi Bryant pointed out last year at DjangoCon, the technology of the open web is shifting toward the client side. That doesn’t change the need for robust, mature server-side frameworks. Even the richest web applications get and sync their data with servers, mostly over the HTTP protocol. And although a lot of people have opined that rich clients can simply wire directly into something like CouchDB or MongoDB, applications with rich clients tend to have the same kinds of complex needs (like authorization) as more tranditional document-based applications have. I see Rails remaining an excellent server-side solution even as clients become richer over the next few years.

There’s still a lot to do for Rails in the future, and I want to be a part of that. On the other hand, the field of rich client development is just getting starting, with new mobile devices like the iPhone, Android and iPad giving us an almost blank canvas to work on. Unfortunately, a lot of the really big bets in this space are on selling licenses to either entire libraries or the development tools needed to efficiently work with the tools.

What the open web needs is a robust, easy to use framework with development tools that are trivial to get started with and awesome to work with. And both the library and the developer tools need to be offered for free, like Rails or jQuery. Since 2008, I have poured my soul into helping to build an amazing open source Ruby web framework and the ecosystem that powers it. Now, I want to bring that same energy and enthusiasm to building great tools that leverage the power of the modern open web and a free and open source community around them.

That’s why I was so excited when I saw that Charles Jolley, who worked on Apple’s Mobile Me product, announced that he was leaving Apple to form a new company dedicated to those same principles. Charles had helped build Mobile Me on SproutCore, a free and open source framework, and had managed to make sure that Apple continued to contribute back to the open source project under the MIT license.

In his blog post announcing that he was leaving Apple, Charles said “SproutCore is now and will always be totally free and open source. I think this business of charging for a commercial license is not an effective way to grow a project. Sure you make a little cash, but at what expense to the community? My goal is to make SproutCore and all of the developer tools that surround it totally free to everyone. All I ask is that you participate in the community somehow to make things a little better for those who come after you.”

This philosophy closely mirrored my thinking on the future of client-side development, and seeing a mature client-side framework open a permissive open source philosophy really excited me. Since the beginning of 2010, I was starting to think that if I wanted a permissively licensed client-side toolkit, I was going to have to build it and its community myself. I already knew about SproutCore–early versions of its build tools were built on Merb, and had met Charles a few times. After reading his blog post, I emailed him to find out what he was thinking about doing and he told me about his plans for SproutCore.

The SproutCore library itself, today, is definitely recovering from having been developed almost exclusively inside Apple. It comes with a number of amazing low-level primitives, and you can build a large, complex application on it. That said, it never had a big enough new user community to get a very friendly public API. Thankfully, it’s much, much easier to build a great public API than it is to build the core of a web application toolkit. Starting this week, I will be joining Charles’ company, Strobe, and will get right to work on the next generation of the SproutCore tools.

I have to say, I’m excited about what comes next.

Postscript

Because you’re probably wondering, I will still be active in the Rails community, and will be doing work for Engine Yard on Rails 3.1. I plan to use Rails 3 and Engine Yard for the server-side elements of the projects I will be doing, and really want the new Rails 3.1 features I talked about at Windy City Rails. I will also be helping Engine Yard start a new Technical Advisory Board, which will help provide community feedback for its cloud products, and guidance about what open source efforts could use the Engine Yard bump. I forsee being part of both the Rails and Engine Yard family for years to come.

Automatic Flushing: The Rails 3.1 Plan

preamble: this post explains, in some detail, how we will implement a nice performance boost for Rails developers. Understanding the details might help gain the full benefits of the optimization, but you will gain some benefits even if you have no idea how it works.

As you’ve probably seen, DHH announced that we’d be looking at flushing in Rails 3.1 to improve the client-side performance of typical Rails applications.

The most obvious solution, and one that already exists in plugin form, is to allow a layout to have a new flush method, which would immediately flush the contents of the layout to the browser. By putting the flush method below the JavaScript and CSS includes, the browser could begin downloading and evaluating those static assets while the server continues building the page.

Unfortunately, this solution has a major problem: it requires a fairly significant change in the current model of how people build applications. In general, for performance optimizations (including client-side optimizations), we like to make the default as fast as possible, without asking people to understand a brand new paradigm, centered around the optimization.

The problem lies in the fact that a Rails layout is essentially a template with a bunch of holes to fill in.

<html>
  <head>
    <title><%= yield :title %></title>
    <%= javascript_include_tag :defaults %>
    <%= yield :extra_javascripts %>
    <%= stylesheet_link_tag :defaults %>
    <%= yield :extra_stylesheets %>
  </head>
  <body>
    <%= yield :sidebar %>
    <%= yield %>
  </body>
</html>

I this simple example, each yield is a slot that is filled in by the template (usually via content_for). In order to achieve this, Rails evaluates the template first, which populates a Hash with each piece of content. Next, it renders the layout, and each yield checks the Hash for that content. In short, because of the way layouts work, Rails renders the template first, and then the layout.

To get around this, one option would be to say that everything before the flush must not use yield, and must be able to run before the template. Unfortunately, it’s somewhat common for people to set up a content_for(:javascripts) in a template, to keep the JavaScript needed for a particular snippet of HTML close to the HTML. This means that not only does the user have to be careful about what can go above and below the flush, he can no longer use content_for for things high up in the template, which is a fairly significant change to the overall design of Rails applications.

For Rails 3.1, we wanted a mostly-compatible solution with the same programmer benefits as the existing model, but with all the benefits of automatic flushing. After a number of very long discussions on the topic, José Valim came up with the idea of using Ruby 1.9 fibers to jump back and forth between the template and layout.

Let’s start by taking a look at a very simplified version of the current Rails rendering pipeline. First, we set up a Buffer object purely for logging purposes, so we can see what’s happening as we push things onto the buffer.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
    end
 
    def <<(value)
      super
 
      puts "#{@name} is pushing #{value.inspect}"
    end
  end
end

Next, we create a simple version of ActionView::Base. We implement the content_for method simply, to print out a bit of logging information and stash the value into the @content_for Hash. Note that the real version is pretty similar, with some added logic for capturing the value of the block from ERB.

module Basic
  class ViewContext
    def initialize
      @buffer      = Buffer.new(:main, self)
      @content_for = {}
    end
 
    def content_for(name, value = nil)
      value = yield if block_given?
      puts "Setting #{name} to #{value.inspect}"
      @content_for[name] = value
    end
 
    def read_content(name)
      @content_for[name]
    end
  end
end

Next, we create a number of methods on the ViewContext that look like compiled ERB templates. In real life, the ERB (or Haml) compiler would define these methods.

module Basic
  class ViewContext
    def layout
      @buffer << "<html><head>"
      @buffer << yield(:javascripts).to_s
      @buffer << yield(:stylesheets).to_s
      @buffer << "</head><body>"
      @buffer << yield.to_s
      @buffer << yield(:not_existant).to_s
      @buffer << "</body></html>"
      @buffer
    end
 
    def template
      buffer =  Buffer.new(:template, self)
      content_for(:javascripts) do
        "<script src='application.js'></script>"
      end
      content_for(:stylesheets) do
        "<link href='application.css' rel='stylesheet' />"
      end
      puts "Making a SQL call"
      sleep 1 # Emulate a slow SQL call
      buffer << "Hello world!"
      content_for(:body, buffer)
    end
  end
end

Finally, we define the basic rendering logic:

module Basic
  class ViewContext
    def render
      template
      layout { |value| read_content(value || :body) }
    end
  end
end

As you can see, we first render the template, which will fill up the @content_for Hash, and then call the layout method, with a block which pulls the value from that Hash. This is how yield :javascripts in a layout works.

Unfortunately, this means that the entire template must be rendered first, including the (fake) slow SQL query. We’d prefer to flush the buffer after the JavaScripts and CSS are determined, but before the SQL query is made. Unfortunately, that requires running half of the template method, then continuing with the layout method, retaining the ability to resume the template method later.

You can think of the way that templates are currently rendered (in Rails 2.x and 3.0) like this:

flush.001.png

Unfortunately, this makes it very hard to get any more performance juice out without asking the end-developer to make some hard choices. The solution we came up with is to use Ruby 1.9 fibers to allow the rendering to jump back and forth between the template and layout.

flush.002.png

Instead of starting with the template and only rendering the layout when ready, we’ll start with the layout, and jump over to the template when a yield is called. Once the content_for that piece is provided by the template, we can jump back to the layout, flush, and continue rendering. As we need more pieces, we can jump back and forth between the template and layout, flushing as we fill in the holes specified by the yield statements.

The implementation is mostly straight-forward:

require "fiber"
 
module Fibered
  class ViewContext < Basic::ViewContext
    def initialize
      super
      @waiting_for = nil
      @fiber       = nil
    end
 
    def content_for(name, value = nil)
      super
      @fiber.resume if @waiting_for == name
    end
 
    def read_content(name)
      content = super
      return content if content
 
      begin
        @waiting_for = name
        Fiber.yield
      ensure
        @waiting_for = nil
      end
 
      super
    end
 
    def layout
      @fiber = Fiber.new do
        super
      end
      @fiber.resume
      @buffer
    end
 
    def render
      layout { |value| read_content(value || :body) }
      template
      @fiber.resume while @fiber.alive?
      @buffer
    end
  end
end

For our fibered implementation, we’ll inherit from Basic::ViewContext, because we want to be able to use the same templates as we used in the original implementation. We update the content_for, read_content, layout and render methods to be fiber-aware. Let’s take them one at a time.

def layout
  @fiber = Fiber.new do
    super
  end
  @fiber.resume
  @buffer
end

First, we wrap the original implementation of layout in a Fiber, and start it right away. Next, we modify the read_content method to become Fiber-aware:

def read_content(name)
  content = super
  return content if content
 
  begin
    @waiting_for = name
    Fiber.yield
  ensure
    @waiting_for = nil
  end
 
  super
end

If the @content_for Hash already has the content, return it right away. Otherwise, say that we’re waiting for the key in question, and yield out of the Fiber. We modify the render method so that the layout is rendered first, followed by the template. As a result, yielding out of the layout will start the template’s rendering.

def render
  layout { |value| read_content(value || :body) }
  template
  @fiber.resume while @fiber.alive?
  @buffer
end

Next, modify the content_for method so that when the content we’re waiting for is provided, we jump back into the layout.

def content_for(name, value = nil)
  super
  @fiber.resume if @waiting_for == name
end

With this setup, the layout and template will ping-pong back and forth, with the layout requesting data, and the template rendering only as far as it needs to go to provide the data requested.

Finally, let’s update the Buffer to take our fibered implementation into consideration.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
      @fibered = context.fibered?
    end
 
    def <<(value)
      super
 
      if @fibered
        puts "Flushing #{value.inspect}" if @fibered
      else
        puts "#{@name} is pushing #{value.inspect}"
      end
    end
  end
 
  class ViewContext
    def fibered?
      false
    end
  end
end
 
module Fibered
  class ViewContext
    def fibered?
      true
    end
  end
end

Now that we’re rendering the layout in order, we can flush as we go, instead of being forced to wait for the entire template to render before we can start flushing.

It’s worth mentioning that optimal flushing performance will be based on the order of the content_for in your template. If you run your queries first, then put the expensive template rendering, and only finally do the content_for(:javascript) at the end, the flushing behavior will look like this:

flush.003.png

Instead of flushing quickly, before the SQL call, things are barely better than they are in Rails 2.3, when the entire template must be rendered before the first flush. Because things are no worse, even in the worst-case scenario, we can make this the default behavior. Most people will see some benefit from it, and people interested in the best performance can order their content_for blocks so they cause the most beneficial flushing.

Even for people willing to put in the effort, this API is better than forcing a manual flush, because you can still put your content_for blocks alongside the templates that they are related to.

Look for this feature in Rails 3.1!

Small Caveat

For the purposes of this simplified example, I assumed that content_for can only be run once, immediately setting the value in the @content_for Hash. However, in some cases, people want to accumulate a String for a particular value. Obviously, we won’t be able to flush until the full String for that value is accumulated.

As a result, we’ll be adding a new API (likely called provide), which will behave exactly the same as content_for, but without the ability to accumulate. In the vast majority of cases, people will want to use provide (for instance, provide :sidebar), and get all the benefits of autoflushing. In a few cases, people will want to be able to continue accumulating a String, and will still be able to use content_for, but the template will not return control to the layout when that happens.

Also note that this (fairly detailed) explanation is not something that you will need to understand as a Rails developer. Instead, you will continue to go about your development as before, using provide if you have just a single piece of content to add, and content_for if you have multiple pieces of content to add, and Rails will automatically optimize flushing for you as well as we can.

My Common Git Workflow

A recent post that was highly ranked on Hacker News complained about common git workflows causing him serious pain. While I won’t get into the merit of his user experience complaints, I do want to talk about his specific use-case and how I personally work with it in git.

Best I can tell, Mike Taylor (the guy in the post) either tried to figure out a standard git workflow on his own, or he followed poor instructions that tried to bootstrap someone from an svn background while intentionally leaving out important information. In any event, I’ll step through my personal workflow for his scenario, contrasting with subversion as I go.

Cloning the Repository

The very first step when working with a repository is to clone it. In subversion, this is accomplished via svn checkout svn://url/to/repo/trunk. This retrieves the most recent revision of the trunk branch of the repository.

In git, this is accomplished via git clone git://url/to/repo (the http protocol is also possible). This retrieves the entire repository, including other branches and tags.

Making the Change

In both git and subversion, you make the change using a normal text editor.

After Making the Change

In git, you make a local commit, marking the difference between the most recent pulled version (master) and the changes you made. In subversion, the normal workflow does not involve making a change, but apparently some people make manual diffs in order to have a local copy of the changes before updating from the remote. Here’s an example comment from the Hacker News post:

I’ll tell you what happens when I use svn and there’s been an upstream change: I never update my local tree with local modifications. Instead, I extract all my local changes into a diff, then I update my local tree, and then I merge my diff back into the updated tree and commit.

When I need three-way merging, which isn’t often – usually patch can resync simple things like line offsets – it’s handled by a file comparison tool. I have a simple script which handles this

My personal process for making the commit in git almost always involves the gitx GUI, which lets me see the changes for each individual file, select the files (or chunks in the files) to commit, and then commit the whole thing. I sometimes break up the changes into several granular commits, if appropriate.

Updating from the remote

Now that we have our local changes, the next step is to update from the remote. In subversion, you would run svn up. Here, subversion will apply a merge strategy to attempt to merge the remote changes with the local ones that you made. If a merge was unsuccessful, subversion will tell you that a conflict has occurred. If you did not manually save off a diff file, there is no way to get back to the status from before you made the change.

In git, you would run git pull. By default, git applies the “recursive” strategy, which tries to merge your current files with the remote files at the most recent revision. As with subversion, this can result in a conflict. You can also pass the --rebase flag to pull, which is how I usually work. This tells git to stash away your commits, pull the remote changes, and then reapply your changes on top one at a time.

If you use --rebase, you may get a conflict for each of your local commits, which is usually easier to handle than a bunch of conflicts all at once.

I definitely recommend using --rebase which also provides instructions for dealing with conflicts as they arise.

In either case, in my experience, git’s merging capabilities are more advanced than subversion’s. This will result in many fewer cases where conflicts occur.

Resolving Conflicts

From here on, I am assuming you followed my advice and used git pull --rebase.

If a conflict has occurred, you will find that if you run git status, all of the non-conflicting files are already listed as “staged”, while the conflicting files are listed outside the staging area. This means that the non-conflicting files are already considered “added” to the current commit.

To resolve the conflicts, fix up the files listed outside the staging area and git add them. Again, I normally use gitx to move the resolved files into the staging area.

Once you have resolved the conflict, run git rebase --continue. This tells git to use the fixed up changes you just made instead of the original commit it was trying to put on top of the changes you got from the remote.

In subversion, if you got a conflict, subversion will create three files for you: file.mine, file.rOLD, and file.rNEW. You are responsible for fixing up the conflicts and getting back a working file. Once you are done, you run svn resolved.

NOTE: If you had not used git pull --rebase, but instead did raw git pull, you would fix up the files, add the files using git add or gitx, and the run git commit to seal the deal

Yikes! Something went wrong!

In git, if something goes wrong, you just run git reset --hard, which will bring you back to your last local commit.

In subversion, it’s not always possible unless you manually stored off a diff before you started.

Pushing

Now that you’re in sync with the remote server, you push your changes. In git, you run git push. In subversion, you run svn commit.

One Glossed-Over Difference

Subversion allows you to commit changes even if you haven’t svn uped and there have been changes to the remote, as long as there are no conflicts between your local files and the remote files.

Git never allows you to push changes to the remote if there have been remote changes. I personally prefer the git behavior, but I could see why someone might prefer the subversion behavior. However, I glossed over this difference because every subversion reference I’ve found advises running svn up before a commit, and I personally always did that in my years using subversion.

Comparison

Here’s a workflow comparison between git and subversion:

Operation git svn
Clone a repository git clone git://github.com/rails/rails.git svn checkout http://dev.rubyonrails.org/svn/rails/trunk
Preparing changes git commit (using gitx) nothing or create a manual diff
Update from the remote git pull --rebase svn up
Resolving conflicts git add then git rebase --continue svn resolve
Resolving conflicts without –rebase git add then git commit N/A
Yikes! Rolling back git reset –hard svn up -rOLD then apply diff (only if you manually made a diff first)
Pushing git push svn commit

Note that I am not attempting to provide an exhaustive guide to git here; there are many more git features that are quite useful. Additionally, I personally do a lot of local branching, and prefer to be able to think about git in terms of cheap branches, but the original poster explicitly said that he’d rather not. As a result, I didn’t address that here.

I also don’t believe that thinking of git in terms of subversion is a good idea. That said, the point of this post (and the point of the original poster) is that there are a set of high-level version control operations that you’d expect git to be able to handle in simple cases without a lot of fuss.

The How and Why of Bundler Groups

Since version 0.9, Bundler has had a feature called “groups”. The purpose of this feature is to allow you to specify groups of dependencies which may be used in certain situations, but not in others.

For instance, you may use ActiveMerchant only in production. In this case, you could say:

group :production do
  gem "activemerchant"
end

Specifying groups allows you to do two things. First, you can install the gems in your Gemfile, minus specific groups. For instance, Rails puts mysql and pg in a database group so that if you’re just working on ActionPack, you can bundle install --without db and run the ActionPack tests without having to worry about getting the gems installed.

Second, you can list specific groups to autorequire using Bundler.require. By default, Bundler.require requires all the gems in the default group (which is all the gems that have no explicit group). You can also say Bundler.require(:default, :another_group) to require specific groups.

Note the difference between these operations: bundle install is opt-out, while Bundler.require is opt-in. This is because the common usage of groups is to specify gems for different environments (such as development, test and production) and you shouldn’t need to specify that you want the “development” and “test” gems just to get up and running. On the other hand, you don’t want your test dependencies loaded in development or production.

It is also worth noting that all gems that you installed (i.e. not the ones that you excluded at install time with --without) will be available to require. This has no effect unless you actually require them. This means that in development mode, if you explicitly require rspec, it will work.

Rails 3 defaults to mapping groups to environment names, and explicitly autorequiring the implicit default group and the group named the same as the current environment. For example, in development mode, Rails will require the default group and the development group. The code that does this is in your application.rb:

Bundler.require(:default, Rails.env) if defined?(Bundler)

Consistency

In order to ensure consistency across all environments, bundler resolves the dependencies of your application using the gems listed in all groups, even if you specify --without. This means that while you can skip installing the gems listed in the production group by saying --without production, bundler will still download and examine the gems in order to properly resolve all dependencies.

As a result, the dependencies you install in development mode and test with will be compatible with the gems in other environments. In essence, this policy ensures that if your tests pass and run in development, your app will not fail to run in production because the dependencies resolved differently.

Multiple Inconsistent Configurations

Sometimes, especially when developing gems for wider use, you want to test your code against multiple incompatible configurations. At first glance, you might think that you could use groups for this case, but as described above, groups are designed for cases where all of the gems are compatible, but you don’t always want to have to install them in all situations.

Instead, use multiple Gemfiles, one for each incompatible configuration. When installing, do bundle install --gemfile Gemfile.rails2. This will tell Bundler to use Gemfile.rails2 rather than the default Gemfile. As in all cases in Bundler, you can also specify this option globally with an environment variable (BUNDLE_GEMFILE).

Ruby 1.9 Encodings: A Primer and the Solution for Rails

UPDATE: The DataObjects drivers, which are used in DataMapper, are now updated to honor default_internal. Let’s keep this moving.

Since Ruby 1.9 announced support for encodings, there has been a flurry of activity to make existing libraries encoding aware, and a tornado of confusion as users of Ruby and Rails have tried to make sense of it.

In this post, I will lay out the most common problems people have had, and what we can do as a community to put these issues to bed in time for Ruby 1.9.2 final.

A Quick Tour

I’m going to simplify some of this, but the broad strokes are essentially correct.

Before we begin, many of you are probably wonder what exactly an “encoding” is. For me, getting a handle on this was an important part of helping me understand the possible solution space.

On disk, all Strings are stored as a sequence of bytes. An encoding simply specifies how to take those bytes and convert them into “codepoints”. In some languages, such as English, a “codepoint” is exactly equivalent to “a character”. In most other languages, there is not a one-to-one correspondence. For example, a German codepoint might specify that the next codepoint should get an ümlaut.

The list of English characters represented by the first seven bits of ASCII (characters 0 through 127 in “ASCII-7″) have the same representation in many (but not all) encodings. This means that if you only use English characters, the on-disk representation of the characters will often be exactly the same regardless of the source encoding.

However, once you start to use other characters, the bytes on disk mean different things in different encodings. Have you ever seen a page on the Internet filled with something like “Führer”? That is the consequence of the bytes of “Führer” stored as UTF-8 being interpreted as Latin-1.

You can trivially see this problem using Ruby 1.9′s encoding support by running this program:

# encoding: UTF-8
 
puts "hello ümlaut".force_encoding("ISO-8859-1").encode("UTF-8")
 
# Output
# hello ümlat

First, we create a String (“hello ümlaut”) in the UTF-8 encoding. Next, we tell Ruby that the String is actually Latin-1. It’s not, so an attempt to read the characters will interpret the raw bytes of the “ü” as though they were Latin-1 bytes. We ask Ruby to give us that interpretation of the data in UTF-8 via encode and print it out.

We can see that while the bytes for “hello ” and “mlat” were identical in both UTF-8 and Latin-1, the bytes for “ü” in UTF-8 mean “ü” in Latin-1.

Note that while force_encoding simply tags the String with a different encoding, encode converts the bytes of one encoding into the equivalent bytes of the second. As a result, while force_encoding should almost never be used unless you know for sure that the bytes actually represent the characters you want in the target encoding, encode is relatively safe to use to convert a String into the encoding you want.

You’ve probably also seen the reverse problem, where bytes encoded in Latin-1 ended up inside a page encoded in UTF-8.

# encoding: ISO-8859-1
 
puts "hello ümlaut".force_encoding("UTF-8")
 
# Output
# hello ?mlat

Here, the sequence of bytes that represents an “ü” in Latin-1 could not be recognized in UTF-8, so they were replaced with a “?”. Note that puts will always simply write out the bytes to your terminal, and the terminal’s encoding will determine how they are interpreted. The examples in this post are all outputted to a terminal using UTF-8 encoding.

As you can imagine, this presents quite the issue when concatenating two Strings of different encodings. Simply smashing together the raw bytes of the two Strings can result in output that is incomprehensible in either encoding. To make matters worse, it’s not always possible to represent all of the characters in one encoding in another. For instance, the characters of the Emoji encoding cannot be represented in the ISO-8859-1 encoding (or even in a standardized way onto the UTF-8 encoding).

As a result, when you attempt to concatenate two Strings of different encodings in Ruby 1.9, Ruby displays an error.

# encoding: UTF-8
 
puts "hello ümlaut".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# incompatible character encodings: ISO-8859-1 and UTF-8 (Encoding::CompatibilityError)

Because it’s extremely tricky for Ruby to be sure that it can make a lossless conversion from one encoding to another (Ruby supports almost 100 different encodings), the Ruby core team has decided to raise an exception if two Strings in different encodings are concatenated together.

There is one exception to this rule. If the bytes in one of the two Strings are all under 127 (and therefore valid characters in ASCII-7), and both encodings are compatible with ASCII-7 (meaning that the bytes of ASCII-7 represent exactly the same characters in the other encoding), Ruby will make the conversion without complaining.

# encoding: UTF-8
 
puts "hello umlat".encode("ISO-8859-1") + "hello ümlaut"
 
# Output
# hello umlathello ümlaut

Since Ruby does not allow characters outside of the ASCII-7 range in source files without a declared encoding, this exception eliminates a large number of potential problems that Ruby’s strict concatenation rules might have introduced.

Binary Strings

By default, Strings with no encoding in Ruby are tagged with the ASCII-8BIT encoding, which is an alias for BINARY. Essentially, this is an encoding that simply means “raw bytes here”.

In general, code in Rails applications should not encounter BINARY strings, except for Strings created in source files without encodings. However, since these Strings will virtually always fall under the ASCII-7 exception, Ruby programmers should never have to deal with incompatible encoding exceptions where one of the two encodings is ASCII-8BIT (i.e. BINARY).

That said, almost all of the encoding problems reported by users in the Rails bug tracker involved ASCII-8BIT Strings. How did this happen?

There are two reasons for this.

The first reason is that early on, database drivers generally didn’t properly tag Strings they retrieved from the database with the proper encoding. This involves a manual mapping from the database’s encoding names to Ruby’s encoding names. As a result, it was extremely common from database drivers to return Strings with characters outside of the ASCII-7 range (because the original content was encoded in the database as UTF-8 or ISO-8859-1/Latin-1).

When attempting to concatenate that content onto another UTF-8 string (such as the buffer in an ERB template), Ruby would raise an incompatible encoding exception.

# encoding: UTF-8
 
puts "hello ümlaut" + "hello ümlaut".force_encoding("BINARY")
 
# Output
# incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)

This is essentially identical to the scenario many people encountered. A UTF-8 String was presented to Ruby as a BINARY String, since the database driver didn’t tag it. When attempting to concatenate it onto UTF-8, Ruby had no way to do so reliably, so it raised an exception.

One reason that many people didn’t encounter this problem was that either the contents of the template or the text from the database were entirely in the ASCII-7 subset of their character set. As a result, Ruby would not complain. This is deceptive, because if they made a small change to their template, or if a user happened to enter non-ASCII-7 data (for instance, they got their first user named José), they would suddenly start seeing an incompatible encoding exception.

When people see this incompatible encoding exception, one common reaction is to call force_encoding("UTF-8") on the BINARY data. This will work great for Strings whose bytes actually are encoded in UTF-8. However, if people whose Strings were encoded in ISO-8859-5 (Russian) followed this instruction, they would end up with scrambled output.

Additionally, it’s impossible to simply encode the data, since Ruby doesn’t actually know the source encoding. In essence, a crucial piece of information has been lost at the database driver level.

Unfortunately, this means that well-meaning people who have solved their problem by force_encoding their Strings to UTF-8 (because the bytes actually did represent UTF-8 characters) become baffled when their solution doesn’t work for someone working on a Russian website.

Thankfully, this situation is now mostly solved. There are updates for all database drivers that map the encodings from the database to a Ruby encoding, which means that UTF-8 text from the database will be UTF-8 Strings in Ruby, and Latin-1 text from the database will be ISO-8859-1 Strings in Ruby.

Unfortunately, there is a second large source of BINARY Strings in Ruby. Specifically, data received from the web in the form of URL encoded POST bodies often do not specify the content-type of the content sent from forms.

In many cases, browsers send POST bodies in the encoding of the original document, but not always. In addition, some browsers say that they’re sending content as ISO-8859-1 but actually send it in Windows-1251. There is a long thread on the Rack tracker about this, but the bottom line is that it’s extremely difficult to determine the encoding of a POST body sent from the web.

As a result, Rack handlers send the raw bytes through as BINARY (which is reasonable, since handlers shouldn’t be in the business of trying to wade through this problem) and no middleware exists (yet) to properly tag the String with the correct encoding.

This means that if the stars align, the raw bytes are UTF-8, end up in a UTF-8 database, and end up coming back out again tagged as UTF-8. If the stars do not align, the text might actually be encoded in ISO-8859-1, get put into a UTF-8 database, and come out tagged as UTF-8 (and we know what happens when ISO-8859-1 data is mistakenly tagged as UTF-8).

In this case, because the ISO-8859-1 data is improperly tagged as UTF-8, Ruby happily concatenates it with other UTF-8 Strings, and hilarity ensues.

Because English characters have the same byte representation in all commonly used encodings, this problem is not as common as you might imagine. Unfortunately, this simply means that people who do encounter it are baffled and find it hard to get help. Additionally, this problem doesn’t manifest itself as a hard error. it can go unnoticed and dismissed as a minor annoyance if the number of non-ASCII-7 characters are low.

In order to properly solve this problem for Ruby 1.9, we need a very good heuristic for properly determining the encoding of web-sent POST bodies. There are some promising avenues that will get it right 99.9% of the time, and we need to package them into up a middleware that will tag Strings correctly.

Incompatible Encodings

If you’ve been paying attention, you’ve probably noticed that while the database drivers have solved one problem, they actually introduced another one.

Imagine that you’re using a MySQL database encoded in ISO-8859-1 (or ISO-8859-5, popular for Russian applications, or any other non-UTF-8 encoding). Now that the String coming back from the database is properly tagged as ISO-8859-1, Ruby will refuse to concatenate it onto the ERB buffer (which is encoded in UTF-8). Even if we solved this problem for ERB, it could be trivially reintroduced in other parts of the application through regular concatenation (+, concat, or even String interpolation).

Again, this problem is somewhat mitigated due to the ASCII-7 subset exception, which means that as long as one of the two incompatible Strings uses only English characters, users won’t see any problems. Again, because this “solution” means that the Ruby developer in question still may not understand encodings, this simply defers the problem to some uncertain point in the future when they either add a non-ASCII-7 character to their template or the user submits a non-ASCII-7 String.

The Solution

If you got this far, you’re probably thinking “Holy shit this encoding stuff is crazy. I don’t want to have to know any of this! I just want to write my web app!”

And you’d be correct.

Other languages, such as Java and Python, solve this problem by encodeing every String that enters the language as UTF-8 (or UTF-16). Theoretically, it is possible to represent the characters of every encoding in UTF-8. By doing this, programmers only ever deal with one kind of String, and concatenation happens between UTF-8 Strings.

However, this solution does not work very well for the Japanese community. For a variety of complicated reasons, Japanese encoding, such as SHIFT-JIS, are not considered to losslessly encode into UTF-8. As a result, Ruby has a policy of not attempting to simply encode any inbound String into UTF-8.

This decision is debatable, but the fact is that if Ruby transparently transcoded all content into UTF-8, a large portion of the Ruby community would see invisible lossy changes to their content. That part of the community is willing to put up with incompatible encoding exceptions because properly handling the encodings they regularly deal with is a somewhat manual process.

On the other hand, many Rails applications work mostly with encodings that trivially encode to UTF-8 (such as UTF-8 itself, ASCII, and the ISO-8859-1 family). For this rather large part of the community, having to manually encode Strings to solve incompatible encoding problem feels like a burden that belongs on the machine has been inappropriately shifted onto Rails application developers.

But there is a solution.

By default, Ruby should continue to support Strings of many different encodings, and raise exceptions liberally when a developer attempts to concatenate Strings of different encodings. This would satisfy those with encoding concerns that require manual resolution.

Additionally, you would be able to set a preferred encoding. This would inform drivers at the boundary (such as database drivers) that you would like them to convert any Strings that they tag with an encoding to your preferred encoding immediately. By default, Rails would set this to UTF-8, so Strings that you get back from the database or other external source would always be in UTF-8.

If a String at the boundary could not be converted (for instance, if you set ISO-8859-1 as the preferred encoding, this would happen a lot), you would get an exception as soon as that String entered the system.

In practice, almost all usage of this setting would be to specify UTF-8 as a preferred encoding. From your perspective, if you were dealing in UTF-8, ISO-8859-* and ASCII (most Western developers), you would never have to care about encodings.

Even better, Ruby already has a mechanism that is mostly designed for this purpose. In Ruby 1.9, setting Encoding.default_internal tells Ruby to encode all Strings crossing the barrier via its IO system into that preferred encoding. All we’d need, then, is for maintainers of database drivers to honor this convention as well.

It doesn’t require any changes to Ruby itself, and places the burden squarely on the few people who already need to deal with encodings (because taking data from outside of Ruby, via C, always already requires a tagging step). I have spoken with Aaron Patterson, who has been working on the SQLite3 driver, and he feels that this change is simple enough for maintainers of drivers dealing with external Strings to make it a viable option. He has already patched SQLite3 to make it respect default_internal.

However you feel about Ruby’s solution to expose String encodings directly in the language, you should agree that since we’re stuck with it for the forseeable future, this solution shifts the burden of dealing with it from the unwashed masses (most of whom have no idea what an encoding is) to a few maintainers of C extensions and libraries that deal in binary data. Getting this right as soon as possible will substantially ease the transition from Ruby 1.8 to Ruby 1.9.

Postscript: What Happened in 1.8!?

When people first move to 1.9 and encounter these shenanigans, they often wonder why everything seemed so simple in Ruby 1.8, and yet seemed to work.

There are a few reasons for this.

First, keep in mind that in Ruby 1.8, Strings are simple sequences of bytes. Ruby String operations just concatenate those byte sequences together without any kind of check. This means that concatenating two UTF-8 Strings together will just work, since the combined byte sequence is still valid UTF-8. As long as the client for the Ruby code (such as the browser) is told that the bytes are encoded in UTF-8, all is well. Rails does this by setting the default charset for all documents to UTF-8.

Second, Ruby 1.8 has a “UTF-8″ mode that makes its regular expression engine treat all Strings as UTF-8. In this mode (which is triggered by setting $KCODE = “UTF-8″), the regular expression engine correctly matches a complete UTF-8 character for /./, for instance. Rails sets this global by default, so if you were using Rails, regular expressions respect unicode characters, not raw bytes.

Third, very little non-English content in the wild is actually encoded in ISO-8859-1. If you were expecting to deal with content that was not English, you would probably set your MySQL database to use a UTF-8 encoding. Since Rails sets UTF-8 as the charset of outbound documents, most browsers will in fact return UTF-8 encoded data.

Fourth, the problems caused when an ISO-8859-1 String is accidentally concatenated into a UTF-8 String are not as jarring as the errors produced by Ruby 1.9. Let’s try a little experiment. First, open up a text editor, create a new file, and save it in the ISO-8859-1 encoding.

$KCODE = "UTF-8"
 
iso_8859_1 = "ümlaut"
 
# the byte representation of ümlaut in unicode
utf8 = "\xC3\xBCmlat"
 
puts iso_8859_1
puts utf8
 
puts iso_8859_1 + utf8
puts utf8 + iso_8859_1
 
# Output
# ?mlat
# ümlaut
# ?mlatümlaut
# ümlaut?mlat

If you somehow get ISO-8859-1 encoded content that uses characters outside of the ASCII-7 range, Ruby doesn’t puke. Instead, it simply replaces the unidentified character with a “?”, which can easily go unnoticed in a mostly English site with a few “José”s thrown into the mix. It could also easily be dismissed as a “weird bug that we don’t have time to figure out right now”.

Finally, Rails itself provides a pure-Ruby UTF-8 library that mops up a lot of the remaining issues. Specifically, it provides an alternate String class that can properly handle operations like split, truncate, index, justify and other operations that need to operate on characters, not bytes. It then uses this library internally in helpers like truncate, transparently avoiding a whole other class of issue.

In short, if you’re dealing mostly with English text, and you get unlucky enough the get ISO-8859-1 input from somewhere, the worst case is that you get a “?” instead of a “é”. If you’re dealing with a lot of non-English text, you’re probably being not using ISO-8859-1 sources. In either case, English (ASCII) text is compatible with UTF-8, and Rails provides solid enough pure-Ruby UTF-8 support to get you most of the rest of the way.

That said, anyone dealing with encodings other than UTF-8 and ISO-8859-1 (Japanese and Russian Rubyists) were definitely not in a good place with Ruby 1.8.

Thanks

I want to personally thank Jay Freeman (aka saurik), who in addition to being a general badass, spent about 15 hours with me patiently explaining these issues and working through the Ruby 1.9 source to help fully understand the tradeoffs available.

The Web Doesn’t Suck. Browsers Are Innovating.

This week saw a flurry of back-and-forth about the future of the web platform. In the “web sucks” camp were Sachin Agarwal of Posterous (The Web Sucks. Browsers need to innovate) and Joe Hewitt (Sachin summarized some of his tweets at @joehewitt agrees with me).

Chris Blizzard responded with a few narrow examples of what Firefox has been doing lately (geolocation, orientation, WebGL).

On Hacker News, most of the comments took Sachin to task for his argument that standards don’t matter, pointing people back to the “bad old days” of the browser wars.

In my opinion, both camps kind of miss the point.

Sachin made some very pointed factual claims which are just complete, pure hokum. Close to the top of his post, he says:

Web applications don’t have threading, GPU acceleration, drag and drop, copy and paste of rich media, true offline access, or persistence. Are you kidding me?

Really? Are you kidding me. In fact, browsers have implemented, and the WHAT-WG has been busily standardizing, all of these things for quite a number of years now. Threading? Check. GPU acceleration? Check and check. Drag and drop? Check. Offline access and persistence? Check and check.

Almost universally, these features, which exist in shipping browsers today, were based on experiments conducted years ago by browsers (notably Firefox and Safari, more recently Chrome), and did not wait for approval from the standards bodies to begin implementing.

In fact, large swaths of HTML5 are based on proposals from browsers, who have either already implemented the feature (CSS-based animations and transitions) or who then build the feature without waiting for final approval. And this is transparently obvious to anyone, since HTML5 has not not yet been approved, and some parts are the subject of internal W3C debate, yet the browsers are implementing the parts they like and innovating in other areas.

You can see this explicitly in the features prefixed with -moz or -webkit, because they’re going ahead and implementing features that have not gotten consensus yet.

In 2010, the WHAT-WG is a functioning place to bring a proposal for further review once you’re conducting some experiments or have a working implementation. Nobody is sitting on their hands waiting for final approval of HTML5. Don’t confuse the fact that people are submitting their ideas to the standards bodies with the misguided idea that only approved standards are being implemented.

And in fact, this is basically never how it’s worked. Apple implemented the <canvas> tag and <input type="search" /> in 2004 (here’s a 2004 rant from the editor of the then-brand-new HTML5 spec). Opera and Apple worked on the <video> tag in 2007. The new CSS flexible box model is based on XUL, which Firefox has had for over a decade. HTML5 drag and drop support is based on implementations shipped by the IE team over a decade ago. Firefox has been extending JavaScript for almost its entire history (most recently with JavaScript 1.8).

CSS transition, transforms and animations were built by Apple for the iPhone before they were a spec, and only later ported to the desktop. Firefox did the initial experimentation on WebGL in 2006, back when they were calling it Canvas 3d (here‘s a working add-on by Mozilla developer Vladimir Vukićević).

Google Gears shipped what would become the Application Cache, Web Storage, and Web Workers, proving out the concepts. It also shipped geolocation, which is now part of the HTML5 spec.

@font-face originally shipped in Internet Explorer 5.5. Apple has added new mobile events (touchstart, touchmove, touchend, onorientationchange) with accompanying JavaScript APIs (Apple developer account needed) without any spec activity that I can discern. Firefox, at the same time, added support for hardware accelerometers.

Even Microsoft bucked the standards bodies by shipping its own implementation of the W3C’s “cross-origin-resource-sharing” spec called Cross domain request, and shipped it in IE8.

It’s perfectly understandable that people who haven’t been following along for the past few years might have missed all of this. But the fact remains that browser vendors have been moving at a very rapid clip, implementing all kinds of interesting features, and then sharing them with other browsers for feedback, and, one hopes, consensus.

Some of these implementations suck (I’ve heard some people say not-nice things about WebGL and drag-and-drop, for instance). But that’s quite a bit different from saying that browsers are sitting on their hands waiting for the W3C or WHAT-WG to tell them what to do.

Named Gem Environments and Bundler

In the beginning, Rubygems made a decision to allow multiple versions of individual gems in the system repository of gems. This allowed people to use whatever versions of gems they needed for individual scripts, without having to partition the gems for specific purposes.

This was a nice starting place. Being able to just install whatever gems into one place and have scripts just work, without having to partition your gems into buckets made Rubygems an extremely pleasant tool to work with. In my opinion, it was a good decision.

As the ecosystem has matured, and more people build gems with dependencies (especially loose dependencies, like “rack”, “>= 1.0″), this original sin has introduced the dreaded activation error:

can't activate rspec-rails (= 1.3.2, runtime) for [], 
already activated rspec-rails-2.0.0.beta.6

This error occurs because of the linear way that gems are “activated”. For instance, consider the following scenario:

# On your system:
thin (1.2.7)
- daemons (>= 1.0.9)
- eventmachine (>= 0.12.6)
- rack (>= 1.0.0)

actionpack (2.3.5)
- activesupport (= 2.3.5)
- rack (~> 1.0.0)

rack (1.0.0)
rack (1.1.0)
activesupport (2.3.5)
daemons (1.0.9)
eventmachine (0.12.6)

Quickly glancing at these dependencies, you can see that these two gems are “compatible”. The gems have the rack dependency in common, and the >= 1.0.0 is compatible with ~> 1.0.0.

However, there are two ways to require these gems. First, you could require actionpack first.

require "action_pack"
# will result in "activating" ActiveSupport 2.3.5 and Rack 1.0.0
 
require "thin"
# will notice that Rack 1.0.0 is already activated, and satisfies
# the >= 1.0.0 requirement, so just activates daemons and eventmachine

Second, you could require thin first.

require "thin"
# will result in "activating" Rack 1.1.0, Daemons 1.0.9, 
# and EventMachine 0.12.6
 
require "action_pack"
# will notice that Rack 1.1.0 is already activated, but that
# it is incompatible with ~> 1.0.0. Therefore, it will emit:
# ---
# can't activate rack (~> 1.0.0, runtime) for ["actionpack-2.3.5"], 
# already activated rack-1.1.0 for ["thin-1.2.7"]

In this case, because thin pulled in Rack 1.1, it becomes impossible to load in actionpack, despite the fact that a potentially valid combination exists.

This problem is fundamental to the approach of supporting different versions of the same gem in the system and activating gems linearly. In other words, because no single entity ever has the opportunity to examine the entire list of dependencies, the onus is on the user to make sure that the gems requires are ordered correctly. Sometimes, it means that the user must explicitly require the right version of a child dependency just to make sure that the right versions are loaded.

There are two possible solutions to this problem.

Multiple Named Environments

One solution to this problem is to catch potential conflicts at installation time and ask the user to manage multiple named environments, each with a fully consistent, non-conflicting view of the world.

The best way to implement this looks something like this (I’ll use a fictitious gemenv command to illustrate):

$ gemenv install thin
- Installing daemons (1.0.10)
- Installing eventmachine (0.12.10)
- Installing rack (1.1.0)
- Installing thin (1.2.7)

$ gemenv install actionpack -v 2.3.5
- Uninstalling rack (1.1.0)
- Installing rack (1.0.1)
- Installing actionpack (2.3.5)

$ gemenv install rack -v 1.1.0
- rack (1.1.0) conflicts with actionpack (2.3.5)
- if you want rack (1.1.0) and actionpack (2.3.5)
  you will need a new environment.

$ gemenv create rails3
$ gemenv install actionpack -v 3.0.0.beta3
- Installing abstract (1.0.0)
- Installing builder (2.1.2)
- Installing i18n (0.3.7)
- Installing memcache-client (1.8.2)
- Installing tzinfo (0.3.20)
- Installing activesupport (3.0.0.beta3)
- Installing activemodel (3.0.0.beta3)
- Installing erubis (2.6.5)
- Installing rack (1.1.0)
- Installing rack-mount (0.6.3)
- Installing rack-test (0.5.3)
- Installing actionpack (3.0.0.beta3)

$ gemenv use default
$ ruby -e "puts Rack::VERSION"
1.0.1
$ gemenv use rails3
$ ruby -e "puts Rack::VERSION"
1.1.0

Essentially, the single entity with full knowledge of all dependencies is the installer, and the user creates as many environments as he or she needs for the various non-conflicting sets of gems in use.

This works nicely, because it guarantees that once using an environment, all gems available are compatible. Note that the above command is fictitious, but it bears similarity to rip.

Virtual, Anonymous Environments

Another solution, the one bundler uses, is to allow multiple, conflicting versions of gems to exist in the system repository of packages, but to ensure a lack of conflicts by resolving the dependencies used by individual applications.

First, install conflicting gems. In this case, Rails 2.3 and Rails 3.0 require different, incompatible versions of Rack (1.0.x and 1.1.x).

$ gem install rails
... output ...
$ gem install rails -v 3.0.0.beta3
... output ...

Next, specify which version of Rails to use in your application’s Gemfile:

gem "rails", "2.3.5"

When running script/server in an app using Bundler, Bundler can determine that Rails needs Rack 1.0.x, and pulls in Rack 1.0.1 from the system. If you had specified gem "rails", "3.0.0.beta3", bundler would have pulled in Rack 1.1.0 from the system.

In essence, we have the same kind of isolation as the fictitious command above, but instead of manually managing named environments, Bundler creates virtual isolated environments based on the list of gems used in your application.

Why Did We Use Virtual, Anonymous Environments?

When considering the tradeoffs between these two solutions, we realized that applications already typically have a list (executable or not) of its dependencies. The gem install command already works great for installing dependencies, and introducing a dependency resolution step there feels awkward and out of place.

Additionally, as an application evolves, it’s natural to continue updating its list of gems, keeping a record of the changes as you go. You could keep a separate named environment for each application, but you’d probably want to keep a list of the dependencies in the application anyway so that it’s possible to get up and running on a second machine.

In short, since a list of an application’s dependencies makes sense anyway, why burden the end-user with the need to maintain separate named environments and manually install gems.

And once we’re doing this, why not build a toolchain around installing gems and keeping records of not only the top-level installed gems, but the exact versions of all gems used at a given time? Why not build in support for “edge” gems on your local machine or in git repositories? Why not create workflows for sharing your application with other developers and deploying your application?

As application and gem developers ourselves, we wanted a tool that managed application’s dependencies across the lifecycle of an application, in the context of an application.

That said, there is absolutely room for experimentation in this space. Tools that enforce consistency at install time might be extremely appropriate for managing the gems that you use in scripts, but which you don’t share with other machines. Context matters.

Postscript

We commonly hear something to the effect of “but why add all this complexity? Without all these features, bundler could be so much smaller!”. The truth is that the bundler code itself is under 2,000 lines of code, and hasn’t grown a whole lot in the past few major revisions.

The new rpg tool, recently released by the venerable Ryan Tomayko, is also in that range. The original rip (currently in the “classic” branch) was also in that ballpark. The new rip (the current master branch), may well demonstrate that a fully-featured dependency management system for Ruby can be written in many fewer lines of code. If so, I’m excited to see the abstractions that they use to make it so.

But the bottom line is that all of the new package management solutions have done a good job of packing features into a small number of new lines of code, bundler included. By starting with good abstractions, it’s often possible to add rich new features without having to add a whole lot of new code (or even by deleting code!). We’ve certainly found that in our journey with bundler.

Ruby Require Order Problems

Bundler has inadvertantly exposed a number of require order issues in existing gems. I figured I’d take the opportunity to talk about them. There are basically two kinds of gem ordering issues:

Missing Requires

Imagine a gem that uses nokogiri, but never requires it. Instead, it assumes that something that is required before it will do the requiring. This happens relatively often with Rails plugins (which were previously able to assume that they were loaded at a particular time and place).

A well-known example of this problem is gems that didn’t require “yaml” because Rubygems required it. When Rubygems removed its require in 1.3.6, a number of gems broke. In general, bundler assumes that gems require what they need. If gems do so, this class of ordering problems is eliminated.

Constant Definition as API

This is where a gem checked for defined?(SomeConstant) to decide whether to activate some functionality.

This is a bit tricky. Essentially, the gem is saying that the API for using it is “require some other gem before me”. Personally, I consider that to be a problematic API, because it’s very implicit, and can be hard to track down exactly who did what require.

A better solution is to provide an explicit hook for people to activate the optional functionality. For instance, Haml
provides: Haml.init_rails(binding) which you can run after activating Haml.

This is slightly more manual than some would like, but the API of “make sure you require the optional dependencies before me” is also manual, and more error-prone.

Even if Bundler “respected require order”, which we plan to do (in some way) in 0.10, it’s still up to the user of the gem to ensure that they listed the optional gem above the required gem. This is not ideal.

A workaround that works great in Bundler 0.9 is to simply require the order-dependent gems above Bundler.require. We do this in Rails, so that gems can test for the existence of the Rails constant to decide whether to add optional Rails dependencies.

require "rails/all" 
Bundler.require(:default, Rails.env)

In the case of shoulda and mocha, a better solution could be:

# Gemfile 
group :test do 
  gem "shoulda" 
  gem "mocha" 
  # possible other gems where order doesn't matter 
end
 
# application.rb 
Bundler.require(:default) 
Bundler.require(Rails.env) unless Rails.env.test? 
 
# test_helper.rb 
# Since the order matters, require these gems manually, in the right order 
require "shoulda" 
Bundler.require(:test)

In my opinion, if you have gems that specifically depend on other gems, it is appropriate to manually require them first before automatically requiring things using Bundler.require. You should treat Bundler.require as a shortcut
for listing out a stack of requires.

Possible solutions?

One long-term solution, if we get gem metadata in Rubygems 1.4 (or optional dependencies down the line), would be to specify that a gem has optional dependencies on another gem and specify a file to run both gems are
available.

For instance, the Haml gem could say:

# gemspec 
s.integrates_with "rails", "~> 3.0.0.beta2", "haml/rails" 
 
# haml/rails.rb 
require "haml" 
require "rails" 
Haml.init_rails(binding)

Bundler would handle this in Bundler.require (or possibly Bundler.setup).

If this feature existed in Rubygems, it would work via gem activation. If the Haml gem was activated after the Rails gem, it would require “haml/rails” immediately.

If the Haml gem was activated otherwise, it wouldn’t do anything until the Rails gem was activated. When the Rails gem was activated, it would require the file.

Using .gemspecs as Intended

When you clone a repository containing a Unix tool (or download a tarball), there’s a standard way to install it. This is expected to work without any other dependencies, on all machines where the tool is supported.

$ autoconf
$ ./configure
$ make
$ sudo make install

This provides a standard way to download, build and install Unix tools. In Ruby, we have a similar (little-known) standard:

$ gem build gem_name.gemspec
$ gem install gem_name-version.gem

If you opt-into this convention, not only will it simplify the install process for your users, but it will make it possible for bundler (and other future automated tools) to build and install your gem (including binaries, proper load path handling and compilation of C extensions) from a local path or git repository.

What to Do

Create a .gemspec in the root of your repository and check it in.

Feel free to use dynamic code in here. When your gem is built, Rubygems will run that code and create a static representation. This means it’s fine to pull your gem’s version or other shared details out of your library itself. Do not, however, use other libraries or dependencies.

You can also use Dir[] in your .gemspec to get a list of files (and remove files you don’t want with -; see the example below).

Here’s bundler’s .gemspec:

# -*- encoding: utf-8 -*-
lib = File.expand_path('../lib/', __FILE__)
$:.unshift lib unless $:.include?(lib)
 
require 'bundler/version'
 
Gem::Specification.new do |s|
  s.name        = "bundler"
  s.version     = Bundler::VERSION
  s.platform    = Gem::Platform::RUBY
  s.authors     = ["Carl Lerche", "Yehuda Katz", "André Arko"]
  s.email       = ["carlhuda@engineyard.com"]
  s.homepage    = "http://github.com/carlhuda/bundler"
  s.summary     = "The best way to manage your application's dependencies"
  s.description = "Bundler manages an application's dependencies through its entire life, across many machines, systematically and repeatably"
 
  s.required_rubygems_version = ">= 1.3.6"
  s.rubyforge_project         = "bundler"
 
  s.add_development_dependency "rspec"
 
  s.files        = Dir.glob("{bin,lib}/**/*") + %w(LICENSE README.md ROADMAP.md CHANGELOG.md)
  s.executables  = ['bundle']
  s.require_path = 'lib'
end

If you didn’t already know this, the DSL for gem specifications is already pretty clean and straight-forward, there is no need to generate your gemspec using alternative tools.

Your gemspec should run standalone, ideally with no additional dependencies. You can assume its __FILE__ is located in the root of your project.

When it comes time to build your gem, use gem build.

$ gem build bundler.gemspec

This will spit out a .gem file properly named with a static version of the gem specification inside, having resolved things like Dir.glob calls and the version you might have pulled in from your library.

Next, you can push your gem to Rubygems.org quickly and painlessly:

$ gem push bundler-0.9.15.gem

If you’ve already provided credentials, you’ve now published your gem. If not, you will be asked for your credentials (once per machine).

You can easily automate this process using Rake:

$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
require "bundler/version"
 
task :build do
  system "gem build bundler.gemspec"
end
 
task :release => :build do
  system "gem push bundler-#{Bunder::VERSION}"
end

Using tools that are built into Ruby and Rubygems creates a more streamlined, conventional experience for all involved. Instead of trying to figure out what command to run to create a gem, expect to be able to run gem build mygem.gemspec.

A nice side-effect of this is that those who check in valid .gemspec files can take advantage of tools like bundler that allow git repositories to stand in for gems. By using the gem build convention, bundler is able to generate binaries and compile C extensions from local paths or git repositories in a conventional, repeatable way.

Try it. You’ll like it.

Ruby’s Implementation Does Not Define its Semantics

When I was first getting started with Ruby, I heard a lot of talk about blocks, and how you could “cast” them to Procs by using the & operator when calling methods. Last week, in comments about my last post (Ruby is NOT a Callable Oriented Language (It’s Object Oriented)), I heard that claim again.

To be honest, I never really thought that much about it, and the idea of “casting” a block to a Proc never took hold in my mental model, but when discussing my post with a number of people, I realized that a lot of people have this concept in their mental model of Ruby.

It is not part of Ruby’s semantics.

In some cases, Ruby’s internal implementation performs optimizations by eliminating the creation of objects until you specifically ask for them. Those optimizations are completely invisible, and again, not part of Ruby’s semantics.

Let’s look at a few examples.

Blocks vs. Procs

Consider the following scenarios:

def foo
  yield
end
 
def bar(&block)
  puts block.object_id
  baz(&block)
end
 
def baz(&block)
  puts block.object_id
  yield
end
 
foo { puts "HELLO" } #=> "HELLO"
bar { puts "HELLO" } #=> "2148083200\n2148083200\nHELLO"

Here, I have three methods using blocks in different ways. In the first method (foo), I don’t specify the block at all, yielding to the implicit block. In the second case, I specify a block parameter, print its object_id, and then send it on to the baz method. In the third case, I specify the block, print out its object_id and yield to the block.

In Ruby’s semantics, these three uses are identical. In the first case, yield calls an implicit Proc object. In the second case, it takes an explicit Proc, then sends it on to the next method. In the last case, it takes an explicit Proc, but yields to the implicit copy of the same object.

So what’s the & thing for?

In Ruby, in addition to normal arguments, methods can receive a Proc in a special slot. All methods can receive such an argument, and Procs passed in that slot are silently ignored if not yielded to:

def foo
  puts "HELLO"
end
 
foo { something_crazy } #=> "HELLO"

On the other hand, if you want a method to receive a Proc in that slot, and thus be able to yield to it, you specify that by prefixing it with an &:

def foo
  yield
end
 
my_proc = Proc.new { puts "HELLO" }
foo(&my_proc)

Here, you’re telling the foo method that my_proc is not a normal argument; it should be placed into the proc slot and made available to yield.

Additionally, if you want access to the Proc object, you can give it a name:

def foo(&block)
  puts block.object_id
  yield
end
 
foo { puts "HELLO" } #=> "2148084320\nHELLO"

This simply means that you want access to the implicit Proc in a variable named block.

Because, in most cases, you’re passing in a block (using do/end or {}), and calling it using yield, Ruby provides some syntax sugar to make that simple case more pleasing. That does not, however, mean that there is a special block construct in Ruby’s semantics, nor does it mean that the & is casting the a block to a Proc.

You can tell that blocks are not being semantically wrapped and unwrapped because blocks passed along via & share the same object_id across methods.

Mental Models

For the following code there are two possible mental models.

def foo(&block)
  puts block.object_id
  yield
end
 
b = Proc.new { puts "OMG" }
puts b.object_id
foo(&b) #=> 2148084040\n2148084040\nOMG

In the first, the &b unwraps the Proc object, and the &block recasts it into a Proc. However, it somehow also wraps it back into the same wrapper that it came from into the first place.

In the second, the &b puts the b Proc into the block slot in foo‘s argument list, and the &block gives the implicit Proc a name. There is no need to explain why the Proc has the same object_id; it is the same Object!

These two mental models are perfectly valid (the first actually reflects Ruby’s internal implementation). I claim that those who want to use the first mental model have the heavy burden of introducing the new concept to the Ruby language of a non-object block, and that as a result, it should be generally rejected.

Metaclasses

Similarly, in Ruby’s internal implementation, an Object does not get a metaclass until you ask for one.

obj = Object.new
# internally, obj does not have a metaclass here
 
obj.to_s
# internally, Ruby skips right up to Object when searching for #to_s, since
# it knows that no metaclass exists
 
def obj.hello
  puts "HELLO"
end
# now, Ruby internally rewrites obj's class pointer to point to a new internal
# metaclass which has the hello method on it
 
obj.to_s
# Now, Ruby searches obj's metaclass before jumping up to Object
 
obj.to_s
# Now, Ruby skips the search because it's already cached the method
# lookup

All of the comments in the above snippet are correct, but semantically, none of them are important. In all cases, Ruby is semantically looking for methods in obj‘s metaclass, and when it doesn’t find any, it searches higher up. In order to improve performance, Ruby skips creating a metaclass if no methods or modules are added to an object’s metaclass, but that doesn’t change Ruby’s semantics. It’s just an optimization.

By thinking in terms of Ruby’s implementation, instead of Ruby’s semantics, you are forced to think about a mutable class pointer and consider the possibility that an object has no metaclass.

Mental Models

Again, there are two possible mental models. In the first, Ruby objects have a class pointer, which they manipulate to point to new metaclass objects which are created only when methods or modules are added to an object. Additionally, Ruby objects have a method cache, which they use to store method lookups. When a method is looked up twice, in this mental model, some classes are skipped because Ruby already knows that they don’t have the method.

In the second mental model, all Ruby objects have a metaclass, and method lookup always goes through the metaclass and up the superclass chain until a method is found.

As before, I claim that those who want to impose the first mental model on Ruby programmers have the heavy burden of introducing the new concepts of “class pointer” and “method cache”, which are not Ruby objects and have no visible implications on Ruby semantics.

Regular Expression Matches

In Ruby, certain regular expression operations create implicit local variables that reflect parts of the match:

def foo(str)
  str =~ /e(.)l/
  p $~
  p $`
  p $'
  p $1
  p $2
end
 
foo("hello") #=> #<MatchData "ell" 1:"l">\n"h"\n"o"\n"l"\nnil

This behavior is mostly inherited from Perl, and Matz has said a few times that he would not support Perl’s backrefs if he had it to do over again. However, the provide another opportune example of implicit objects in Ruby.

Mental Models

In this case, there are three possible mental models.

In the first mental model, if you don’t use any $ local variables, they don’t exist. When you use a specific one, it springs into existence. For instance, when using $1, Ruby looks at some internal representation of the last match and retrieves the last capture. If you use $~, Ruby creates a MatchData object out of it.

In the second mental model, when you call a method that uses regular expressions, Ruby walks back up the stack frames, and inserts the $ local variables on it when it finds the original caller. If you later use the variables, they are already there. Ruby must be a little bit clever, because the most recent frame on the stack (which might include C frames, Java frames, or internal Ruby frames in Rubinius) is not always the calling frame.

In the last mental model, when you call a method that uses regular expressions, there is an implicit match object available (similar to the implicit Proc object that is available in methods). The $~ variable is mapped to that implicit object, while the $1 variable is the equivalent of $~[1].

Again, this last mental model introduces the least burdensome ideas into Ruby. The first mental model introduces the idea of an internal representation of the last match, while the second mental model (which again, has the upside of being how most implementations actually do it) introduces the concept of stack frames, which are not Ruby objects.

The last mental model uses an actual Ruby object, and does not introduce new concepts. Again, I prefer it.

Conclusion

In a number of places, it is possible to imbue Ruby semantics with mental models that reflect the actual Ruby implementation, or the fact that it’s possible to imagine that a Ruby object only springs into existence when it is asked for.

However, these mental models require that Ruby programmers add non-objects to the semantics of Ruby, and requiring contortions to explain away Ruby’s own efforts to hide these internals from the higher-level constructs of the language. For instance, while Ruby internally wraps and unwraps Procs when passing them to methods, it makes sure that the Proc object attached to a block is always the same, in an effort to hide the internal details from programmers.

As a result, explaining Ruby’s semantics in terms of these internals requires contortions and new constructs that are not natively part of Ruby’s object model, and those explanations should be avoided.

To be clear, I am not arguing that it’s not useful to understand Ruby’s implementation. It can, in fact, be quite useful, just as it’s useful to understand how C’s calling convention works under the covers. However, just as day-to-day programmers in C don’t need to think about the emitted Assembler, day-to-day Ruby programmers don’t need to think about the implementation. And finally, just as C implementations are free to use different calling conventions without breaking existing processor-agnostic C (or the mental model that C programmers use), Ruby implementations are free to change the internal implementation of these constructs without breaking pure-Ruby code or the mental model Ruby programmers use.

Archives

Categories

Meta