I officially joined Engine Yard on January 1, 2008, about a week before we announced our Series A funding, becoming its twenty-second employee. I was Engine Yard’s very first “Engineering” hire, and I would spend the next year working on Ezra’s Merb project, finally releasing Merb 1.0 at MerbCamp that October.

When I joined Engine Yard, I had already been working on the jQuery project for a couple years, having started visualjquery.com at a time in jQuery’s history before jQuery had version numbers. I learned a lot from John Resig about building and motivating open source communities, and I did my first professional work writing jQuery in Action.

The Merb project was my first full-time open source work, and I met some of my earliest open source friends while working on it. When I first met Carl, he was working on a rewrite of the Merb router, the one part of Merb I was still scared to work on. I met Andy Delcambre, who now works on Engine Yard’s AppCloud, after he volunteered to help with Merb.

I met Loren Segal, well-known for the YARD documentation tool, while working to give Merb documentation I could be proud of. And I became the first Github user outside of Chris, PJ and Tom in order to get the Merb project on git. If you’re paying attention, that made me Github user nil.id, and was the source of some hilarious bugs.

In Hampton Catlin’s first Ruby survey in 2008, about one in eight respondents said that they preferred Merb, and by the end of the year, the Rails and Merb teams were coming to blows. During this same time, Engine Yard had taken on a Series B round of funding (this time for $15 million), and had grown from a small company of a couple dozen to a company pushing 100. We had Sales and Marketing departments. We also had an Engineering department, which had spent a large part of the year on our first fully automated product, Engine Yard Solo.

On a personal level, I started getting heavily involved in Open Source other than Merb, especially DataMapper and Rubinius. I started speaking at conferences, starting with a talk on DataMapper at MountainWest RubyConf in 2008. Engine Yard also threw its first Hackfest at that event, an event we would repeat in 2009 and again in 2010. This event, and the work our great community marketing, Open Source and support teams would do over the next several years elevated Engine Yard’s reputation as a team of experts that would keep your app running no matter what.

As Engine Yard grew, a lot of people asked us why we were supporting Merb when so much of our business depended on the success of the Rails community. By late 2008, it was probably the most common question I got when speaking at conferences. In truth, the best I could tell was that Engine Yard was worried about the future of Rails, and wanted to make sure that Ruby remained successful. What we didn’t realize at the time was that the one full-time employee (me) that Engine Yard had on Merb was more than the total number of worldwide full-time developers on Rails. In short, Rails was a fully volunteer project, and that had something to do with some of the issues that drove people to Merb in the first place.

At the end of 2008, Tom Mornini, the CTO of Engine Yard, recognized that if we could join forces (and this was definitely a big if), we could help shore up Rails into the future. I remember the first time Tom asked me about it in the stairwell of our South Park office. I thought he was off the wall to even suggest it. After all, it didn’t seem like the teams could get along at all. But the Merb team had adopted so much of the Rails philosophy that at its core, the only question that remained was how aggressive the Rails project would get to clean things up.

In the years since Ezra first announced Merb, Rails had gotten thread-safe, Rack support, and some improvements to the internals. After hashing out the details (in some very contentious conversations), we decided to give it a try, announcing we would merge the projects on December 23, 2008. That announcement quite literally changed the trajectory of my life. In the almost two years since that announcement, I have worked harder than I have ever worked before, and am more proud of the work I have done than I have ever been before.

Perhaps more importantly, I have worked with some of the best open source developers in the business without whom Rails 3 would have been a pale shadow of itself. Including me, Rails 3 brought on eight new committers. The Rails 3 project did more than improve the quality of the codebase: it reinvigorated the community of developers actively contributing. And by moving toward a model that allowed Rails to have more dependencies, Rails became the center of a thriving Ruby community, where anyone can reuse the code that makes Rails tick for their non-Rails projects.

In the summer of 2009, José Valim became a Google Summer of Code student dedicated to rewriting the Rails generators so that users of DataMapper, RSpec, and Haml could use the standard Rails generators. A lot of people thought it was too optimistic to do in a summer, but José finished the job well before the end of the summer, giving him time to continue his contributions.

Around the same time, Mikel Lindsaar decided to write a ground-up mail implementation, releasing the imaginatively named mail gem in October 2009. This project, which involved writing a parser that could parse the gigabytes of mail in the Trec and Enron email databases, strained credulity, and would enable us to give ActionMailer the facelift it needed for Rails 3.0.

Santiago Pastorino showed up out of nowhere, with a level of pluck that is rarely seen in the Ruby community. Despite a noticeable language barrier, Santiago went from a warning fixer to a Rails committer. This month, he became an official member of the Rails core team. While working on Rails, I learned a lot from a lot of people, but Santiago’s raw determination and skill should stand out as an example to the Ruby community.

Everyone knows Aaron Patterson, and I was just as surprised as everyone else when he started the long, dedicated task of improving the performance of the Arel library. He already has quite the reputation in the Ruby community, and even has commit access to Ruby itself, so I’m personally gratified to work with him on Rails. If you haven’t yet watched it, watch his Worst Ideas Ever presentation from RubyConf last year. Be prepared to laugh.

Xavier Noria has been a one-man documentation machine, helping to grow and maintain the Rails guides, and constantly reminds everyone that documentation is just as essential to the codebase as tests. Without him, Rails 3 might have been great internally, but nobody would know it. Documentation folks often get little love in open source projects, but to me, the work Xavier does is key to our success.

More recently, I’ve been extremely impressed by Piotr Sarnacki’s work (for Ruby Summer of Code) on finally delivering on the promise of truly mountable engines. His work is already merged into Rails master, and will ship with Rails 3.1. As with the rest of the guys I’ve talked about, he took on a pretty complicated and entangled task, and managed to get more done in less time than people expected. Keep an eye on him!

In short, the Rails 3 project was about more than code; it was about reshaping the Rails community. I’m proud to have been a part of growing the Rails ecosystem, and humbled by the number of people who have worked so hard for so little.

Now that we released Rails 3, I have had an opportunity to think about what to do next. I could, of course, continue full-time work on Rails for Engine Yard, which has been a fantastic place to do Open Source for the past several years. I am still extremely interested in open web technologies, and want to do my part to help the open web succeed. Having robust, mature server side frameworks that are easy for anyone to use is certainly one large piece of the puzzle, and I’ve really enjoyed helping to bring the core of Rails up to date with modern web technologies.

But as Avi Bryant pointed out last year at DjangoCon, the technology of the open web is shifting toward the client side. That doesn’t change the need for robust, mature server-side frameworks. Even the richest web applications get and sync their data with servers, mostly over the HTTP protocol. And although a lot of people have opined that rich clients can simply wire directly into something like CouchDB or MongoDB, applications with rich clients tend to have the same kinds of complex needs (like authorization) as more tranditional document-based applications have. I see Rails remaining an excellent server-side solution even as clients become richer over the next few years.

There’s still a lot to do for Rails in the future, and I want to be a part of that. On the other hand, the field of rich client development is just getting starting, with new mobile devices like the iPhone, Android and iPad giving us an almost blank canvas to work on. Unfortunately, a lot of the really big bets in this space are on selling licenses to either entire libraries or the development tools needed to efficiently work with the tools.

What the open web needs is a robust, easy to use framework with development tools that are trivial to get started with and awesome to work with. And both the library and the developer tools need to be offered for free, like Rails or jQuery. Since 2008, I have poured my soul into helping to build an amazing open source Ruby web framework and the ecosystem that powers it. Now, I want to bring that same energy and enthusiasm to building great tools that leverage the power of the modern open web and a free and open source community around them.

That’s why I was so excited when I saw that Charles Jolley, who worked on Apple’s Mobile Me product, announced that he was leaving Apple to form a new company dedicated to those same principles. Charles had helped build Mobile Me on SproutCore, a free and open source framework, and had managed to make sure that Apple continued to contribute back to the open source project under the MIT license.

In his blog post announcing that he was leaving Apple, Charles said “SproutCore is now and will always be totally free and open source. I think this business of charging for a commercial license is not an effective way to grow a project. Sure you make a little cash, but at what expense to the community? My goal is to make SproutCore and all of the developer tools that surround it totally free to everyone. All I ask is that you participate in the community somehow to make things a little better for those who come after you.”

This philosophy closely mirrored my thinking on the future of client-side development, and seeing a mature client-side framework open a permissive open source philosophy really excited me. Since the beginning of 2010, I was starting to think that if I wanted a permissively licensed client-side toolkit, I was going to have to build it and its community myself. I already knew about SproutCore–early versions of its build tools were built on Merb, and had met Charles a few times. After reading his blog post, I emailed him to find out what he was thinking about doing and he told me about his plans for SproutCore.

The SproutCore library itself, today, is definitely recovering from having been developed almost exclusively inside Apple. It comes with a number of amazing low-level primitives, and you can build a large, complex application on it. That said, it never had a big enough new user community to get a very friendly public API. Thankfully, it’s much, much easier to build a great public API than it is to build the core of a web application toolkit. Starting this week, I will be joining Charles’ company, Strobe, and will get right to work on the next generation of the SproutCore tools.

I have to say, I’m excited about what comes next.

Postscript

Because you’re probably wondering, I will still be active in the Rails community, and will be doing work for Engine Yard on Rails 3.1. I plan to use Rails 3 and Engine Yard for the server-side elements of the projects I will be doing, and really want the new Rails 3.1 features I talked about at Windy City Rails. I will also be helping Engine Yard start a new Technical Advisory Board, which will help provide community feedback for its cloud products, and guidance about what open source efforts could use the Engine Yard bump. I forsee being part of both the Rails and Engine Yard family for years to come.

preamble: this post explains, in some detail, how we will implement a nice performance boost for Rails developers. Understanding the details might help gain the full benefits of the optimization, but you will gain some benefits even if you have no idea how it works.

As you’ve probably seen, DHH announced that we’d be looking at flushing in Rails 3.1 to improve the client-side performance of typical Rails applications.

The most obvious solution, and one that already exists in plugin form, is to allow a layout to have a new flush method, which would immediately flush the contents of the layout to the browser. By putting the flush method below the JavaScript and CSS includes, the browser could begin downloading and evaluating those static assets while the server continues building the page.

Unfortunately, this solution has a major problem: it requires a fairly significant change in the current model of how people build applications. In general, for performance optimizations (including client-side optimizations), we like to make the default as fast as possible, without asking people to understand a brand new paradigm, centered around the optimization.

The problem lies in the fact that a Rails layout is essentially a template with a bunch of holes to fill in.

<html>
  <head>
    <title><%= yield :title %></title>
    <%= javascript_include_tag :defaults %>
    <%= yield :extra_javascripts %>
    <%= stylesheet_link_tag :defaults %>
    <%= yield :extra_stylesheets %>
  </head>
  <body>
    <%= yield :sidebar %>
    <%= yield %>
  </body>
</html>

I this simple example, each yield is a slot that is filled in by the template (usually via content_for). In order to achieve this, Rails evaluates the template first, which populates a Hash with each piece of content. Next, it renders the layout, and each yield checks the Hash for that content. In short, because of the way layouts work, Rails renders the template first, and then the layout.

To get around this, one option would be to say that everything before the flush must not use yield, and must be able to run before the template. Unfortunately, it’s somewhat common for people to set up a content_for(:javascripts) in a template, to keep the JavaScript needed for a particular snippet of HTML close to the HTML. This means that not only does the user have to be careful about what can go above and below the flush, he can no longer use content_for for things high up in the template, which is a fairly significant change to the overall design of Rails applications.

For Rails 3.1, we wanted a mostly-compatible solution with the same programmer benefits as the existing model, but with all the benefits of automatic flushing. After a number of very long discussions on the topic, José Valim came up with the idea of using Ruby 1.9 fibers to jump back and forth between the template and layout.

Let’s start by taking a look at a very simplified version of the current Rails rendering pipeline. First, we set up a Buffer object purely for logging purposes, so we can see what’s happening as we push things onto the buffer.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
    end
 
    def <<(value)
      super
 
      puts "#{@name} is pushing #{value.inspect}"
    end
  end
end

Next, we create a simple version of ActionView::Base. We implement the content_for method simply, to print out a bit of logging information and stash the value into the @content_for Hash. Note that the real version is pretty similar, with some added logic for capturing the value of the block from ERB.

module Basic
  class ViewContext
    def initialize
      @buffer      = Buffer.new(:main, self)
      @content_for = {}
    end
 
    def content_for(name, value = nil)
      value = yield if block_given?
      puts "Setting #{name} to #{value.inspect}"
      @content_for[name] = value
    end
 
    def read_content(name)
      @content_for[name]
    end
  end
end

Next, we create a number of methods on the ViewContext that look like compiled ERB templates. In real life, the ERB (or Haml) compiler would define these methods.

module Basic
  class ViewContext
    def layout
      @buffer << "<html><head>"
      @buffer << yield(:javascripts).to_s
      @buffer << yield(:stylesheets).to_s
      @buffer << "</head><body>"
      @buffer << yield.to_s
      @buffer << yield(:not_existant).to_s
      @buffer << "</body></html>"
      @buffer
    end
 
    def template
      buffer =  Buffer.new(:template, self)
      content_for(:javascripts) do
        "<script src='application.js'></script>"
      end
      content_for(:stylesheets) do
        "<link href='application.css' rel='stylesheet' />"
      end
      puts "Making a SQL call"
      sleep 1 # Emulate a slow SQL call
      buffer << "Hello world!"
      content_for(:body, buffer)
    end
  end
end

Finally, we define the basic rendering logic:

module Basic
  class ViewContext
    def render
      template
      layout { |value| read_content(value || :body) }
    end
  end
end

As you can see, we first render the template, which will fill up the @content_for Hash, and then call the layout method, with a block which pulls the value from that Hash. This is how yield :javascripts in a layout works.

Unfortunately, this means that the entire template must be rendered first, including the (fake) slow SQL query. We’d prefer to flush the buffer after the JavaScripts and CSS are determined, but before the SQL query is made. Unfortunately, that requires running half of the template method, then continuing with the layout method, retaining the ability to resume the template method later.

You can think of the way that templates are currently rendered (in Rails 2.x and 3.0) like this:

Unfortunately, this makes it very hard to get any more performance juice out without asking the end-developer to make some hard choices. The solution we came up with is to use Ruby 1.9 fibers to allow the rendering to jump back and forth between the template and layout.

Instead of starting with the template and only rendering the layout when ready, we’ll start with the layout, and jump over to the template when a yield is called. Once the content_for that piece is provided by the template, we can jump back to the layout, flush, and continue rendering. As we need more pieces, we can jump back and forth between the template and layout, flushing as we fill in the holes specified by the yield statements.

The implementation is mostly straight-forward:

require "fiber"
 
module Fibered
  class ViewContext < Basic::ViewContext
    def initialize
      super
      @waiting_for = nil
      @fiber       = nil
    end
 
    def content_for(name, value = nil)
      super
      @fiber.resume if @waiting_for == name
    end
 
    def read_content(name)
      content = super
      return content if content
 
      begin
        @waiting_for = name
        Fiber.yield
      ensure
        @waiting_for = nil
      end
 
      super
    end
 
    def layout
      @fiber = Fiber.new do
        super
      end
      @fiber.resume
      @buffer
    end
 
    def render
      layout { |value| read_content(value || :body) }
      template
      @fiber.resume while @fiber.alive?
      @buffer
    end
  end
end

For our fibered implementation, we’ll inherit from Basic::ViewContext, because we want to be able to use the same templates as we used in the original implementation. We update the content_for, read_content, layout and render methods to be fiber-aware. Let’s take them one at a time.

def layout
  @fiber = Fiber.new do
    super
  end
  @fiber.resume
  @buffer
end

First, we wrap the original implementation of layout in a Fiber, and start it right away. Next, we modify the read_content method to become Fiber-aware:

def read_content(name)
  content = super
  return content if content
 
  begin
    @waiting_for = name
    Fiber.yield
  ensure
    @waiting_for = nil
  end
 
  super
end

If the @content_for Hash already has the content, return it right away. Otherwise, say that we’re waiting for the key in question, and yield out of the Fiber. We modify the render method so that the layout is rendered first, followed by the template. As a result, yielding out of the layout will start the template’s rendering.

def render
  layout { |value| read_content(value || :body) }
  template
  @fiber.resume while @fiber.alive?
  @buffer
end

Next, modify the content_for method so that when the content we’re waiting for is provided, we jump back into the layout.

def content_for(name, value = nil)
  super
  @fiber.resume if @waiting_for == name
end

With this setup, the layout and template will ping-pong back and forth, with the layout requesting data, and the template rendering only as far as it needs to go to provide the data requested.

Finally, let’s update the Buffer to take our fibered implementation into consideration.

module Basic
  class Buffer < String
    def initialize(name, context)
      @name    = name
      @fibered = context.fibered?
    end
 
    def <<(value)
      super
 
      if @fibered
        puts "Flushing #{value.inspect}" if @fibered
      else
        puts "#{@name} is pushing #{value.inspect}"
      end
    end
  end
 
  class ViewContext
    def fibered?
      false
    end
  end
end
 
module Fibered
  class ViewContext
    def fibered?
      true
    end
  end
end

Now that we’re rendering the layout in order, we can flush as we go, instead of being forced to wait for the entire template to render before we can start flushing.

It’s worth mentioning that optimal flushing performance will be based on the order of the content_for in your template. If you run your queries first, then put the expensive template rendering, and only finally do the content_for(:javascript) at the end, the flushing behavior will look like this:

Instead of flushing quickly, before the SQL call, things are barely better than they are in Rails 2.3, when the entire template must be rendered before the first flush. Because things are no worse, even in the worst-case scenario, we can make this the default behavior. Most people will see some benefit from it, and people interested in the best performance can order their content_for blocks so they cause the most beneficial flushing.

Even for people willing to put in the effort, this API is better than forcing a manual flush, because you can still put your content_for blocks alongside the templates that they are related to.

Look for this feature in Rails 3.1!

Small Caveat

For the purposes of this simplified example, I assumed that content_for can only be run once, immediately setting the value in the @content_for Hash. However, in some cases, people want to accumulate a String for a particular value. Obviously, we won’t be able to flush until the full String for that value is accumulated.

As a result, we’ll be adding a new API (likely called provide), which will behave exactly the same as content_for, but without the ability to accumulate. In the vast majority of cases, people will want to use provide (for instance, provide :sidebar), and get all the benefits of autoflushing. In a few cases, people will want to be able to continue accumulating a String, and will still be able to use content_for, but the template will not return control to the layout when that happens.

Also note that this (fairly detailed) explanation is not something that you will need to understand as a Rails developer. Instead, you will continue to go about your development as before, using provide if you have just a single piece of content to add, and content_for if you have multiple pieces of content to add, and Rails will automatically optimize flushing for you as well as we can.