First off, apologies for the long delay in posting. Last week, I was giving a training down at Integrum in Phoenix. It was a nice three-day training that starting with building a Merb application, and ended with a walk through the BootLoader and Request path. All in all, a very successful endeavor, but definitely one that took up quite a bit of time (and sadly left me with no time for blogging).

At the end of last week, I continued my work on callbacks, which was actually pretty interesting. The initial approach, if you recall, compiled before, around, and after callbacks, with conditions, into a method that could be called. This effectively unrolled the loop and a number of checks into a nice hardcoded method that was about 10x faster than the previous technique. Take a look at my first post on this topic for more.

When I started to look at integrating it with ActionPack, I realized that the API there is a bit different, in an interesting way. In addition to allowing the typical if/unless conditions, ActionController filters allow you to supply :only/:except conditions. At first glance, these compile as follows:

before_filter :authenticate, :except => :index
# to
dispatch_callback :before, :authenticate, :unless => proc {|c| c.action_name == :index }

However, this would force the check every time, even though these conditions are always the same, for each action. In other words, every trip through index will run exactly the same set of callbacks (as for as :only/:except conditions are concerned). I could have hardcoded this optimization into ActionPack’s usage of the ActiveSupport callbacks, but I decided instead to make it generic.

After thinking about the problem for a couple of days, I realized that it reduces down to “compile a specialized method for these callbacks, per key.” In this case, the “key” is the action name. Now, instead of having the AP code call run_dispatch_callbacks { ... stuff ... }, it calls run_dispatch_callbacks(action_name) { ... stuff ... }. Every time the callbacks are run with a new key, it compiles a version of the method for that key.

The last piece of the puzzle is in specifying these conditions. Now:

before_filter :authenticate, :except => :index
# compiles to
dispatch_callback :before, :authenticate, :per_key => {:unless => proc {|c| c.action_name == :index } }

Per-key options are calculated when the method is compiled, while regular options are compiled into the method. Of course, the vagaries of the compiler are a bit messy, but I plan to clean it up, document it, and integrate it in the next few days.

Tomorrow, I fly out to Chicago to hack with David, Josh, and Rick. Wish us luck!

This is something of a follow-up to Bruce Williams’ wrap-up of “wrapping a method” approaches. Bruce covered two approaches:

alias_method_chain: rename the original method, then copy the new method over it and call the old one.

class Foo
  def bar
    1
  end

  alias_method :original_bar, :bar

  def bar
    original_bar + 1
  end
end

super and extend: override #new and extend the new object’s metaclass with the method.

class Foo
  def bar
    1
  end

  module Incrementor
    def bar
      super + 1
    end
  end

  def self.new
    super.extend Incrementor
  end
end

I wanted to cover two other options, which come with more serious caveats than Bruce’s approaches, but which have their benefits.

explicitly declaring a method as chainable: if you are in control of the code you want to extend, make it chainable from the beginning

class Foo
  chainable = Module.new do
    def bar
      1
    end
  end
  include chainable

  def chainable
    super + 1
  end
end

What’s going on above is that instead of declaring the method on the class itself, we’re sticking it into an (anonymous) module, then including the module into Foo. As a result, we can now define a method on Foo itself, and super to the module we included. We can even wrap up the idiom pretty simply:

class Module
  def overridable(&blk)
    mod = Module.new(&blk)
    include mod
  end
end

class Foo
  overridable do
    def bar
      1
    end
  end

  def bar
    super + 1
  end
end

I like this approach a lot, but it does require that you have control over the original code. In some cases (for instance, Rails chaining its own code via modules) that is the case. This approach, in my opinion, is superior to alias_method_chain because it does not mutate method names, so it eliminates the confusion of looking for a method from a stack trace on a certain line and finding a method with a different name. Also, having to explicitly declare a method chainable provides a cue to the reader of the code that the method is not complete, and that there might be other components elsewhere.

the best of all worlds (with a caveat): the best option would allow you the benefits of the approach immediately above without requiring that you declare the method chainable (so you could add advice to other peoples’ code). The solution, unfortunately, requires a (rather crude) C extension I whipped up.

There are a few things going on above. The interesting part is the little bit of C code. This code lives on any Module. It grabs a method from the module’s method table, and inserts it (with the same name) on any other Module. The TrackAddedMethods module can be extended onto any module, and it will remember all the methods that were added (using a method added hook).

The override method uses the same technique as the chainable method above to create a new module with the methods inside of it. Only this time, as the methods get added, we track them. Each method that is added to the module is then removed from the class itself and moved up (via the C extension) to a module, which is then included into the base class. At this point, the base class doesn’t have the method on it anymore, but the original method is included into the base class via a module.

At this point, we’re at the same place as we would have been using the first technique I described above, so we simply include the module that was created from the override() method and voila: you can tweak someone else’s methods without needing to rename them.

I hope the forgoing didn’t blow anyone’s minds. It blew mine the first time I started playing with it ;)

Hey guys!

We released Merb 1.0.8 today. The notes:

• Fix one remaining place where merb.main.pid was being overwritten by merb -i or merb -r.
• Fix an incompatibility between Merb middleware and other rack middleware
• Missing webrat no longer fatals. It prints a warning instead (so you don’t need webrat for model specs)
• Support cookies being set by the user directly in request specs
• Bump DM versions to 0.9.9 and DO versions to 0.9.10.1
• Improve some convoluted code in Haml
• Merb actions now wrap DataMapper repository blocks (to support a per-action IdentityMap)

Enjoy!

Over the past few months, people have been using thor for increasingly serious things. The project itself was extracted out of my textmate gem, and is a nice Railsish DSL for command-line applications. It uses classes and methods as its abstraction, in much the same way that Rails uses classes and methods. Here’s an example:

class Speak < Thor
  desc "name", "the name to say hello to"
  def hello(name)
    puts "Hello #{name}"
  end
end

Put a class like that in a file named Thorfile or *.thor into any directory or its tasks directory, and you’ll be able to invoke thor speak:hello Yehuda, and it will print Hello Yehuda. Thor also supports full option parsing:

class Speak < Thor
  desc "name", "the name to say hello to"
  method_options :loudly => false
  def hello(name)
    name.upcase! if options[:loudly]
    puts "Hello #{name}"
  end
end

Interestingly, the thor runner itself is just a thor script. That’s because the following is a valid stand-alone Ruby file that does not require the thor runner:

require 'rubygems'
require 'thor'

class Speak < Thor

  desc "name", "the name to say hello to"
  def hello(name)
    puts "Hello #{name}"
  end
end

Speak.start

You would invoke that script with binary_name hello Yehuda. The thor runner simply uses method_missing on the Thor::Runner class, so that thor foo:bar works, even though there is no foo:bar method in Thor::Runner. Here’s the method_missing method on Thor::Runner:

  def method_missing(meth, *args)
    meth = meth.to_s
    super(meth.to_sym, *args) unless meth.include? ?:

    initialize_thorfiles(meth)
    task = Thor[meth]
    task.parse task.klass.new, ARGV[1..-1]
  end

In the past few weeks, I finally got around to adding some sorely needed features. The first, to transparently wrap thor files in a Thor::Tasks namespace, so, for instance, the Merb < Thor class doesn’t conflict with the Merb module. The second was to add thor bundles (a surprisingly small commit), which allow the packaging of several files into a bundle. In the case of thor bundles, all you need to do is name a directory *.thor (instead of a file) and put a main.thor file inside. Everything will then work as expected.

Finally, thor tasks can be installed systemwide by simply calling thor install filename or thor install http://example.com/foo.thor. All of thor’s features, including bundling, work transparently with installed tasks. As far as thor is concerned, installed gems are available in addition to any local gems, everywhere.

The final bit of news is that SproutCore (which is used in Apple’s MobileMe and iWork.com), which is already built on Merb, will be adopting thor for its build tools. Charles Jolley (of SproutCore) recently submitted quite a few interesting patches yesterday to make thor ready for use with SproutCore, and I expect to announce a 1.0 release in the next few weeks. I plan to fix up global options, make it possible to install bundles from remote locations. Is there something you want in thor before the 1.0 release? Let me know!

With most of the pieces of the puzzle in place, I took today to go back over the codebase and remove artifacts of the refactoring work I’ve been doing, and spent some time renaming variables and methods I’ve created so they more accurately reflect what they’re doing:

• ActionView::Template had a render method on it in which almost every line was an instance_variable_get or send() into ActionView::Base. I (slowly, very very slowly) moved the method over into ActionView::Base.
• Similarly, I moved the partial rendering code out of ActionView::RenderablePartial and onto ActionView::Base as well.
• I reorganized ActionView, creating directories for “render” and “template”, and moving the appropriate files into the directories. This should significantly reduce the amount of visual clutter in the action_view base directory and make it easier to find the part of the code you’re looking for.
• I was finally able to remove ActionView::Base#render_for_parts, which was the temporary shim I created when I started this round of refactoring to force all the callers into ActionView to call in with unified parameters. This was replaced with render_for_template yesterday (renamed to render_template_with_layout today), which means that the conduit between parts of ActionPack is now reduced to Template objects and layouts (also Template objects).
• The reorganization I referenced above also moved common methods into common files. For instance, the ActionView methods related to top-level rendering are in render/rendering.rb, while the ones related to rendering partials are in render/partials.rb.

On another note, I’ve been playing with an interesting memoization technique that lets you declare the methods to be memoized in a block:

class Foo
  extend Memoizable

  memoize do
    def memoed(baz)
      baz * 1000
    end
  end

  def unmemoed(baz)
    baz * 1000
  end
end

After creating this class, Foo.new.memoed will be a memoized method, but creating the memoization will not use alias_method_chain. Instead, it uses normal Ruby structures, and super. You can see the full details in a gist I posted with the code. It definitely gets a bit into the deep end of metaprogramming magic, but the end result is memoized methods that will retain their names in stack traces (probably my biggest personal gripe with alias_method_chain).

Finally, I’m going to start putting together a quick prototype of how Rails3 can be JavaScript agnostic while retaining the existing JS helpers next week some time. I’ve already gotten a lot of good feedback from the jQuery and Mootools communities, and plan to get as much involvement as possible as I move forward. Feel free to ping me (wycats on freenode or gmail) if you have any specific suggestions or just want to chat about the project. More details to follow next week :)

I’ve heard more than a few people express some confusion about what is meant by “public API”, especially since I have lauded Rails for having an excellent user-facing API. How does that jive with the agenda of creating a defined, documented API? 

To clarify, I am applauding Rails for having created a developer interface to the framework that is pleasant to use. A lot of code that is created using the Rails DSL is immensely understandable, even to relatively new users, because of a passionate focus on making it so.

When I talk about “public API” I mean something much more boring and technical. In particular, I mean dividing up the parts of the Rails internals that are meant to be used, documenting them as public, and writing tests that are specifically marked as public and cannot be changed without a darn good reason. At present, while the public-facing API is relatively stable, there is no infrastructure around making it so.

But perhaps more important is the idea of doing the same for plugin developers that Rails has already done for end-users, by forming known, defined, and tested mechanisms for adding or modifying Rails behavior. In the same was as for public-facing functionality, we should document the methods that are intended to be used by plugin developers, create hooks where appropriate, and write tests to ensure that these contracts with plugin developers remain so.

Once this is done, and the private functionality of Rails is well cordoned off from the parts that are intended for public or plugin use, the Rails team will have significantly more latitude to keep the internals of the framework from getting crufty too quickly. We found this approach to be quite useful in Merb, and the plan is to do the same for Rails.

Of course, it can hardly be said that by creating a public/plugin API we will have completely removed the need for anyone to ever monkey-patch Rails internals directly. However, it will be more clear which modifications are likely to survive from version to version, and which modifications will require constant vigilance (and should probably be the subject of a conversation with the Rails core team itself to determine whether a more explicit hook could be added).

In short, when I say “public API”, I mean documentation, testing and commitment.

During work today, I made more progress in unifying the ActionController=>ActionView API:

The API into ActionView is now ActionView::Base#_render_for_template. This is where the refactor was going all along, and with Josh’s help, we were able to really solidify how this should work.

At the moment, there are a few stragglers that may or may not be cleaned up: AV::Base#_render_text and AV::Base#_render_inline. Using _render_inline in particular ensures that render :inline does the same thing in both ActionController and ActionView. This is the code in ActionController that delegates to ActionView for :inline:

if inline = options[:inline]
  render_for_text(@template._render_inline(inline, _pick_layout(layout_name), options))

This is the code in ActionView for rendering inline:

elsif inline = options[:inline]
  _render_inline(inline, layout, options)

As you can see, having a single method allows us to be sure that the code does the same thing in both places. You can compare with the code from before the refactor. ActionController:

elsif inline = options[:inline]
  render_for_text(@template.render(options.merge(:layout => layout)), options[:status])

ActionView:

elsif options[:inline]
  InlineTemplate.new(options[:inline], options[:type]).render(self, options[:locals])

Not particularly terrible, but the mechanism is a bit more obscure, and it’s less obvious that both pieces of code do the same thing. Before the refactor, you can see that ActionController delegated to ActionView’s render method, after twiddling with the options and passing them through uncritically.

Another interesting item in today’s refactor: previously, layout selection happened at the beginning of the render method, before the options were examined. As a result, it was necessary for the template chooser to scan through all the options to determine whether it should fall back to the default layout if no layout was specified. For instance, render :action assumes that a layout should be used by default, while render :text assumes that a layout should not be used. Of course, :layout => “…” can be specified, but the default behavior is different.

In the refactored version, the _pick_layout method is called with information about whether it should fall back to the default, so it no longer has to scan the options. It also means that a layout is never selected if it is not required (render :nothing or RJS, for instance).

After work, I hung out with a friend from New York, but when I got back I decided to do a bit more work on Rails.

Some interesting stuff: I’m experimenting with ConcurrentHash and SafelyMemoizable, which exploits the fact that Ruby implicitly locks instance variable mutations to create a cache that should both be threadsafe and not require locks for almost all cases. This also allows a very simple threadsafe memoization technique for global caches. Again, this is a pretty early experiment, so the details may change some. Note that it’s not as expensive as it looks, because Hash#dup is a shallow clone, so the memory-expensive values will exist only once in memory.

I also cleaned up layouts a bit more, including replacing a somewhat complex mechanism for dealing with default layouts that was implemented via an inherited hook that involved the rules for whether or not a missing layout should raise an error. It still is fairly arcane stuff, but I’m pretty pleased with the revised code. I needed to add some tests for layout :except and layout :only, as there were no tests for that functionality.

The very last thing tonight was to put the final piece in place to have a single place for determining whether a response is exempt from layout. Now that all options that involve rendering templates go through the same method, all of the logic for skipping layout is encapsulated in layout = _pick_layout(*layout) unless tmp.exempt_from_layout?. Putting this piece in place was very personally satisfying.

At this point, I’m pretty sure I’m posting simply to create a personal record of my work. If some of you are enjoying it, that’s a bonus!