Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Rubinius’ Category

How to Marshal Procs Using Rubinius

The primary reason I enjoy working with Rubinius is that it exposes, to Ruby, much of the internal machinery that controls the runtime semantics of the language. Further, it exposes that machinery primarily in order to enable user-facing semantics that are typically implemented in the host language (C for MRI, C and C++ for MacRuby, Java for JRuby) to be implemented in Ruby itself.

There is, of course, quite a bit of low-level functionality in Rubinius implemented in C++, but a surprising number of things are implemented in pure Ruby.

One example is the Binding object. To create a new binding in Rubinius, you call Binding.setup:

def self.setup(variables, code, static_scope, recv=nil)
  bind = allocate()
  bind.self = recv || variables.self
  bind.variables = variables
  bind.code = code
  bind.static_scope = static_scope
  return bind

This method takes a number of more primitive constructs, which I will explain as this article progresses, but we can describe the constructs that make up the high-level Ruby Binding in pure Ruby.

In fact, Rubinius implements Kernel#binding itself in terms of Binding.setup.

def binding
  return Binding.setup(

Yes, you’re reading that right. Rubinius exposes the ability to extract the constructs that make up a binding, one at a time, from a caller’s scope. And this is not just a hack (like Binding.of_caller for a short time in MRI). It’s core to how Rubinius manages eval, which of course makes heavy use of bindings.

Marshalling Procs

For a while, I have wanted the ability to Marshal.dump a proc in Ruby. MRI has historically disallowed it, but there’s nothing conceptually impossible about it. A proc itself is a blob of executable code, a local variable scope (which is just a bunch of pointers to other objects), and a constant lookup scope. Rubinius exposes each of these constructs to Ruby, so Marshaling a proc simply means figuring out how to Marshal each of these constructs.

Let’s take a quick detour to learn about the constructs in question.


Rubinius represents Ruby’s constant lookup scope as a Rubinius::StaticScope object. Perhaps the easiest way to understand it would be to look at Ruby’s built-in Module.nesting function.

module Foo
  p Module.nesting
  module Bar
    p Module.nesting
module Foo::Bar
  p Module.nesting
# Output:
# [Foo]
# [Foo::Bar, Foo]
# [Foo::Bar]

Every execution context in Rubinius has a Rubinius::StaticScope, which may optionally have a parent scope. In general, the top static scope (the static scope with no parent) in any execution context is Object.

Because Rubinius allows us to get the static scope of a calling method, we can implement Module.nesting in Rubinius:

def nesting
  scope = Rubinius::StaticScope.of_sender
  nesting = []
  while scope and scope.module != Object
    nesting << scope.module
    scope = scope.parent

A static scope also has an addition property called current_module, which is used during class_eval to define which module the runtime should add new methods to.

Adding Marshal.dump support to a static scope is therefore quite easy:

class Rubinius::StaticScope
  def marshal_dump
    [@module, @current_module, @parent]
  def marshal_load(array)
    @module, @current_module, @parent = array

These three instance variables are defined as Rubinius slots, which means that they are fully accessible to Ruby as instance variables, but don’t show up in the instance_variables list. As a result, we need to explicitly dump the instance variables that we care about and reload them later.


A compiled method holds the information necessary to execute a blob of Ruby code. Some important parts of a compiled method are its instruction sequence (a list of the compiled instructions for the code), a list of any literals it has access to, names of local variables, its method signature, and a number of other important characteristics.

It’s actually quite a complex structure, but Rubinius has already knows how to convert an in-memory CompiledMethod into a String, as it dumps compiled Ruby files into compiled files as part of its normal operation. There is one small caveat: this String form that Rubinius uses for its compiled method does not include its static scope, so we will need to include the static scope separately in the marshaled form. Since we already told Rubinius how to marshal a static scope, this is easy.

class Rubinius::CompiledMethod
  def _dump(depth)
  def self._load(string)
    scope, dump = Marshal.load(string)
    cm =
    cm.scope = scope


A variable scope represents the state of the current execution context. It contains all of the local variables in the current scope, the execution context currently in scope, the current self, and several other characteristics.

I wrote about the variable scope before. It’s one of my favorite Rubinius constructs, because it provides a ton of useful runtime information to Ruby that is usually locked away inside the native implementation.

Dumping and loading the VariableScope is also easy:

class VariableScope
  def _dump(depth)
    Marshal.dump([@method, @module, @parent, @self, nil, locals])
  def self._load(string)
    VariableScope.synthesize *Marshal.load(string)

The synthesize method is new to Rubinius master; getting a new variable scope previously required synthesizing its locals using class_eval, and the new method is better.


A Proc is basically nothing but a wrapper around a Rubinius::BlockEnvironment, which wraps up all of the objects we’ve been working with so far. Its scope attribute is a VariableScope and its code attribute is a CompiledMethod.

Dumping it should be quite familiar by now.

class BlockEnvironment
  def marshal_dump
    [@scope, @code]
  def marshal_load(array)
    scope, code = *array
    under_context scope, code

The only thing new here is the under_context method, which gives a BlockEnvironment its variable scope and compiled method. Note that we dumped the static scope along with the compiled method above.


Finally, a Proc is just a wrapper around a BlockEnvironment, so dumping it is easy:

class Proc
  def _dump(depth)
  def self._load(string)
    block = Marshal.load(string)

The __from_block__ method constructs a new Proc from a BlockEnvironment.

So there you have it. Dumping and reloading Proc objects in pure Ruby using Rubinius! (the full source is at

What’s Up With All These Changes in Rails?

Yesterday, there was a blog post entitled “What the Hell is Happening to Rails” that stayed at the number one spot on Hacker News for quite a while. The post and many (but not most) the comments on the post reflect deep-seated concern about the recent direction of Rails. Others have addressed the core question about change in the framework, but I’d like to address questions about specific changes that came up in the post and comments.

The intent of this post is not to nitpick the specific arguments that were made, or to address the larger question of how much change is appropriate in the framework, but rather to provide some background on some of the changes that have been made since Rails 2.3 and to explain why we made them.

Block Helpers

I too get a feeling of “change for the sake of change” from Rails at times. That’s obviously not something they’re doing, as all the changes have some motivation, but at times it feels a bit like churn.
At one point in time, you did forms like <%= form …. and then they switched to <% form …. do and now they’ve switched back to <%= form … do again.
Also, the upgrade to Rails 3 is not an easy one. Yeah, you get some nice stuff, but because it’s so painful, it’s not happening for a lot of people, which is causing more problems.

Prior to Rails 3.0, Rails never used <= form_for because it was technically very difficult to make it work. I wrote a post about it in Summer 2009 that walked through the technical problems. The short version is that every ERB template is compiled into a Ruby method, and reliably converting &lt%= with blocks proved to be extremely complicated.

However, knowing when to use <% and when to use <= caused major issues for new developers, and it was a major 3.0 priority to make this work. In addition, because of the backwards compatibility issue, we went to extreme Ruby-exploiting lengths to enable deprecation warnings about the change, so that we could fix the issue for new apps, but also not break old apps.

The night José and I figured out how to do this (the Engine Yard party at Mountain West RubyConf 2009), we were pretty close to coming to the conclusion that it couldn’t be done short of using a full language parser in ERB, which should give you some sense of how vexing a problem it was for us.


The general disregard for improving performance, inherited from Ruby, is also something endemic from a while back.

Yes, there have been some performance regressions in Rails 3.0. However, the idea that the Rails core team doesn’t care about performance, and neither does the Ruby team doesn’t pass the smell test.

Aaron Patterson, the newest core team member, worked full time for almost a year to get the new ActiveRecord backend into decent performance shape. Totally new code often comes with some performance regressions, and the changes to ActiveRecord were important and a long-time coming. Many of us didn’t see the magnitude of the initial problem until it was too late for 3.0, but we take the problem extremely seriously.

Ruby core (“MRI”) itself has sunk an enormous amount of time into performance improvements in Ruby 1.9, going so far as to completely rewrite the core VM from scratch. This resulted in significant performance improvements, and the Ruby team continues to work on improvements to performance and to core systems like the garbage collector.

The Ruby C API poses some long-term problems for the upper-bound of Ruby performance improvements, but the JRuby and Rubinius projects are showing how you can use Ruby C extensions inside a state-of-the-art virtual machine. Indeed, the JRuby and Rubinius projects show that the Ruby community both cares about, and is willing to invest significant energy into improving the performance of Ruby.

Heroku and the Asset Pipeline

The assets pipeline feels horrible, it’s really slow. I upgraded to Rails 3.1rc, realized it fights with Heroku unless upgrading to the Cedar stack

The problem with Heroku is that the default Gemfile that comes with Rails 3.1 currently requires a JavaScript runtime to boot, even in production. This is clearly wrong and will be fixed posthaste. Requiring Rails apps to have node or a compiled v8 in production is an unacceptable burden.

On the flip side, the execjs gem, which manages compiling CoffeeScript (and more importantly minifying your JavaScript), is actually a pretty smart piece of work. It turns out that both Mac OSX and Windows ship with usable JavaScript binaries, so in development, most Rails users will already have a JavaScript library ready to use.

It’s worth noting that a JavaScript engine is needed to run “uglify.js”, the most popular and most effective JavaScript minifier. It is best practice to minify your JavaScript before deployment, so you can feel free to format and comment your code as you like without effecting the payload. You can learn more about minification in this excellent post by Steve Souders from a few years back. Rails adding minification by default is an unambiguous improvement in the workflow of Rails applications, because it makes it easy (almost invisible) to do something that everyone should be doing, but which has previously been something of a pain.

Again, making node a dependency in production is clearly wrong, and will be removed before the final release.

Change for Change’s Sake

The problem with Rails is not the pace of change so much as the wild changes of direction it takes, sometimes introducing serious performance degradations into official releases. Sometimes it’s hard to see a guiding core philosophy other than the fact they want to be on the shiny edge

When Rails shipped, it came with a number of defaults:

  • ActiveRecord
  • Test::Unit
  • ERB
  • Prototype

Since 2004, alternatives to all of those options have been created, and in many cases (Rspec, jQuery, Haml) have become somewhat popular. Rails 3.0 made no changes to the defaults, despite much clamoring for a change from Prototype to jQuery. Rails 3.1 changed to jQuery only when it became overwhelmingly clear that jQuery was the de facto standard on the web.

As someone who has sat in on many discussions about changing defaults, I can tell you that defaults in Rails are not changed lightly, and certainly not “to be on the shiny edge.” In fact, I think that Rails is more conservative than most would expect in changing defaults.

Right, but the problem is that there doesn’t seem to be a ‘right’ way, That’s the problem.

We were all prototype a few years ago, now it jquery … we (well I) hadn’t heard of coffeescript till a few months ago and now its a default option in rails, The way we were constructing ActiveRecord finders had been set all through Rails 2, now we’ve changed it, the way we dealt with gems was set all through rails 2 now its changed completely in Rails 3.

I like change, I like staying on the cutting edge of web technologies, but I don’t want to learn something, only to discard it and re-do it completely to bring it up to date with a new way of doing things all the time.

First, jQuery has become the de facto standard on the web. As I said earlier, Rails resisted making this change in 3.0, despite a lot of popular demand, and the change to jQuery is actually an example of the stability of Rails’ default choices over time, rather than the opposite.

Changes to gem handling evolved as the Rails community evolved to use gems more. In Rails 1, all extensions were implemented as plugins that got pulled out of svn and dumped into your project. This didn’t allow for versioning or dependencies, so Rails 2 introduced first-class support for Rails plugins as gems.

During the Rails 2 series, Rails added a dependency on Rack, which caused serious problems when Rails was used with other gems that rely on Rack, due to the way that raw Rubygems handles dependency activation. Because Rails 3 uses gem dependencies more extensively, we spent a year building bundler, which adds per-application dependency resolution to Rubygems. This was simply the natural evolution of the way that Rails has used dependencies over time.

The addition of CoffeeScript is interesting, because it’s a pretty young technology, but it’s also not really different from shipping a new template handler. When you create a new Rails app, the JavaScript file is a regular JS file, and the asset compilation and dependency support does not require CoffeeScript. Shipping CoffeeScript is basically like shipping Builder: should you want it, it’s there for you. Since we want to support minification out of the box anyway, CoffeeScript doesn’t add any new requirements. And since it’s just there in the default Gemfile, as opposed to included in Rails proper like Builder, turning it off (if you really want to) is as simple as removing a line in your Gemfile. Nothing scary here.

ActiveRecord Changes

The way we were constructing ActiveRecord finders had been set all through Rails 2, now we’ve changed it

The Rails core team does seem to treat the project as if it’s a personal playground

One of the biggest problems with ActiveRecord was the way it internally used Strings to represent queries. This meant that changes to queries often required gsubing String to make simple changes. Internally, it was a mess, and it expressed itself in the public API in how conditions were generated, and more importantly how named scopes were created.

One goal of the improvements in Rails 3 was to get rid of the ad-hoc query generation in Rails 2 and replace it with something better. ActiveRelation, this library, was literally multiple years in the making, and a large amount of the energy in the Rails 3.0 process was spent on integrating ActiveRelation.

From a user-facing perspective, we wanted to unify all of the different ways that queries were made. This means that scopes, class methods, and one-time-use queries all use the same API. As in the <= case, a significant amount of effort was spent on backwards compatibility with Rails 2.3. In fact, we decided to hold onto support for the old API as least as long as Rails 3.2, in order to soften the community transition to the new API.

In general, we were quite careful about backwards compatibility in Rails 3.0, and while a project as complex as Rails was not going to be perfect in this regard, characterizing the way we handled this as “irresponsible” or “playground” disregards the tremendous amount of work and gymnastics that the core team and community contributors put into supporting Rails 2.3 APIs across the entire codebase when changes were made.

Getting Comfortable With Rubinius’ Pure-Ruby Internals

You probably know that Rubinius is a Ruby whose implementation is mostly written in Ruby. While that sounds nice in theory, you may not know what that means in practice. Over the past several years, I’ve contributed on and off to Rubinius, and feel that as Rubinius has matured since the 1.0 release, a lot of its promise has gelled.

One of the great things about Rubinius is that it exposes, as first-class concepts, a lot of things that are merely implicit in other Ruby implementations. For instance, Ruby methods have a variable scope which is implicitly accessible by blocks created in the scope. Rubinius exposes that scope as Rubinus::VariableScope. It also exposes Ruby bindings as richer objects that you can create yourself and use. In this post, I will talk about Rubinius::VariableScope, how it works, and how it fits into the Rubinius codebase.


For those of you who don’t know much about Rubinius, it’s a Ruby implementation whose core classes are mostly written in Ruby. Some functionality, such as the core of the object model, as well as low-level methods, are written in C++ and exposed into Ruby via its primitive system. However, the functionality is typically fairly low-level, and the vast majority of the functionality is written on top of those primitives. For example, a Ruby String in Rubinius is a mostly pure-Ruby object with a backing ByteArray, and ByteArray’s implementation is mostly written as a primitive.

For instance, String’s allocate method looks like this:

def self.allocate
  str = super()
  str.__data__ =
  str.num_bytes = 0
  str.characters = 0

If you just want to get to the meat of the matter, feel free to skip this section and go directly to the Internals section below.

In general, Rubinius’ code is broken up into three stages. The first stage, called alpha, creates just enough raw Ruby constructs to get to the next stage, the bootstrap stage. It defines methods like Class#new, Kernel#__send__, Kernel#raise, Kernel#clone, basic methods on Module, Object, String, and Symbol, and some classes on the Rubinius module, used for internal use. It’s a pretty short file, weighing it at under 1,000 lines, and I’d encourage you to read it through. It is located in kernel/alpha.rb.

Next, the bootstrap stage creates the basic methods needed on many of Ruby’s core classes. In some cases, it defines a simpler version of a common structure, like Rubinius::CompactLookupTable and Rubinius::LookupTable instead of Hash. It also contains Rubinius constructs like the Rubinius::CompiledMethod, Rubinius::MethodTable, Rubinius::StaticScope and Rubinius::VariableScope. Many of these methods are defined as primitives. Primitive methods are hooked up in Ruby; check out the bootstrap definition of Rubinius::VariableScope:

module Rubinius
  class VariableScope
    def self.of_sender
      Ruby.primitive :variable_scope_of_sender
      raise PrimitiveFailure, "Unable to get VariableScope of sender"
    def self.current
      Ruby.primitive :variable_scope_current
      raise PrimitiveFailure, "Unable to get current VariableScope"
    def locals
      Ruby.primitive :variable_scope_locals
      raise PrimitiveFailure, "Unable to get VariableScope locals"
    # To handle Module#private, protected
    attr_accessor :method_visibility

The bootstrap definitions are in kernel/bootstrap. The file kernel/bootstrap/load_order.txt defines the order that bootstrap methods are loaded.

Next, Rubinius fleshes out the core classes, mostly using Ruby methods. There is some primitive use in this stage, but it’s fairly few and far between. For instance, the Hash class is written entirely in Ruby, using a Rubinius::Tuple for its entries. This code can be found in kernel/common.

Some additional code lives in kernel/platform, which contains platform-specific functionality, like File and POSIX functionality, and kernel/delta, which runs after kernel/common. For the full description of the bootstrapping process, check out Rubinius: Bootstrapping.


One of the really nice things about Rubinius is that many concepts that are internal in traditional Ruby are exposed directly into the Ruby runtime. In many cases, this is so that more of Rubinius can be implemented in pure Ruby. In other words, if a core method in MRI needs some behavior, it is often implemented directly in C and unavailable to custom Ruby code.

One good example is the behavior of $1 in a Ruby method. If you call the =~ method, MRI’s internal C code will walk back up to the caller and set an internal match object on it. If you write your own method that uses the =~ method, there is no way for you to set the match object on the caller. However, since Rubinius itself implements these methods in Ruby, it needs a way to perform this operation directly from Ruby code. Check out the Rubinius implementation of =~:

class Regex
  def =~(str)
    # unless str.nil? because it's nil and only nil, not false.
    str = StringValue(str) unless str.nil?
    match = match_from(str, 0)
    if match
      Regexp.last_match = match
      return match.begin(0)
      Regexp.last_match = nil
      return nil

There are a few interesting things going on here. First, Rubinius calls StringValue(str), which is a Ruby method that implements Ruby’s String coercion protocol. If you follow the method, you will find that it calls Type.coerce_to(obj, String, :to_str), which then, in pure-Ruby, coerces the object into a String. You can use this method anywhere in your code if you want to make use of the String coercion protocol with the standard error messages.

Next, Rubinius actually invokes the match call, which ends up terminating in a primitive called search_region, which binds directly into the Oniguruma regular expression engine. The interesting part is next: Regexp.last_match = match. This code sets the $1 property on the caller, from Ruby. This means that if you write a custom method involving regular expressions and want it to obey the normal $1 protocol, you can, because if Rubinius needs functionality like this, it is by definition exposed to Ruby.


One result of this is that you can get access to internal structures like the current “variable scope”. A variable scope is an object that wraps the notion of which locals are available (both real locals and locals defined using eval), as well as the current contents of each of those variables. If the current context is a block, a VariableScope also has a pointer to the parent VariableScope. You can get access to the current VariableScope by using VariableScope.current.

def foo
  x = 1
  p [:root, Rubinius::VariableScope.current.locals]
  ["a", "b", "c"].each do |item|
    p [:parent, Rubinius::VariableScope.current.parent.locals]
    p [:item, Rubinius::VariableScope.current.locals]

The output will be:

[:root, #<Rubinius::Tuple: 1>]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "a">]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "b">]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "c">]

A Rubinius::Tuple is a Ruby object that is similar to an Array, but is fixed-size, simpler, and usually more performant. The VariableScope has a number of additional useful methods, which you can learn about by reading kernel/bootstrap/variable_scope.rb and kernel/common/variable_scope.rb. One example: locals_layout gives you the names of the local variables for each slot in the tuple. You can also find out the current visibility of methods that would be declared in the scope (for instance, if you call private the VariableScope after that call will have method_visibility of :private).

Perhaps the most exciting thing about VariableScope is that you can look up the VariableScope of your calling method. Check this out:

def root
  x = 1
  ["a", "b", "c"].each do |item|
def call_me
  sender = Rubinius::VariableScope.of_sender
  p [:parent, sender.parent.locals]
  p [:item, sender.locals]

In this case, you can get the VariableScope of the caller, and get any live information from it. In this case, you can tell that the method was called from inside a block, what the locals inside the block contain, and what block contexts exist up to the method root. This is extremely powerful, and gives you a level of introspection into a running Ruby program heretofore not possible (or very difficult). Better yet, the existence of this feature doesn’t slow down the rest of the program, because Rubinius uses it internally for common Ruby functionality like the implementation of private and protected.

Implementing caller_locals

Now that we understand the VariableScope object, what if we want to print out a list of all of the local variables in the caller of a method.

def main
  a = "Hello"
  [1,2,3].each do |b|
    say_hello do
      puts "#{a}: #{b}"
def say_hello
  # this isn't working as we expect, and we want to look at
  # the local variables of the calling method to see what's
  # going on
  puts "Goodbye"

Let’s implement a method called caller_locals which will give us all of the local variables in the calling method. We will not be able to use VariableScope.of_sender, since that only goes one level back, and since we will implement caller_locals in Ruby, we will need to go two levels back. Thankfully, Rubinius::VM.backtrace has the information we need.

Rubinius::VM.backtrace takes two parameters. The first is the number of levels back to start from. In this case we want to start two levels back (the caller will be the say_hello method, and we want its caller). The second parameter indicates whether the Location objects should contain the VariableScope, which is what we want.

NOTE: Yes, you read that right. In Rubinius, you can obtain a properly object-oriented backtrace object, where each entry is a Location object. Check out kernel/common/location.rb to see the definition of the class.

In this case, if we pass true, to Rubinius::VM.backtrace, each Location will have a method called variables, which is a Rubinus::VariableScope object.

def caller_locals
  variables = Rubinius::VM.backtrace(2, true).first.variables

The next thing you need to know is that each block in a method has its own VariableScope and a parent pointing at the VariableScope for the parent block.

Each VariableScope also has a pointer to its method, an object that, among other things, keeps a list of the local variable names. These lists are both ordered the same.

To collect up the locals from the caller, we will walk this chain of VariableScope objects until we get nil.

def caller_locals
  variables = Rubinius::VM.backtrace(2, true).first.variables
  locals = []
  while variables
    names  = variables.method.local_names
    values = variables.locals
    locals << Hash[*]
    variables = variables.parent

This will return an Array of local variable Hashes, starting from the caller’s direct scope and working up to the method body itself. Let’s change the original method to use caller_locals:

def say_hello
  # this isn't working as we expect, and we want to look at
  # the local variables of the calling method to see what's
  # going on. Let's use caller_locals.
  p caller_locals
  puts "Goodbye"

If we run the program now, the output will be:

[{:b=>1}, {:a=>"Hello"}]
[{:b=>2}, {:a=>"Hello"}]
[{:b=>3}, {:a=>"Hello"}]

This is just the tip of the iceberg of what is possible because of Rubinius’ architecture. To be clear, these features are not accidents; because so much of Rubinius (including things like backtraces), it is forced to expose details of the internals to normal Ruby program. And because the Rubinius kernel is real Ruby (and not some kind of special Ruby subset), anything that works in the kernel will work in your Ruby code as well.