Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

How to Marshal Procs Using Rubinius

The primary reason I enjoy working with Rubinius is that it exposes, to Ruby, much of the internal machinery that controls the runtime semantics of the language. Further, it exposes that machinery primarily in order to enable user-facing semantics that are typically implemented in the host language (C for MRI, C and C++ for MacRuby, Java for JRuby) to be implemented in Ruby itself.

There is, of course, quite a bit of low-level functionality in Rubinius implemented in C++, but a surprising number of things are implemented in pure Ruby.

One example is the Binding object. To create a new binding in Rubinius, you call Binding.setup:

def self.setup(variables, code, static_scope, recv=nil)
  bind = allocate()
 
  bind.self = recv || variables.self
  bind.variables = variables
  bind.code = code
  bind.static_scope = static_scope
  return bind
end

This method takes a number of more primitive constructs, which I will explain as this article progresses, but we can describe the constructs that make up the high-level Ruby Binding in pure Ruby.

In fact, Rubinius implements Kernel#binding itself in terms of Binding.setup.

def binding
  return Binding.setup(
    Rubinius::VariableScope.of_sender,
    Rubinius::CompiledMethod.of_sender,
    Rubinius::StaticScope.of_sender,
    self)
end

Yes, you’re reading that right. Rubinius exposes the ability to extract the constructs that make up a binding, one at a time, from a caller’s scope. And this is not just a hack (like Binding.of_caller for a short time in MRI). It’s core to how Rubinius manages eval, which of course makes heavy use of bindings.

Marshalling Procs

For a while, I have wanted the ability to Marshal.dump a proc in Ruby. MRI has historically disallowed it, but there’s nothing conceptually impossible about it. A proc itself is a blob of executable code, a local variable scope (which is just a bunch of pointers to other objects), and a constant lookup scope. Rubinius exposes each of these constructs to Ruby, so Marshaling a proc simply means figuring out how to Marshal each of these constructs.

Let’s take a quick detour to learn about the constructs in question.

Rubinius::StaticScope

Rubinius represents Ruby’s constant lookup scope as a Rubinius::StaticScope object. Perhaps the easiest way to understand it would be to look at Ruby’s built-in Module.nesting function.

module Foo
  p Module.nesting
 
  module Bar
    p Module.nesting
  end
end
 
module Foo::Bar
  p Module.nesting
end
 
# Output:
# [Foo]
# [Foo::Bar, Foo]
# [Foo::Bar]

Every execution context in Rubinius has a Rubinius::StaticScope, which may optionally have a parent scope. In general, the top static scope (the static scope with no parent) in any execution context is Object.

Because Rubinius allows us to get the static scope of a calling method, we can implement Module.nesting in Rubinius:

def nesting
  scope = Rubinius::StaticScope.of_sender
  nesting = []
  while scope and scope.module != Object
    nesting << scope.module
    scope = scope.parent
  end
  nesting
end

A static scope also has an addition property called current_module, which is used during class_eval to define which module the runtime should add new methods to.

Adding Marshal.dump support to a static scope is therefore quite easy:

class Rubinius::StaticScope
  def marshal_dump
    [@module, @current_module, @parent]
  end
 
  def marshal_load(array)
    @module, @current_module, @parent = array
  end
end

These three instance variables are defined as Rubinius slots, which means that they are fully accessible to Ruby as instance variables, but don’t show up in the instance_variables list. As a result, we need to explicitly dump the instance variables that we care about and reload them later.

Rubinius::CompiledMethod

A compiled method holds the information necessary to execute a blob of Ruby code. Some important parts of a compiled method are its instruction sequence (a list of the compiled instructions for the code), a list of any literals it has access to, names of local variables, its method signature, and a number of other important characteristics.

It’s actually quite a complex structure, but Rubinius has already knows how to convert an in-memory CompiledMethod into a String, as it dumps compiled Ruby files into compiled files as part of its normal operation. There is one small caveat: this String form that Rubinius uses for its compiled method does not include its static scope, so we will need to include the static scope separately in the marshaled form. Since we already told Rubinius how to marshal a static scope, this is easy.

class Rubinius::CompiledMethod
  def _dump(depth)
    Marshal.dump([@scope, Rubinius::CompiledFile::Marshal.new.marshal(self)])
  end
 
  def self._load(string)
    scope, dump = Marshal.load(string)
    cm = Rubinius::CompiledFile::Marshal.new.unmarshal(dump)
    cm.scope = scope
    cm
  end
end

Rubinius::VariableScope

A variable scope represents the state of the current execution context. It contains all of the local variables in the current scope, the execution context currently in scope, the current self, and several other characteristics.

I wrote about the variable scope before. It’s one of my favorite Rubinius constructs, because it provides a ton of useful runtime information to Ruby that is usually locked away inside the native implementation.

Dumping and loading the VariableScope is also easy:

class VariableScope
  def _dump(depth)
    Marshal.dump([@method, @module, @parent, @self, nil, locals])
  end
 
  def self._load(string)
    VariableScope.synthesize *Marshal.load(string)
  end
end

The synthesize method is new to Rubinius master; getting a new variable scope previously required synthesizing its locals using class_eval, and the new method is better.

Rubinius::BlockEnvironment

A Proc is basically nothing but a wrapper around a Rubinius::BlockEnvironment, which wraps up all of the objects we’ve been working with so far. Its scope attribute is a VariableScope and its code attribute is a CompiledMethod.

Dumping it should be quite familiar by now.

class BlockEnvironment
  def marshal_dump
    [@scope, @code]
  end
 
  def marshal_load(array)
    scope, code = *array
    under_context scope, code
  end
end

The only thing new here is the under_context method, which gives a BlockEnvironment its variable scope and compiled method. Note that we dumped the static scope along with the compiled method above.

Proc

Finally, a Proc is just a wrapper around a BlockEnvironment, so dumping it is easy:

class Proc
  def _dump(depth)
    Marshal.dump(@block)
  end
 
  def self._load(string)
    block = Marshal.load(string)
    self.__from_block__(block)
  end
end

The __from_block__ method constructs a new Proc from a BlockEnvironment.

So there you have it. Dumping and reloading Proc objects in pure Ruby using Rubinius! (the full source is at https://gist.github.com/1378518).

A Proposal for ES.next Proposals

Over the past few years, I have occasionally expressed frustration (in public and private) about the process for approving new features to the next edition of ECMAScript. In short, the process is extremely academic in nature, and is peppered with inside baseball terms that make it nearly impossible for lay developers to provide feedback about proposed new features. In general, this frustration was met with a general assumption that the current process works the way it does for good reason, and that academic descriptions of the new features was the correct (and only) way to properly discuss them.

I have nothing against new features being described in the language of implementors, but I would like to propose some additions to the current process that would make it significantly easier for language consumers (especially framework and library implementors) to provide timely feedback about the proposal in the hope of making an impact before it’s too late.

I would like proposals for new features to have the following elements, in addition to whatever elements are already normally included (such as BNF for any new syntax).

What Problem Are We Solving?

At a high level, what can language users do now that they could not do before. In some cases, proposals may provide simpler or more convenient ways to achieve already-possible goals. These kinds of proposals are often just as important. For example, there is a current proposal to provide a new syntax for class creation. In this case, the new class syntax significantly improves the experience of building a common JavaScript construct.

Provide Non-Trivial Example Code Showing the Problem

If the proposal is solving a problem that exists in the wild, it should be possible to identify non-trivial examples of the problem rearing its head. At the very least, the process of identifying or synthesizing these examples will help language users understand what real-world problems the proposal is attempting to solve. At best, finding real-world examples will help refine the proposal at an early stage.

Show How the Example Code is Improved With the Proposal

After identifying or synthesizing example code to illustrate the problem, show how the problem would be improved if the proposal was accepted. In some cases, the problem is as simple as “needing a large library to perform this operation” and the solution is “building common functionality into the language”. In an example from a related field, the DOM library, the problem addressed by querySelectorAll was “many incompatible implementations of a CSS3 selector library”. The solution was to build the functionality into the browser.

In this case, a mistake in the querySelectorAll specification, which was resolved by the addition of queryScopedSelectorAll in the Selectors API Level 2 could have been addressed ahead of time by evaluating real-life code using selector engine libraries. Of course, the DOM library is not the same as the language specification, so the example is merely an analogy.

What are Alternate ES3 (or ES5) Solutions to the Same Problem?

Simply enumerate the ways that existing JavaScript developers have attempted to resolve the problem without language support. In small language changes, this overlaps considerably with the previous questions. By having proposal authors enumerate existing solutions to the problem, it will be easier for language users to identify the scope of the solution.

This will allow language users to provide feedback about how the solution on offer stacks up compared to existing pure-JS solutions.

Are there any restrictions that do not exist in original pure-JS solutions?

Are there any restrictions in the proposal that limit its utility as a solution to the problem in question, especially if those restrictions do not apply to solutions currently used by language users.

If the Proposal is For New Syntax, What if Anything Does it Desugar To?

Also, if the proposal desugars, why choose this particular desugaring as opposed to alternatives?

If New Syntax, Can it Be Compiled to ES3 (or ES5)?

If the proposal can desugar to an older version of the specification, can a source-to-source translator be written? If so, is there a reference implementation of a source-to-source translator written in that version?

By writing such a source-to-source translator, existing language users can experiment with the new syntax easily in a browser environment without requiring a separate compilation pass. This also allows users to build an in-browser translation UI (similar to try CoffeeScript), which can improve general understanding of the new syntax and produce important feedback.

To be more specific, what I would like to see here is a general-purpose source-to-source translation engine written in ES3 with a mechanism for plugging in translation passes for specific features. If new features come with translation passes, it would be trivial for language users to try out new features incrementally in production applications (with a nice development-time workflow). This would provide usability feedback at an early enough stage for it to be useful.

If a New Standard Library, Can it Be Polyfilled to ES3 (or ES5)?

If the proposal is for a new library whose syntax is valid in an earlier version of the specification, can it be implemented in terms of the existing primitives available in that version. If necessary, primitives not defined by the language, but provided historically by web browsers, can be used instead. The goal is to provide shims for older browsers so that a much broader group of people can experiment with the APIs and provide feedback.

In general, new libraries that parse in older versions of the specifications should also come with a well-defined mechanism to detect whether the feature is present, to make it easy for library and framework vendors, as well as the general public, to opt their users into the new features where appropriate.

Even if a fully backwards-compatible shim cannot be provided, it is still useful to provide a partial implementation together with a feature detection mechanism. At the very least, error-checking shims can be useful, so language users can easily understand the interface to the proposed library.

Does the Proposal Change any Existing Semantics?

In some cases, the proposal unavoidably changes existing semantics. For example, ES5 changed the semantics of an indirect call to eval (a call to an alias to eval, such as x = eval; x('some code') to use the global environment for the evaluated code. In ES3, indirect calls to eval behaved the same as direct calls to eval.

These cases are rare, and in most cases, require an explicit opt-in (such as the directive "use strict";).

When such changes are made, especially when they do not require an opt-in, they should be explicitly called out in the proposal to gather feedback about their likely impact on existing code. Even when they require an opt-in, information about the frequency of their use could be useful to assess the difficulty of opting in.

Since these opt-ins often enable new features as well as changing existing semantics, understanding the impact of the opt-in on existing code would help language users assess their overall utility and timeframe for adoption. This information could also help drive these decisions.

New Hope for The Ruby Specification

For a few years, a group of Japanese academics have been working on formalizing the Ruby programming language into a specification they hoped would be accepted by ISO. From time to time, I have read through it, and I had one major concern.

Because Ruby 1.9 was still in a lot of flux when they were drafting the specification, the authors left a lot of details out. Unfortunately, some of these details are extremely important. Here’s one example, from the specification of String#[]:

Behavior:
 
a) If the length of args is 0 or larger than 2, raise a
   direct instance of the class ArgumentError.
b) Let P be the first element of args. Let n be the length
   of the receiver.
c) If P is an instance of the class Integer, let b be the
   value of P.
   1) If the length of args is 1:
      i)   If b is smaller than 0, increment b by n. If b is
           still smaller than 0, return nil.
      ii)  If b >= n, return nil.
      iii) Create an instance of the class Object which
           represents the bth character of
           the receiver and return this instance.

The important bit here is c(1)(iii), which says to create “an instance of the class Object which represents the btw character of the receiver”. The reason for this ambiguity, as best as I can determine, is that Ruby 1.8 and Ruby 1.9 differ on the behavior:

1.8 >> "hello"[1]
=> 101
 
1.9 >> "hello"[1]
=> "e"

Of course, neither of these results in a direct instance of the class Object, but since Fixnums and Strings are both “instances of the class Object”, this is technically true. Unfortunately, any real-life Ruby code will need to know what actual object this method will return.

Another very common reason for unspecified behaviors is a failure to specify Ruby’s coercion protocol, so String#+ is unspecified if the other is not a String, even though all Ruby implementations will call to_str on the other to attempt to coerce it. The coercion protocol has been underspecified for a long time, and it’s understandable that the group punted on it, but because it is heavily relied on by real-life code, it is important that we actually describe the behavior.

This week, I am in Matsue in Japan for RubyWorld, and I was glad to learn that the group working on the ISO specification sees the current work as a first step that will continue with a more rigid specification of currently “unspecified” behavior based on Ruby 1.9.

The word “unspecified” appears 170 times in the current draft of the Ruby specification. I hope that the next version will eliminate most if not all of these unspecified behaviors in favor of explicit behavior or explicitly requiring an exception to be thrown. In cases that actually differ between implementations (for instance, Rubinius allows Class to be subclassed), I would hope that these unspecified behaviors be the subject of some discussion at the implementor level.

In any event, I am thrilled at the news that the Ruby specification will become less ambiguous in the future!

Understanding “Prototypes” in JavaScript

For the purposes of this post, I will be talking about JavaScript objects using syntax defined in ECMAScript 5.1. The basic semantics existed in Edition 3, but they were not well exposed.

A Whole New Object

In JavaScript, objects are pairs of keys and values (in Ruby, this structure is called a Hash; in Python, it’s called a dictionary). For example, if I wanted to describe my name, I could have an object with two keys: `firstName` would point to “Yehuda” and `lastName` would point to “Katz”. Keys in a JavaScript object are Strings.

To create the simplest new object in JavaScript, you can use Object.create:

var person = Object.create(null); // this creates an empty objects

Why didn’t we just use var person = {};? Stick with me! To look up a value in the object by key, use bracket notation. If there is no value for the key in question, JavaScript will return `undefined`.

person['name'] // undefined

If the String is a valid identifier[1], you can use the dot form:

person.name // undefined

[1] in general, an identifier starts with a unicode letter, $, _, followed by any of the starting characters or numbers. A valid identifier must also not be a reserved word. There are other allowed characters, such as unicode combining marks, unicode connecting punctuation, and unicode escape sequences. Check out the spec for the full details

Adding values

So now you have an empty object. Not that useful, eh? Before we can add some properties, we need to understand what a property (what the spec calls a “named data property”) looks like in JavaScript.

Obviously, a property has a name and a value. In addition, a property can be enumerable, configurable and writable. If a value is enumerable, it will show up when enumerating over an object using a for(prop in obj) loop. If a property is writable, you can replace it. If a property is configurable, you can delete it or change its other attributes.

In general, when we create a new property, we will want it to be enumerable, configurable, and writable. In fact, prior to ECMAScript 5, that was the only kind of property a user could create directly.

We can add a property to an object using Object.defineProperty. Let’s add a first name and last name to our empty object:

var person = Object.create(null);
Object.defineProperty(person, 'firstName', {
  value: "Yehuda",
  writable: true,
  enumerable: true,
  configurable: true
});
 
Object.defineProperty(person, 'lastName', {
  value: "Katz",
  writable: true,
  enumerable: true,
  configurable: true
});

Obviously, this is extremely verbose. We can make it a bit less verbose by eliminating the common defaults:

var config = {
  writable: true,
  enumerable: true,
  configurable: true
};
 
var defineProperty = function(obj, name, value) {
  config.value = value;
  Object.defineProperty(obj, name, config);
}
 
var person = Object.create(null);
defineProperty(person, 'firstName', "Yehuda");
defineProperty(person, 'lastName',   "Katz");

Still, this is pretty ugly to create a simple property list. Before we can get to a prettier solution, we will need to add another weapon to our JavaScript object arsenal.

Prototypes

So far, we’ve talked about objects as simple pairs of keys and values. In fact, JavaScript objects also have one additional attribute: a pointer to another object. We call this pointer the object’s prototype. If you try to look up a key on an object and it is not found, JavaScript will look for it in the prototype. It will follow the “prototype chain” until it sees a null value. In that case, it returns undefined.

You’ll recall that we created a new object by invoking Object.create(null). The parameter tells JavaScript what it should set as the Object’s prototype. You can look up an object’s prototype by using Object.getPrototypeOf:

var man = Object.create(null);
defineProperty(man, 'sex', "male");
 
var yehuda = Object.create(man);
defineProperty(yehuda, 'firstName', "Yehuda");
defineProperty(yehuda, 'lastName', "Katz");
 
yehuda.sex       // "male"
yehuda.firstName // "Yehuda"
yehuda.lastName  // "Katz"
 
Object.getPrototypeOf(yehuda) // returns the man object

We can also add functions that we share across many objects this way:

var person = Object.create(null);
defineProperty(person, 'fullName', function() {
  return this.firstName + ' ' + this.lastName;
});
 
// this time, let's make man's prototype person, so all
// men share the fullName function
var man = Object.create(person);
defineProperty(man, 'sex', "male");
 
var yehuda = Object.create(man);
defineProperty(yehuda, 'firstName', "Yehuda");
defineProperty(yehuda, 'lastName', "Katz");
 
yehuda.sex        // "male"
yehuda.fullName() // "Yehuda Katz"

Setting Properties

Since creating a new writable, configurable, enumerable property is pretty common, JavaScript makes it easy to do so using assignment syntax. Let’s update the previous example using assignment instead of defineProperty:

var person = Object.create(null);
 
// instead of using defineProperty and specifying writable,
// configurable, and enumerable, we can just assign the
// value directly and JavaScript will take care of the rest
person['fullName'] = function() {
  return this.firstName + ' ' + this.lastName;
};
 
// this time, let's make man's prototype person, so all
// men share the fullName function
var man = Object.create(person);
man['sex'] = "male";
 
var yehuda = Object.create(man);
yehuda['firstName'] = "Yehuda";
yehuda['lastName'] = "Katz";
 
yehuda.sex        // "male"
yehuda.fullName() // "Yehuda Katz"

Just like when looking up properties, if the property you are defining is an identifier, you can use dot syntax instead of bracket syntax. For instance, you could say man.sex = "male" in the example above.

Object Literals

Still, having to set a number of properties every time can get annoying. JavaScript provides a literal syntax for creating an object and assigning properties to it at one time.

var person = { firstName: "Paul", lastName: "Irish" }

This syntax is approximately sugar for:

var person = Object.create(Object.prototype);
person.firstName = "Paul";
person.lastName  = "Irish";

The most important thing about the expanded form is that object literals always set the newly created object’s prototype to an object located at Object.prototype. Internally, the object literal looks like this:

prototype-chain.png

The default Object.prototype dictionary comes with a number of the methods we have come to expect objects to contain, and through the magic of the prototype chain, all new objects created as object literal will contain these properties. Of course, objects can happily override them by defining the properties directly. Most commonly, developers will override the toString method:

var alex = { firstName: "Alex", lastName: "Russell" };
 
alex.toString() // "[object Object]"
 
var brendan = {
  firstName: "Brendan",
  lastName: "Eich",
  toString: function() { return "Brendan Eich"; }
};
 
brendan.toString() // "Brendan Eich"

This is especially useful because a number of internal coercion operations use a supplied toString method.

Unfortunately, this literal syntax only works if we are willing to make the new object’s prototype Object.prototype. This eliminates the benefits we saw earlier of sharing properties using the prototype. In many cases, the convenience of the simple object literal outweighs this loss. In other cases, you will want a simple way to create a new object with a particular prototype. I’ll explain it right afterward:

var fromPrototype = function(prototype, object) {
  var newObject = Object.create(prototype);
 
  for (var prop in object) {
    if (object.hasOwnProperty(prop)) {
      newObject[prop] = object[prop];      
    }
  }
 
  return newObject;
};
 
var person = {
  toString: function() {
    return this.firstName + ' ' + this.lastName;
  }
};
 
var man = fromPrototype(person, {
  sex: "male"
});
 
var jeremy = fromPrototype(man, {
  firstName: "Jeremy",
  lastName:  "Ashkenas"
});
 
jeremy.sex        // "male"
jeremy.toString() // "Jeremy Ashkenas"

Let’s deconstruct the fromPrototype method. The goal of this method is to create a new object with a set of properties, but with a particular prototype. First, we will use Object.create() to create a new empty object, and assign the prototype we specify. Next, we will enumerate all of the properties in the object that we supplied, and copy them over to the new object.

Remember that when you create an object literal, like the ones we are passing in to fromPrototype, it will always have Object.prototype as its prototype. By default, the properties that JavaScript includes on Object.prototype are not enumerable, so we don’t have to worry about valueOf showing up in our loop. However, since Object.prototype is an object like any other object, anyone can define a new property on it, which may (and probably would) be marked enumerable.

As a result, while we are looping through the properties on the object we passed in, we want to restrict our copying to properties that were defined on the object itself, and not found on the prototype. JavaScript includes a method called hasOwnProperty on Object.prototype to check whether a property was defined on the object itself. Since object literals will always have Object.prototype as their prototype, you can use it in this manner.

The object we created in the example above looks like this:

prototype-chain-2.png

Native Object Orientation

At this point, it should be obvious that prototypes can be used to inherit functionality, much like traditional object oriented languages. To facilitate using it in this manner, JavaScript provides a new operator.

In order to facilitate object oriented programming, JavaScript allows you to use a Function object as a combination of a prototype to use for the new object and a constructor function to invoke:

var Person = function(firstName, lastName) {
  this.firstName = firstName;
  this.lastName = lastName;
}
 
Person.prototype = {
  toString: function() { return this.firstName + ' ' + this.lastName; }
}

Here, we have a single Function object that is both a constructor function and an object to use as the prototype of new objects. Let’s implement a function that will create new instances from this Person object:

function newObject(func) {
  // get an Array of all the arguments except the first one
  var args = Array.prototype.slice.call(arguments, 1);
 
  // create a new object with its prototype assigned to func.prototype
  var object = Object.create(func.prototype);
 
  // invoke the constructor, passing the new object as 'this'
  // and the rest of the arguments as the arguments
  func.apply(object, args);
 
  // return the new object
  return object;
}
 
var brendan = newObject(Person, "Brendan", "Eich");
brendan.toString() // "Brendan Eich"

The new operator in JavaScript essentially does this work, providing a syntax familiar to those comfortable with traditional object oriented languages:

var mark = new Person("Mark", "Miller");
mark.toString() // "Mark Miller"

In essence, a JavaScript “class” is just a Function object that serves as a constructor plus an attached prototype object. I mentioned before that earlier versions of JavaScript did not have Object.create. Since it is so useful, people often created something like it using the new operator:

var createObject = function (o) {
  // we only want the prototype part of the `new`
  // behavior, so make an empty constructor
  function F() {}
 
  // set the function's `prototype` property to the
  // object that we want the new object's prototype
  // to be.
  F.prototype = o;
 
  // use the `new` operator. We will get a new
  // object whose prototype is o, and we will
  // invoke the empty function, which does nothing.
  return new F();
};

I really love that ECMAScript 5 and newer versions have begun to expose internal aspects of the implementation, like allowing you to directly define non-enumerable properties or define objects directly using the prototype chain.

Understanding JavaScript Function Invocation and “this”

Over the years, I’ve seen a lot of confusion about JavaScript function invocation. In particular, a lot of people have complained that the semantics of `this` in function invocations is confusing.

In my opinion, a lot of this confusion is cleared up by understanding the core function invocation primitive, and then looking at all other ways of invoking a function as sugar on top of that primitive. In fact, this is exactly how the ECMAScript spec thinks about it. In some areas, this post is a simplification of the spec, but the basic idea is the same.

The Core Primitive

First, let’s look at the core function invocation primitive, a Function’s call method[1]. The call method is relatively straight forward.

  1. Make an argument list (argList) out of parameters 1 through the end
  2. The first parameter is thisValue
  3. Invoke the function with this set to thisValue and the argList as its argument list

For example:

function hello(thing) {
  console.log(this + " says hello " + thing);
}
 
hello.call("Yehuda", "world") //=> Yehuda says hello world

As you can see, we invoked the hello method with this set to "Yehuda" and a single argument "world". This is the core primitive of JavaScript function invocation. You can think of all other function calls as desugaring to this primitive. (to “desugar” is to take a convenient syntax and describe it in terms of a more basic core primitive).

[1] In the ES5 spec, the call method is described in terms of another, more low level primitive, but it’s a very thin wrapper on top of that primitive, so I’m simplifying a bit here. See the end of this post for more information.

Simple Function Invocation

Obviously, invoking functions with call all the time would be pretty annoying. JavaScript allows us to invoke functions directly using the parens syntax (hello("world"). When we do that, the invocation desugars:

function hello(thing) {
  console.log("Hello " + thing);
}
 
// this:
hello("world")
 
// desugars to:
hello.call(window, "world");

This behavior has changed in ECMAScript 5 only when using strict mode[2]:

// this:
hello("world")
 
// desugars to:
hello.call(undefined, "world");

The short version is: a function invocation like fn(...args) is the same as fn.call(window [ES5-strict: undefined], ...args).

Note that this is also true about functions declared inline: (function() {})() is the same as (function() {}).call(window [ES5-strict: undefined).

[2] Actually, I lied a bit. The ECMAScript 5 spec says that undefined is (almost) always passed, but that the function being called should change its thisValue to the global object when not in strict mode. This allows strict mode callers to avoid breaking existing non-strict-mode libraries.

Member Functions

The next very common way to invoke a method is as a member of an object (person.hello()). In this case, the invocation desugars:

var person = {
  name: "Brendan Eich",
  hello: function(thing) {
    console.log(this + " says hello " + thing);
  }
}
 
// this:
person.hello("world")
 
// desugars to this:
person.hello.call(person, "world");

Note that it doesn't matter how the hello method becomes attached to the object in this form. Remember that we previously defined hello as a standalone function. Let's see what happens if we attach is to the object dynamically:

function hello(thing) {
  console.log(this + " says hello " + thing);
}
 
person = { name: "Brendan Eich" }
person.hello = hello;
 
person.hello("world") // still desugars to person.hello.call(person, "world")
 
hello("world") // "[object DOMWindow]world"

Notice that the function doesn't have a persistent notion of its 'this'. It is always set at call time based upon the way it was invoked by its caller.

Using Function.prototype.bind

Because it can sometimes be convenient to have a reference to a function with a persistent this value, people have historically used a simple closure trick to convert a function into one with an unchanging this:

var person = {
  name: "Brendan Eich",
  hello: function(thing) {
    console.log(this.name + " says hello " + thing);
  }
}
 
var boundHello = function(thing) { return person.hello.call(person, thing); }
 
boundHello("world");

Even though our boundHello call still desugars to boundHello.call(window, "world"), we turn right around and use our primitive call method to change the this value back to what we want it to be.

We can make this trick general-purpose with a few tweaks:

var bind = function(func, thisValue) {
  return function() {
    return func.apply(thisValue, arguments);
  }
}
 
var boundHello = bind(person.hello, person);
boundHello("world") // "Brendan Eich says hello world"

In order to understand this, you just need two more pieces of information. First, arguments is an Array-like object that represents all of the arguments passed into a function. Second, the apply method works exactly like the call primitive, except that it takes an Array-like object instead of listing the arguments out one at a time.

Our bind method simply returns a new function. When it is invoked, our new function simply invokes the original function that was passed in, setting the original value as this. It also passes through the arguments.

Because this was a somewhat common idiom, ES5 introduced a new method bind on all Function objects that implements this behavior:

var boundHello = person.hello.bind(person);
boundHello("world") // "Brendan Eich says hello world"

This is most useful when you need a raw function to pass as a callback:

var person = {
  name: "Alex Russell",
  hello: function() { console.log(this.name + " says hello world"); }
}
 
$("#some-div").click(person.hello.bind(person));
 
// when the div is clicked, "Alex Russell says hello world" is printed

This is, of course, somewhat clunky, and TC39 (the committee that works on the next version(s) of ECMAScript) continues to work on a more elegant, still-backwards-compatible solution.

On jQuery

Because jQuery makes such heavy use of anonymous callback functions, it uses the call method internally to set the this value of those callbacks to a more useful value. For instance, instead of receiving window as this in all event handlers (as you would without special intervention), jQuery invokes call on the callback with the element that set up the event handler as its first parameter.

This is extremely useful, because the default value of this in anonymous callbacks is not particularly useful, but it can give beginners to JavaScript the impression that this is, in general a strange, often mutated concept that is hard to reason about.

If you understand the basic rules for converting a sugary function call into a desugared func.call(thisValue, ...args), you should be able to navigate the not so treacherous waters of the JavaScript this value.

this-table.png

PS: I Cheated

In several places, I simplified the reality a bit from the exact wording of the specification. Probably the most important cheat is the way I called func.call a "primitive". In reality, the spec has a primitive (internally referred to as [[Call]]) that both func.call and [obj.]func() use.

However, take a look at the definition of func.call:

  1. If IsCallable(func) is false, then throw a TypeError exception.
  2. Let argList be an empty List.
  3. If this method was called with more than one argument then in left to right order starting with arg1 append each argument as the last element of argList
  4. Return the result of calling the [[Call]] internal method of func, providing thisArg as the this value and argList as the list of arguments.

As you can see, this definition is essentially a very simple JavaScript language binding to the primitive [[Call]] operation.

If you look at the definition of invoking a function, the first seven steps set up thisValue and argList, and the last step is: "Return the result of calling the [[Call]] internal method on func, providing thisValue as the this value and providing the list argList as the argument values."

It's essentially identical wording, once the argList and thisValue have been determined.

I cheated a bit in calling call a primitive, but the meaning is essentially the same as had I pulled out the spec at the beginning of this article and quoted chapter and verse.

There are also some additional cases (most notably involving with) that I didn't cover here.

What’s Up With All These Changes in Rails?

Yesterday, there was a blog post entitled “What the Hell is Happening to Rails” that stayed at the number one spot on Hacker News for quite a while. The post and many (but not most) the comments on the post reflect deep-seated concern about the recent direction of Rails. Others have addressed the core question about change in the framework, but I’d like to address questions about specific changes that came up in the post and comments.

The intent of this post is not to nitpick the specific arguments that were made, or to address the larger question of how much change is appropriate in the framework, but rather to provide some background on some of the changes that have been made since Rails 2.3 and to explain why we made them.

Block Helpers

I too get a feeling of “change for the sake of change” from Rails at times. That’s obviously not something they’re doing, as all the changes have some motivation, but at times it feels a bit like churn.
At one point in time, you did forms like <%= form …. and then they switched to <% form …. do and now they’ve switched back to <%= form … do again.
Also, the upgrade to Rails 3 is not an easy one. Yeah, you get some nice stuff, but because it’s so painful, it’s not happening for a lot of people, which is causing more problems.

Prior to Rails 3.0, Rails never used <= form_for because it was technically very difficult to make it work. I wrote a post about it in Summer 2009 that walked through the technical problems. The short version is that every ERB template is compiled into a Ruby method, and reliably converting &lt%= with blocks proved to be extremely complicated.

However, knowing when to use <% and when to use <= caused major issues for new developers, and it was a major 3.0 priority to make this work. In addition, because of the backwards compatibility issue, we went to extreme Ruby-exploiting lengths to enable deprecation warnings about the change, so that we could fix the issue for new apps, but also not break old apps.

The night José and I figured out how to do this (the Engine Yard party at Mountain West RubyConf 2009), we were pretty close to coming to the conclusion that it couldn’t be done short of using a full language parser in ERB, which should give you some sense of how vexing a problem it was for us.

Performance

The general disregard for improving performance, inherited from Ruby, is also something endemic from a while back.

Yes, there have been some performance regressions in Rails 3.0. However, the idea that the Rails core team doesn’t care about performance, and neither does the Ruby team doesn’t pass the smell test.

Aaron Patterson, the newest core team member, worked full time for almost a year to get the new ActiveRecord backend into decent performance shape. Totally new code often comes with some performance regressions, and the changes to ActiveRecord were important and a long-time coming. Many of us didn’t see the magnitude of the initial problem until it was too late for 3.0, but we take the problem extremely seriously.

Ruby core (“MRI”) itself has sunk an enormous amount of time into performance improvements in Ruby 1.9, going so far as to completely rewrite the core VM from scratch. This resulted in significant performance improvements, and the Ruby team continues to work on improvements to performance and to core systems like the garbage collector.

The Ruby C API poses some long-term problems for the upper-bound of Ruby performance improvements, but the JRuby and Rubinius projects are showing how you can use Ruby C extensions inside a state-of-the-art virtual machine. Indeed, the JRuby and Rubinius projects show that the Ruby community both cares about, and is willing to invest significant energy into improving the performance of Ruby.

Heroku and the Asset Pipeline

The assets pipeline feels horrible, it’s really slow. I upgraded to Rails 3.1rc, realized it fights with Heroku unless upgrading to the Cedar stack

The problem with Heroku is that the default Gemfile that comes with Rails 3.1 currently requires a JavaScript runtime to boot, even in production. This is clearly wrong and will be fixed posthaste. Requiring Rails apps to have node or a compiled v8 in production is an unacceptable burden.

On the flip side, the execjs gem, which manages compiling CoffeeScript (and more importantly minifying your JavaScript), is actually a pretty smart piece of work. It turns out that both Mac OSX and Windows ship with usable JavaScript binaries, so in development, most Rails users will already have a JavaScript library ready to use.

It’s worth noting that a JavaScript engine is needed to run “uglify.js”, the most popular and most effective JavaScript minifier. It is best practice to minify your JavaScript before deployment, so you can feel free to format and comment your code as you like without effecting the payload. You can learn more about minification in this excellent post by Steve Souders from a few years back. Rails adding minification by default is an unambiguous improvement in the workflow of Rails applications, because it makes it easy (almost invisible) to do something that everyone should be doing, but which has previously been something of a pain.

Again, making node a dependency in production is clearly wrong, and will be removed before the final release.

Change for Change’s Sake

The problem with Rails is not the pace of change so much as the wild changes of direction it takes, sometimes introducing serious performance degradations into official releases. Sometimes it’s hard to see a guiding core philosophy other than the fact they want to be on the shiny edge

When Rails shipped, it came with a number of defaults:

  • ActiveRecord
  • Test::Unit
  • ERB
  • Prototype

Since 2004, alternatives to all of those options have been created, and in many cases (Rspec, jQuery, Haml) have become somewhat popular. Rails 3.0 made no changes to the defaults, despite much clamoring for a change from Prototype to jQuery. Rails 3.1 changed to jQuery only when it became overwhelmingly clear that jQuery was the de facto standard on the web.

As someone who has sat in on many discussions about changing defaults, I can tell you that defaults in Rails are not changed lightly, and certainly not “to be on the shiny edge.” In fact, I think that Rails is more conservative than most would expect in changing defaults.

Right, but the problem is that there doesn’t seem to be a ‘right’ way, That’s the problem.

We were all prototype a few years ago, now it jquery … we (well I) hadn’t heard of coffeescript till a few months ago and now its a default option in rails, The way we were constructing ActiveRecord finders had been set all through Rails 2, now we’ve changed it, the way we dealt with gems was set all through rails 2 now its changed completely in Rails 3.

I like change, I like staying on the cutting edge of web technologies, but I don’t want to learn something, only to discard it and re-do it completely to bring it up to date with a new way of doing things all the time.

First, jQuery has become the de facto standard on the web. As I said earlier, Rails resisted making this change in 3.0, despite a lot of popular demand, and the change to jQuery is actually an example of the stability of Rails’ default choices over time, rather than the opposite.

Changes to gem handling evolved as the Rails community evolved to use gems more. In Rails 1, all extensions were implemented as plugins that got pulled out of svn and dumped into your project. This didn’t allow for versioning or dependencies, so Rails 2 introduced first-class support for Rails plugins as gems.

During the Rails 2 series, Rails added a dependency on Rack, which caused serious problems when Rails was used with other gems that rely on Rack, due to the way that raw Rubygems handles dependency activation. Because Rails 3 uses gem dependencies more extensively, we spent a year building bundler, which adds per-application dependency resolution to Rubygems. This was simply the natural evolution of the way that Rails has used dependencies over time.

The addition of CoffeeScript is interesting, because it’s a pretty young technology, but it’s also not really different from shipping a new template handler. When you create a new Rails app, the JavaScript file is a regular JS file, and the asset compilation and dependency support does not require CoffeeScript. Shipping CoffeeScript is basically like shipping Builder: should you want it, it’s there for you. Since we want to support minification out of the box anyway, CoffeeScript doesn’t add any new requirements. And since it’s just there in the default Gemfile, as opposed to included in Rails proper like Builder, turning it off (if you really want to) is as simple as removing a line in your Gemfile. Nothing scary here.

ActiveRecord Changes

The way we were constructing ActiveRecord finders had been set all through Rails 2, now we’ve changed it

The Rails core team does seem to treat the project as if it’s a personal playground

One of the biggest problems with ActiveRecord was the way it internally used Strings to represent queries. This meant that changes to queries often required gsubing String to make simple changes. Internally, it was a mess, and it expressed itself in the public API in how conditions were generated, and more importantly how named scopes were created.

One goal of the improvements in Rails 3 was to get rid of the ad-hoc query generation in Rails 2 and replace it with something better. ActiveRelation, this library, was literally multiple years in the making, and a large amount of the energy in the Rails 3.0 process was spent on integrating ActiveRelation.

From a user-facing perspective, we wanted to unify all of the different ways that queries were made. This means that scopes, class methods, and one-time-use queries all use the same API. As in the <= case, a significant amount of effort was spent on backwards compatibility with Rails 2.3. In fact, we decided to hold onto support for the old API as least as long as Rails 3.2, in order to soften the community transition to the new API.

In general, we were quite careful about backwards compatibility in Rails 3.0, and while a project as complex as Rails was not going to be perfect in this regard, characterizing the way we handled this as “irresponsible” or “playground” disregards the tremendous amount of work and gymnastics that the core team and community contributors put into supporting Rails 2.3 APIs across the entire codebase when changes were made.

Using SproutCore 2.0 with jQuery UI

One of the goals of SproutCore 2.0 is to make it trivial to integrate the tools you’re already using together with SproutCore.

One way that we do that is to make it possible to drop a SproutCore app into a small part of your existing page. Greg Moeck did a great job demonstrating this functionality on the SproutCore Blog a couple of weeks ago, and I’d definitely recommend checking it out if you’re interested in learning more about embedding SproutCore 2.0 in an existing app.

Another complementary approach is to make it easy to integrate SproutCore with existing JavaScript libraries and tools. Because SproutCore 2.0 uses templates as the primary way to build your application, it integrates well with the tools you’re already using. In this post, I’ll show you the hooks you need to integrate SproutCore with anything you want, and then show you how to do a general-purpose integration with jQuery UI.

Quick Refresh: Handlebars Templates

First of all, you can take any piece of HTML and attach it to a SproutCore view. You would typically do this in order to attach event handlers to the view or to define a function that computes how a particular value should be displayed. For instance, let’s say we have some HTML that shows a business card for a person:

<div class="card">
  <p class="name">{{salutation}} {{fullName}}</p>
  <p class="address">{{number}} {{street}}</p>
  <p class="region">{{city}}, {{state}} {{zip}}</p>
</div>

We can wrap this chunk of HTML in a SproutCore view, which will define the sources of each of the properties used in {{}}}.

<script type="text/html">
{{#view Yehuda.BusinessCard contentBinding="Yehuda.businessContent"}}
  <div class="card">
    <p class="name">{{content.salutation}} {{fullName}}</p>
    <p class="address">{{content.streetNumber}} {{content.street}}</p>
    <p class="region">{{content.city}}, {{content.state}} {{content.zip}}</p>
  </div>
{{/view}}
</script>

Wrapping the HTML snippet in a <script> tag tells SproutCore to render it using its template engine. You’d normally get the {{}} properties from a controller or other data source, but let’s define a view with the values baked in for now:

Yehuda.BusinessCard = SC.View.extend({
  // fill in the content in the template with contentBinding
  content: null,
 
  fullName: function() {
    return this.get('firstName') + ' ' + this.get('lastName');
  }.property('content.firstName', 'content.lastName')
});
 
Yehuda.businessContent = SC.Object.create({
  firstName: "Yehuda",
  lastName: "Katz",
  salutation: "Mr.",
  streetNumber: 300,
  street: "Brannan",
  city: "San Francisco",
  state: "CA",
  zip: "94107"
});

If we reload the page, we will see a static business card with my information in it. If we update my content with Yehuda.businessContent.set(key, value), SproutCore will automatically update the HTML for you. If you update any property, including firstName or lastName, which are used only as dependencies of fullName, SproutCore will automatically handle updating the HTML for you.

didCreateElement

You can also define some code to run every time an element is rendered. You can use this hook to wire up any JavaScript library to the element that SproutCore has created. For instance, let’s say we want to zebra stripe the three lines in our business card once it has been create (note: you probably would not use JS to zebra stripe in real life).

Let’s extend our view:

Yehuda.BusinessCard = SC.View.extend({
  // fill in the content in the template with contentBinding
  content: null,
 
  fullName: function() {
    return this.get('firstName') + ' ' + this.get('lastName');
  }.property('content.firstName', 'content.lastName'),
 
  didCreateElement: function() {
    this._super();
    this.$("p:even").addClass("even");
  }
});

First of all, you will want to call super, to ensure that any hooks in parent classes get called. Second, all SproutCore views have a $ function, which is the normal jQuery function, scoped to the current element. You can also call this.$() to get back a jQuery object wrapping the element itself.

You can use this hook to do anything you want, allowing you to hook in any library you want. You would use this hook a lot like you’d use a jQuery(document).ready hook, but scoped to the view being created.

SproutCore also provides a willDestroyElement hook, which you can use to tear things down that you set up when the element was created. This is relatively rare, and is mostly used when interfacing with another toolkit that requires a teardown, or to tear down custom SproutCore observers.

jQuery UI

We can use these hooks to build some basic integration with jQuery UI. I’ve written a demo that shows how to connect SproutCore bindings to jQuery options at http://sc20-jqui.strobeapp.com/. You can also check out the annotated source code.

The most important thing to look at is the JQ.Widget mixin, which can be mixed into view classes to attach jQuery UI options and events to SproutCore’s binding and event system.

You can get started with the SproutCore 2.0 developer preview by downloading the starter kit.

Until next time!

Gem Versioning and Bundler: Doing it Right

Recently, an upgrade to Rake (from version 0.8.7 to version 0.9.0) has re-raised the issue of dependencies and versioning in the Ruby community. I wanted to take the opportunity to reiterate some of the things I talked about back when working on Bundler 1.0. First, I’ll lay out some basic rules for the road, and then go into some detail about the rationale.

Basic Versioning Rules for Apps

  1. After bundling, always check your Gemfile.lock into version control. If you do this, you do not need to specify exact versions of gems in your Gemfile. Bundler will take care of ensuring that all systems use the same versions.
  2. After updating your Gemfile, always run bundle install first. This will conservatively update your Gemfile.lock. This means that except for the gem that you changed in your Gemfile, no other gem will change.
  3. If a conservative update is impossible, bundler will prompt you to run bundle update [somegem]. This will update the gem and any necessary dependencies. It will not update unrelated gems.
  4. If you want to fully re-resolve all of your dependencies, run bundle update. This will re-resolve all dependencies from scratch.
  5. When running an executable, ALWAYS use bundle exec [command]. Quoting from the bundler documentation: In some cases, running executables without bundle exec may work, if the executable happens to be installed in your system and does not pull in any gems that conflict with your bundle. However, this is unreliable and is the source of considerable pain. Even if it looks like it works, it may not work in the future or on another machine. See below, “Executables” for more information and advanced usage.
  6. Remember that you can always go back to your old Gemfile.lock by using git checkout Gemfile.lock or the equivalent command in your version control.

Executables

When you install a gem to the system, Rubygems creates wrappers for every executable that the gem makes available. When you run an executable from the command line without bundle exec, this wrapper invokes Rubygems, which then uses the normal Rubygems activation mechanism to invoke the gem’s executable. This has changed in the past several months, but Rubygems will invoke the latest version of the gem installed in your system, even if your Gemfile.lock specifies a different version. In addition, it will activate the latest (compatible) installed version of dependencies of that gem, even if a different version is specified in your Gemfile.lock.

This means that invoking executables as normal system executables bypasses bundler’s locked dependencies. In many cases, this will not pose a problem, because developers of your app tend to have the right version of the system-installed executable. For a long time, the Rake gem was a good example of this phenomenon, as most Gemfile.locks declared Rake 0.8.7, and virtually all Ruby developers had Rake 0.8.7 installed in their system.

As a result, users fell into the unfortunate belief that running system executables was compatible with bundler’s locked dependencies. To work around some of the remaining cases, people often advocate the use of rvm gemsets. Combined with manually setting up application-specific gemsets, this can make sure that the “system executables” as provided via the gemset remain compatible with the Gemfile.lock.

Unfortunately, this kludge (and others) sufficiently reduced the pain that most people ignored the advice of the bundler documentation to always use bundle exec when running executables tied to gems in the application’s Gemfile.lock.

It’s worth noting that typing in rake foo (or anyexecutable foo) in the presence of a Gemfile.lock, and expecting it to execute in the bundler sandbox doesn’t make any sense, since you’re not invoking Bundler. Bundler’s sandbox relies on its ability to be present at the very beginning of the Ruby process, and to therefore have the ability to ensure that the versions of all loaded libraries will reflect the ones listed in the Gemfile.lock. By running a system executable, you are executing Ruby code before Bundler can modify the load path and replace the normal Rubygems loading mechanism, allowing arbitrary unmanaged gems to get loaded into memory. Once that happens, all bets are off.

bundle install –binstubs

In order to alleviate some of the noise of bundle exec, Bundler 1.0 ships with a --binstubs flag, which will create a bin directory containing each of the executables that the application’s gems expose. Running bin/cucumber, for instance, is the equivalent of running bundle exec cucumber.

The generated bin directory should contain portable versions of the executables, so it should be safe to add them to version control.

The rails Command

The only exception to the above rules is the rails executable. As of Rails 3.0, the executable simply looks for a script/rails file, and execs that file. The generated script/rails loads the local boot environment, which invokes bundler immediately:

# This command will automatically be run when you run "rails" with Rails 3 gems installed from the root of your application.
 
APP_PATH = File.expand_path('../../config/application',  __FILE__)
require File.expand_path('../../config/boot',  __FILE__)
require 'rails/commands'

The generated boot.rb is very simple:

require 'rubygems'
 
# Set up gems listed in the Gemfile.
ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../../Gemfile', __FILE__)
 
require 'bundler/setup' if File.exists?(ENV['BUNDLE_GEMFILE'])

In short, the rails executable does a bunch of work to guarantee that your application’s bootstrapping logic runs before any dependencies are loaded, and it uses Kernel#exec to purge the current process of any already-loaded gems. This is not something that general-purpose gems can or should do, and it’s an open question whether making the rails executable work without bundle exec sufficiently muddies the waters with regard to other gems as to render the benefit not worth it.

Getting Comfortable With Rubinius’ Pure-Ruby Internals

You probably know that Rubinius is a Ruby whose implementation is mostly written in Ruby. While that sounds nice in theory, you may not know what that means in practice. Over the past several years, I’ve contributed on and off to Rubinius, and feel that as Rubinius has matured since the 1.0 release, a lot of its promise has gelled.

One of the great things about Rubinius is that it exposes, as first-class concepts, a lot of things that are merely implicit in other Ruby implementations. For instance, Ruby methods have a variable scope which is implicitly accessible by blocks created in the scope. Rubinius exposes that scope as Rubinus::VariableScope. It also exposes Ruby bindings as richer objects that you can create yourself and use. In this post, I will talk about Rubinius::VariableScope, how it works, and how it fits into the Rubinius codebase.

Prologue

For those of you who don’t know much about Rubinius, it’s a Ruby implementation whose core classes are mostly written in Ruby. Some functionality, such as the core of the object model, as well as low-level methods, are written in C++ and exposed into Ruby via its primitive system. However, the functionality is typically fairly low-level, and the vast majority of the functionality is written on top of those primitives. For example, a Ruby String in Rubinius is a mostly pure-Ruby object with a backing ByteArray, and ByteArray’s implementation is mostly written as a primitive.

For instance, String’s allocate method looks like this:

def self.allocate
  str = super()
  str.__data__ = Rubinius::ByteArray.new(1)
  str.num_bytes = 0
  str.characters = 0
  str
end

If you just want to get to the meat of the matter, feel free to skip this section and go directly to the Internals section below.

In general, Rubinius’ code is broken up into three stages. The first stage, called alpha, creates just enough raw Ruby constructs to get to the next stage, the bootstrap stage. It defines methods like Class#new, Kernel#__send__, Kernel#raise, Kernel#clone, basic methods on Module, Object, String, and Symbol, and some classes on the Rubinius module, used for internal use. It’s a pretty short file, weighing it at under 1,000 lines, and I’d encourage you to read it through. It is located in kernel/alpha.rb.

Next, the bootstrap stage creates the basic methods needed on many of Ruby’s core classes. In some cases, it defines a simpler version of a common structure, like Rubinius::CompactLookupTable and Rubinius::LookupTable instead of Hash. It also contains Rubinius constructs like the Rubinius::CompiledMethod, Rubinius::MethodTable, Rubinius::StaticScope and Rubinius::VariableScope. Many of these methods are defined as primitives. Primitive methods are hooked up in Ruby; check out the bootstrap definition of Rubinius::VariableScope:

module Rubinius
  class VariableScope
    def self.of_sender
      Ruby.primitive :variable_scope_of_sender
      raise PrimitiveFailure, "Unable to get VariableScope of sender"
    end
 
    def self.current
      Ruby.primitive :variable_scope_current
      raise PrimitiveFailure, "Unable to get current VariableScope"
    end
 
    def locals
      Ruby.primitive :variable_scope_locals
      raise PrimitiveFailure, "Unable to get VariableScope locals"
    end
 
    # To handle Module#private, protected
    attr_accessor :method_visibility
  end
end

The bootstrap definitions are in kernel/bootstrap. The file kernel/bootstrap/load_order.txt defines the order that bootstrap methods are loaded.

Next, Rubinius fleshes out the core classes, mostly using Ruby methods. There is some primitive use in this stage, but it’s fairly few and far between. For instance, the Hash class is written entirely in Ruby, using a Rubinius::Tuple for its entries. This code can be found in kernel/common.

Some additional code lives in kernel/platform, which contains platform-specific functionality, like File and POSIX functionality, and kernel/delta, which runs after kernel/common. For the full description of the bootstrapping process, check out Rubinius: Bootstrapping.

Internals

One of the really nice things about Rubinius is that many concepts that are internal in traditional Ruby are exposed directly into the Ruby runtime. In many cases, this is so that more of Rubinius can be implemented in pure Ruby. In other words, if a core method in MRI needs some behavior, it is often implemented directly in C and unavailable to custom Ruby code.

One good example is the behavior of $1 in a Ruby method. If you call the =~ method, MRI’s internal C code will walk back up to the caller and set an internal match object on it. If you write your own method that uses the =~ method, there is no way for you to set the match object on the caller. However, since Rubinius itself implements these methods in Ruby, it needs a way to perform this operation directly from Ruby code. Check out the Rubinius implementation of =~:

class Regex
  def =~(str)
    # unless str.nil? because it's nil and only nil, not false.
    str = StringValue(str) unless str.nil?
 
    match = match_from(str, 0)
    if match
      Regexp.last_match = match
      return match.begin(0)
    else
      Regexp.last_match = nil
      return nil
    end
  end
end

There are a few interesting things going on here. First, Rubinius calls StringValue(str), which is a Ruby method that implements Ruby’s String coercion protocol. If you follow the method, you will find that it calls Type.coerce_to(obj, String, :to_str), which then, in pure-Ruby, coerces the object into a String. You can use this method anywhere in your code if you want to make use of the String coercion protocol with the standard error messages.

Next, Rubinius actually invokes the match call, which ends up terminating in a primitive called search_region, which binds directly into the Oniguruma regular expression engine. The interesting part is next: Regexp.last_match = match. This code sets the $1 property on the caller, from Ruby. This means that if you write a custom method involving regular expressions and want it to obey the normal $1 protocol, you can, because if Rubinius needs functionality like this, it is by definition exposed to Ruby.

VariableScope

One result of this is that you can get access to internal structures like the current “variable scope”. A variable scope is an object that wraps the notion of which locals are available (both real locals and locals defined using eval), as well as the current contents of each of those variables. If the current context is a block, a VariableScope also has a pointer to the parent VariableScope. You can get access to the current VariableScope by using VariableScope.current.

def foo
  x = 1
  p [:root, Rubinius::VariableScope.current.locals]
  ["a", "b", "c"].each do |item|
    p [:parent, Rubinius::VariableScope.current.parent.locals]
    p [:item, Rubinius::VariableScope.current.locals]
  end
end
 
foo

The output will be:

[:root, #<Rubinius::Tuple: 1>]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "a">]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "b">]
[:parent, #<Rubinius::Tuple: 1>]
[:item, #<Rubinius::Tuple: "c">]

A Rubinius::Tuple is a Ruby object that is similar to an Array, but is fixed-size, simpler, and usually more performant. The VariableScope has a number of additional useful methods, which you can learn about by reading kernel/bootstrap/variable_scope.rb and kernel/common/variable_scope.rb. One example: locals_layout gives you the names of the local variables for each slot in the tuple. You can also find out the current visibility of methods that would be declared in the scope (for instance, if you call private the VariableScope after that call will have method_visibility of :private).

Perhaps the most exciting thing about VariableScope is that you can look up the VariableScope of your calling method. Check this out:

def root
  x = 1
  ["a", "b", "c"].each do |item|
    call_me
  end
end
 
def call_me
  sender = Rubinius::VariableScope.of_sender
  p [:parent, sender.parent.locals]
  p [:item, sender.locals]
end
 
root

In this case, you can get the VariableScope of the caller, and get any live information from it. In this case, you can tell that the method was called from inside a block, what the locals inside the block contain, and what block contexts exist up to the method root. This is extremely powerful, and gives you a level of introspection into a running Ruby program heretofore not possible (or very difficult). Better yet, the existence of this feature doesn’t slow down the rest of the program, because Rubinius uses it internally for common Ruby functionality like the implementation of private and protected.

Implementing caller_locals

Now that we understand the VariableScope object, what if we want to print out a list of all of the local variables in the caller of a method.

def main
  a = "Hello"
 
  [1,2,3].each do |b|
    say_hello do
      puts "#{a}: #{b}"
    end
  end
end
 
def say_hello
  # this isn't working as we expect, and we want to look at
  # the local variables of the calling method to see what's
  # going on
  yield
ensure
  puts "Goodbye"
end

Let’s implement a method called caller_locals which will give us all of the local variables in the calling method. We will not be able to use VariableScope.of_sender, since that only goes one level back, and since we will implement caller_locals in Ruby, we will need to go two levels back. Thankfully, Rubinius::VM.backtrace has the information we need.

Rubinius::VM.backtrace takes two parameters. The first is the number of levels back to start from. In this case we want to start two levels back (the caller will be the say_hello method, and we want its caller). The second parameter indicates whether the Location objects should contain the VariableScope, which is what we want.

NOTE: Yes, you read that right. In Rubinius, you can obtain a properly object-oriented backtrace object, where each entry is a Location object. Check out kernel/common/location.rb to see the definition of the class.

In this case, if we pass true, to Rubinius::VM.backtrace, each Location will have a method called variables, which is a Rubinus::VariableScope object.

def caller_locals
  variables = Rubinius::VM.backtrace(2, true).first.variables
end

The next thing you need to know is that each block in a method has its own VariableScope and a parent pointing at the VariableScope for the parent block.

Each VariableScope also has a pointer to its method, an object that, among other things, keeps a list of the local variable names. These lists are both ordered the same.

To collect up the locals from the caller, we will walk this chain of VariableScope objects until we get nil.

def caller_locals
  variables = Rubinius::VM.backtrace(2, true).first.variables
 
  locals = []
 
  while variables
    names  = variables.method.local_names
    values = variables.locals
 
    locals << Hash[*names.zip(values).flatten]
    variables = variables.parent
  end
 
  locals
end

This will return an Array of local variable Hashes, starting from the caller’s direct scope and working up to the method body itself. Let’s change the original method to use caller_locals:

def say_hello
  # this isn't working as we expect, and we want to look at
  # the local variables of the calling method to see what's
  # going on. Let's use caller_locals.
  p caller_locals
  yield
ensure
  puts "Goodbye"
end

If we run the program now, the output will be:

[{:b=>1}, {:a=>"Hello"}]
Goodbye
[{:b=>2}, {:a=>"Hello"}]
Goodbye
[{:b=>3}, {:a=>"Hello"}]
Goodbye

This is just the tip of the iceberg of what is possible because of Rubinius’ architecture. To be clear, these features are not accidents; because so much of Rubinius (including things like backtraces), it is forced to expose details of the internals to normal Ruby program. And because the Rubinius kernel is real Ruby (and not some kind of special Ruby subset), anything that works in the kernel will work in your Ruby code as well.

Clarifying the Roles of the .gemspec and Gemfile

TL;DR

Although apps and gems look like they share the concept of “dependency”, there are some important differences between them. Gems depend on a name and version range, and intentionally don’t care about where exactly the dependencies come from. Apps have more controlled deployments, and need a guarantee that the exact same code is used on all machines (dev, ci and production).

When developing a gem, use the gemspec method in your Gemfile to avoid duplication. In general, a gem’s Gemfile should contain the Rubygems source and a single gemspec line. Do not check your Gemfile.lock into version control, since it enforces precision that does not exist in the gem command, which is used to install gems in practice. Even if the precision could be enforced, you wouldn’t want it, since it would prevent people from using your library with versions of its dependencies that are different from the ones you used to develop the gem.

When developing an app, check in your Gemfile.lock, since you will use the bundler tool across all machines, and the precision enforced by bundler is extremely desirable for applications.


Since my last post on this topic, a lot of people have asked follow-up questions about the specific roles of the Gemfile, used by bundler, and the .gemspec, used by Rubygems. In short, people seem to feel that there is duplication between the two files during gem development, and end up using yet another tool to reduce the duplication.

Ruby Libraries

Ruby libraries are typically packaged up using the gem format, and distributed using rubygems.org. A gem contains a number of useful pieces of information (this list is not exhaustive):

  • A bunch of metadata, like name, version, description, summary, email address, etc.
  • A list of all the files contained in the package.
  • A list of executables that come with the gem and their location in the gem (usually bin)
  • A list of directories that should be put on the Ruby load path (usually lib)
  • A list of other Ruby libraries that this library needs in order to function (dependencies)

Only the last item, the list of dependencies, overlaps with the purpose of the Gemfile. When a gem lists a dependency, it lists a particular name and a range of version numbers. Importantly, it does not care where the dependency comes from. This makes it easy to internally mirror a Rubygems repository or use a hacked version of a dependency that you install to your system. In short, a Rubygems dependency is a symbolic name. It is not a link to a particular location on the Internet where the code can be found.

Among other things, this characteristic is responsible for the resilience of the overall system.

Also, gem authors choose a range of acceptable versions of their dependencies, and have to operate under the assumption that the versions of the dependencies that the author tested against may change before the gem is used in deployment. This is especially true about dependencies of dependencies.

Ruby Applications

A Ruby application (for instance, a Rails application), has a complex set of requirements that are generally tested against a precise set of third-party code. While the application author may not have cared much (up front), which version of nokogiri the application used, he does care a lot that the version used in development will remain the version used in production. Since the application author controls the deployment environment (unlike the gem author), he does not need to worry about the way that a third-party will use the code. He can insist on precision.

As a result, the application author will want to specify the precise gem servers that the deployment environment should use. This is in contrast to the gem author, who should prefer long-term resilience and mirroring to the brittleness that comes with a short-term guarantee.

The application author often also wants a way to tweak shared community gems for the particular project at hand (or just use the “edge” version of a gem which hasn’t yet been released). Bundler handles this problem by allowing application authors to override a particular gem and version with a git repository or a directory. Because the gem author has specified the gem’s dependencies as a name and version range, the application developer can choose to satisfy that dependency from an alternate source (in this case, git).

Now, the application developer must be able to demand that the gem comes from that particular source. As you can see, there are critical differences between the desires of the gem developer (which are focused around longevity and flexibility) and the app developer (which are focused around absolute guarantees that all of the code, including third-party code, remains precisely the same between environments).

Rubygems and Bundler

The Rubygems libraries, CLI and gemspec API are built around the needs of the gem author and the Rubygems ecosystem. In particular, the gemspec is a standard format for describing all of the information that gets packed with gems then deployed to rubygems.org. For the reasons described above, gems do not store any transient information about where to find dependencies.

On the other hand, bundler is built around the needs of the app developer. The Gemfile does not contain metadata, a list of files in the package, executables exposed by the package, or directories that should be put on the load path. Those things are outside the scope of bundler, which focuses on making it easy to guarantee that all of the same code (including third-party code) is used across machines. It does, however, contain a list of dependencies, including information about where to find them.

Git “Gems”

For the reasons described above, it can sometimes be useful for application developers to declare a dependency on a gem which has not yet been released. When bundler fetches that gem, it uses the .gemspec file found in the git repository’s root to extract its metadata as well as discover additional dependencies declared by the in-development gem. This means that git gems can seamlessly be used by bundler, because their .gemspec allows them to look exactly like normal gems (including, importantly, having more dependencies of its own). The only difference is that the application developer has told bundler exactly where to look for the gem in question.

Gem Development

Active development on gems falls in between these two views of the world, for two reasons:

  • Active development on a gem often involves dependencies that have not yet been released to rubygems.org, so it becomes important to specify the precise location to find those dependencies, during development
  • During gem development, you cannot rely on rubygems to set up an environment with everything on the load path. Instead, you need a hybrid environment, with one gem coming from the file system (the current gem under development), some gems possibly coming from git (where you depend on a gem that hasn’t yet been released), and yet additional gems coming from traditional sources. This is the situation that Rails development is in.

Because gems under active development will eventually become regular gems, they will need to declare all of the regular metadata needed by gem build. This information should be declared in a .gemspec, since the Gemfile is an API for declaring dependencies only.

In order to make it easy to use the bundler tool for gem development without duplicating the list of dependencies in both the .gemspec and the Gemfile, bundler has a single directive that you can use in your Gemfile (documented on the bundler site):

source "http://www.rubygems.org"
 
gemspec

The gemspec line tells bundler that it can fine a .gemspec file alongside the Gemfile. When you run bundle install, bundler will find the .gemspec and treat the local directory as a local, unpacked gem. It will find and resolve the dependencies listed in the .gemspec. Bundler’s runtime will add the load paths listed in the .gemspec to the load path, as well as the load paths of any of the gems listed as dependencies (and so on). So you get to use bundler while developing a gem with no duplication.

If you find that you need to develop against a gem that hasn’t yet been released (for instance, Rails often develops against unreleased Rack, Arel, or Mail gems), you can add individual lines in the Gemfile that tell bundler where to find the gems. It will still use the dependencies listed in the .gemspec for dependency resolution, but it now knows exactly where to find the dependency during gem development.

source "http://www.rubygems.org"
 
gemspec
 
# if the .gemspec in this git repo doesn't match the version required by this
# gem's .gemspec, bundler will print an error
gem "rack", :git => "git://github.com/rack/rack.git"

You wouldn’t want to include this information in the .gemspec, which will eventually be released to Rubygems after the in-development gem has also been released. Again, this makes the overall system more resilient, as it doesn’t depend on transient external URLs. This information is purely something used to set up a complete environment during development, which requires some precision.

This is also why we recommend that people do not check in their Gemfile.lock files in gem development. That file is emitted by bundler, and guarantees that all gems, including dependencies of dependencies, remain the same. However, when doing gem development, you want to know immediately when some change in the overall ecosystem breaks your setup. While you may want to insist on using a particular gem from its git location, you do not want to hardcode your development to a very specific set of gems, only to find out later that a gem released after you ran bundle install, but compatible with your version range, doesn’t work with your code.

It’s not a way to guarantee that your code works on all versions that you specified in your .gemspec, but the incentive to lock down the exact versions of each gem simply doesn’t exist when the users of your code will get it via gem install.

Update

Just to clarify, you should check in your Gemfile.lock in applications, and not in gems. The Gemfile.lock remembers the exact versions and sources of every piece of third-party code that you use. For applications, you want this. When developing a gem, this can obscure issues that will occur because gems are deployed (using the gem command) without the benefit of bundler.

Archives

Categories

Blogroll

Meta