Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Ruby’s Implementation Does Not Define its Semantics

When I was first getting started with Ruby, I heard a lot of talk about blocks, and how you could “cast” them to Procs by using the & operator when calling methods. Last week, in comments about my last post (Ruby is NOT a Callable Oriented Language (It’s Object Oriented)), I heard that claim again.

To be honest, I never really thought that much about it, and the idea of “casting” a block to a Proc never took hold in my mental model, but when discussing my post with a number of people, I realized that a lot of people have this concept in their mental model of Ruby.

It is not part of Ruby’s semantics.

In some cases, Ruby’s internal implementation performs optimizations by eliminating the creation of objects until you specifically ask for them. Those optimizations are completely invisible, and again, not part of Ruby’s semantics.

Let’s look at a few examples.

Blocks vs. Procs

Consider the following scenarios:

def foo
  yield
end
 
def bar(&block)
  puts block.object_id
  baz(&block)
end
 
def baz(&block)
  puts block.object_id
  yield
end
 
foo { puts "HELLO" } #=> "HELLO"
bar { puts "HELLO" } #=> "2148083200\n2148083200\nHELLO"

Here, I have three methods using blocks in different ways. In the first method (foo), I don’t specify the block at all, yielding to the implicit block. In the second case, I specify a block parameter, print its object_id, and then send it on to the baz method. In the third case, I specify the block, print out its object_id and yield to the block.

In Ruby’s semantics, these three uses are identical. In the first case, yield calls an implicit Proc object. In the second case, it takes an explicit Proc, then sends it on to the next method. In the last case, it takes an explicit Proc, but yields to the implicit copy of the same object.

So what’s the & thing for?

In Ruby, in addition to normal arguments, methods can receive a Proc in a special slot. All methods can receive such an argument, and Procs passed in that slot are silently ignored if not yielded to:

def foo
  puts "HELLO"
end
 
foo { something_crazy } #=> "HELLO"

On the other hand, if you want a method to receive a Proc in that slot, and thus be able to yield to it, you specify that by prefixing it with an &:

def foo
  yield
end
 
my_proc = Proc.new { puts "HELLO" }
foo(&my_proc)

Here, you’re telling the foo method that my_proc is not a normal argument; it should be placed into the proc slot and made available to yield.

Additionally, if you want access to the Proc object, you can give it a name:

def foo(&block)
  puts block.object_id
  yield
end
 
foo { puts "HELLO" } #=> "2148084320\nHELLO"

This simply means that you want access to the implicit Proc in a variable named block.

Because, in most cases, you’re passing in a block (using do/end or {}), and calling it using yield, Ruby provides some syntax sugar to make that simple case more pleasing. That does not, however, mean that there is a special block construct in Ruby’s semantics, nor does it mean that the & is casting the a block to a Proc.

You can tell that blocks are not being semantically wrapped and unwrapped because blocks passed along via & share the same object_id across methods.

Mental Models

For the following code there are two possible mental models.

def foo(&block)
  puts block.object_id
  yield
end
 
b = Proc.new { puts "OMG" }
puts b.object_id
foo(&b) #=> 2148084040\n2148084040\nOMG

In the first, the &b unwraps the Proc object, and the &block recasts it into a Proc. However, it somehow also wraps it back into the same wrapper that it came from into the first place.

In the second, the &b puts the b Proc into the block slot in foo‘s argument list, and the &block gives the implicit Proc a name. There is no need to explain why the Proc has the same object_id; it is the same Object!

These two mental models are perfectly valid (the first actually reflects Ruby’s internal implementation). I claim that those who want to use the first mental model have the heavy burden of introducing the new concept to the Ruby language of a non-object block, and that as a result, it should be generally rejected.

Metaclasses

Similarly, in Ruby’s internal implementation, an Object does not get a metaclass until you ask for one.

obj = Object.new
# internally, obj does not have a metaclass here
 
obj.to_s
# internally, Ruby skips right up to Object when searching for #to_s, since
# it knows that no metaclass exists
 
def obj.hello
  puts "HELLO"
end
# now, Ruby internally rewrites obj's class pointer to point to a new internal
# metaclass which has the hello method on it
 
obj.to_s
# Now, Ruby searches obj's metaclass before jumping up to Object
 
obj.to_s
# Now, Ruby skips the search because it's already cached the method
# lookup

All of the comments in the above snippet are correct, but semantically, none of them are important. In all cases, Ruby is semantically looking for methods in obj‘s metaclass, and when it doesn’t find any, it searches higher up. In order to improve performance, Ruby skips creating a metaclass if no methods or modules are added to an object’s metaclass, but that doesn’t change Ruby’s semantics. It’s just an optimization.

By thinking in terms of Ruby’s implementation, instead of Ruby’s semantics, you are forced to think about a mutable class pointer and consider the possibility that an object has no metaclass.

Mental Models

Again, there are two possible mental models. In the first, Ruby objects have a class pointer, which they manipulate to point to new metaclass objects which are created only when methods or modules are added to an object. Additionally, Ruby objects have a method cache, which they use to store method lookups. When a method is looked up twice, in this mental model, some classes are skipped because Ruby already knows that they don’t have the method.

In the second mental model, all Ruby objects have a metaclass, and method lookup always goes through the metaclass and up the superclass chain until a method is found.

As before, I claim that those who want to impose the first mental model on Ruby programmers have the heavy burden of introducing the new concepts of “class pointer” and “method cache”, which are not Ruby objects and have no visible implications on Ruby semantics.

Regular Expression Matches

In Ruby, certain regular expression operations create implicit local variables that reflect parts of the match:

def foo(str)
  str =~ /e(.)l/
  p $~
  p $`
  p $'
  p $1
  p $2
end
 
foo("hello") #=> #<MatchData "ell" 1:"l">\n"h"\n"o"\n"l"\nnil

This behavior is mostly inherited from Perl, and Matz has said a few times that he would not support Perl’s backrefs if he had it to do over again. However, the provide another opportune example of implicit objects in Ruby.

Mental Models

In this case, there are three possible mental models.

In the first mental model, if you don’t use any $ local variables, they don’t exist. When you use a specific one, it springs into existence. For instance, when using $1, Ruby looks at some internal representation of the last match and retrieves the last capture. If you use $~, Ruby creates a MatchData object out of it.

In the second mental model, when you call a method that uses regular expressions, Ruby walks back up the stack frames, and inserts the $ local variables on it when it finds the original caller. If you later use the variables, they are already there. Ruby must be a little bit clever, because the most recent frame on the stack (which might include C frames, Java frames, or internal Ruby frames in Rubinius) is not always the calling frame.

In the last mental model, when you call a method that uses regular expressions, there is an implicit match object available (similar to the implicit Proc object that is available in methods). The $~ variable is mapped to that implicit object, while the $1 variable is the equivalent of $~[1].

Again, this last mental model introduces the least burdensome ideas into Ruby. The first mental model introduces the idea of an internal representation of the last match, while the second mental model (which again, has the upside of being how most implementations actually do it) introduces the concept of stack frames, which are not Ruby objects.

The last mental model uses an actual Ruby object, and does not introduce new concepts. Again, I prefer it.

Conclusion

In a number of places, it is possible to imbue Ruby semantics with mental models that reflect the actual Ruby implementation, or the fact that it’s possible to imagine that a Ruby object only springs into existence when it is asked for.

However, these mental models require that Ruby programmers add non-objects to the semantics of Ruby, and requiring contortions to explain away Ruby’s own efforts to hide these internals from the higher-level constructs of the language. For instance, while Ruby internally wraps and unwraps Procs when passing them to methods, it makes sure that the Proc object attached to a block is always the same, in an effort to hide the internal details from programmers.

As a result, explaining Ruby’s semantics in terms of these internals requires contortions and new constructs that are not natively part of Ruby’s object model, and those explanations should be avoided.

To be clear, I am not arguing that it’s not useful to understand Ruby’s implementation. It can, in fact, be quite useful, just as it’s useful to understand how C’s calling convention works under the covers. However, just as day-to-day programmers in C don’t need to think about the emitted Assembler, day-to-day Ruby programmers don’t need to think about the implementation. And finally, just as C implementations are free to use different calling conventions without breaking existing processor-agnostic C (or the mental model that C programmers use), Ruby implementations are free to change the internal implementation of these constructs without breaking pure-Ruby code or the mental model Ruby programmers use.

14 Responses to “Ruby’s Implementation Does Not Define its Semantics”

From PickAxe:

“If the last argument to a method is preceded by an ampersand, Ruby assumes that it is a Proc object. It removes it from the parameter list, converts the Proc object into a block, and associates it with the method.”

Hence being referred by some as the “blockinator”. i.e. take the argument prefixed by “&” and convert it to a block so that the receiver can invoke it via “yield”. This idea of conversion to a block is what steered people towards overriding to_proc (for better or worse).

I can’t see how implementation vs semantics comes into play, though. If you’re saying that people are misunderstanding the semantics of blocks (or other Ruby features) by focusing on the implementation, I don’t think that’s true at all.

Cheers,

Nathan de Vries

Your posts are thoughtful, well written and always a joy to read. It is great to learn some of the deeper details of Ruby as well as the importance of the differences between semantics and implementation. I am glad to see that so many of the software frameworks I use daily are in your capable hands.

After reading many of your articles from my RSS reader I thought I’d stop by and say “hey”. You’re doing a great job and I appreciate it.

Peace,
Daniel X. Moore

Thanks a lot for articles about block/procs.
Sometimes knowledge of implementations as premature optimization (that is the root of all evil :)

Thanks for this, Yehudah! These posts are exceedingly helpful to a newb trying to get my mental models straight. Much appreciated.

In the interests of pedagogical clarity, there’s one point where I think I follow but is a little confusing.

“In Ruby, in addition to normal arguments, methods can receive a Proc in a special slot. All methods can receive such an argument, and Procs passed in that slot are silently ignored if not yielded to…”

Should this paragraph be using the word “block” instead of “Proc”?

Because the next graph says “On the other hand, if you want a method to receive a Proc in that slot…” So, as written, the two “hands” are confusingly similar.

Nice post, Yehuda.

I think a point also worth making (again, because I believe you’ve already mentioned this elsewhere) is that Procs are just objects in Ruby, and methods can take them as regular arguments also:

class Greeter
def call; puts “Hello”; end
end

def foo(thingy)
thingy.call
end

foo(Proc.new {puts “Hello”})
foo(Greeter.new)

In both cases, the result is exactly the same — Proc are just objects. This feature lets Rack support applications that are just Procs as well as objects that respond to call without any special logic, which is extremely powerful.

Out of curiosity, did you consult the draft ISO standard in writing this article?

Not only it just an optimization that metaclasses are lazily created, it is also a requirement for implementation.

Given every object has its own metaclass
And every metaclass is an object
And metaclasses are eagerly created #hypothetically
When a Ruby program starts up
Then it runs out of memory immediately

@Justice, actually in the case of ordinary Ruby classes (T_CLASS) metaclasses are not created lazily but at class creation time (at least in MRI and YARV).

Yehuda, I compared your mental model to the one in the draft ISO spec. Details here: http://avdi.org/devblog/2010/03/01/getting-pedantic-about-ruby-semantics/ Short version: your model differs from the spec; but so, apparently, does MRI.

Mental model are a good metaphor to understand the semantic modifiability. But what is the “true source” for this model? MRI? JRuby? Rubinus? Or should we skip this part and just form our own model according to Rubys language specification project.

Could you please explain how does #block_given? works?
I am having no idea how it can detect the block without accepting any argument by itself.

Thanks.

It may be a bit late to reply to this blog post, but regarding your latest example, aren’t $-prefixed variables global in Ruby? In this case, the $ regexp variables are simply global variables put to new values every time a regexp is executed.
That makes the mental model and the implementation both simpler by the way.

No, they are not global.

def foo
  "hello" =~ /hell/
  bar
  puts $&
end
 
def bar
  "hello" =~ /ell/
end

If you run foo, you get hell outputted, not ell, even though the regex match in bar ran temporally after the one in foo.

Leave a Reply

Archives

Categories

Meta