Yehuda Katz is a member of the Ruby on Rails core team, and lead developer of the Merb project. He is a member of the jQuery Core Team, and a core contributor to DataMapper. He contributes to many open source projects, like Rubinius and Johnson, and works on some he created himself, like Thor.
@yukihiro_matz http://t.co/A5dXWqiX "You don't have to do everything that you can think of" I think of this when I hear of private_const
Ruby is NOT a Callable Oriented Language (It’s Object Oriented)
February 21st, 2010
I recently ran across a presentation entitled Python vs. Ruby: A Battle to the Death. I didn’t consider it to be a particularly fair battle, and may well reply in more detail in a later post.
However, what struck me as most worthy of explanation was the presenter’s concern about the fact that Procs are not callable via parens.
x = Proc.new { puts "HELLO" } x() #=> undefined method `x' for #<Object:0x1001bd298>< x.call #=> "HELLO" x[] #=> "HELLO"
For those coming from a callable-oriented language, like Python, this seems horribly inconsistent. Why are methods called with (), while Procs are called with [].
But what’s going on here is that Ruby doesn’t have a notion of “callable”, like Python. Instead, it has a pervasive notion of Object. Here, x is an instance of the Proc class. Both call and [] are methods on the Proc class.
Designing for the Common Case: Calling Methods
Coming from a callable-oriented language, this might seem jarring. But Ruby is designed around Objects and the common cases of working with objects.
Calling methods is far more common than wanting to get an instance of Method, so Ruby optimizes the case of calling methods, with a slightly less elegant form to access an instance:
class Greeter def say puts "Hello world!" end end Greeter.new.say #=> "Hello world!" Greeter.method(:new) #=> #<Method: Class#new> Greeter.new.method(:say) #=> #<Method: Greeter#say> # This is so that you don't have to say: Greeter.new().say()
Ruby considers the common case of calling methods, and optimizes that case while still making the less common case possible.
Designing for the Common Case: How Blocks Are Really Used
One of the reasons that the Proc.new case throws off Rubyists in debates is that Rubyists literally never call Proc objects using [].
In Ruby, Procs are the object passed to methods when using block syntax. Here is how Procs are actually used:
def hello puts "Hello world!" yield puts "Goodbye cruel world!" end hello { puts "I am in the world!" }
When examining languages that support passing anonymous functions to functions (like JavaScript), it turns out that the vast majority of such cases involve a single anonymous function. As a result, Matz (inspired by Smalltalk) built in the idea of a block as a core construct in Ruby. In the vast majority of cases, blocks are created using lightweight syntax ({} or do/end) and called using yield.
In some cases, blocks are passed from one method to the next, before they are finally called using yield:
def step1(&block) puts "Step 1" step2(&block) end def step2 puts "Step 2" yield end step1 { puts "Do the action!" } #=> "Step 1\nStep 2\nDo the action!"
As you can see, Ruby builds in the idea of calling a block into the core language. I searched through Rails (a fairly large codebase) for instances of using [] to call a Proc and while we use blocks extremely commonly, we don’t use [] to call them.
I suspect that the reason this comes up is that people who are used to having to define standalone functions, pass them around, and then call them are looking for the analogous constructs in Ruby, but are missing the different paradigm used by Ruby.
Consistent Method Execution
Fundamentally, the issue here comes down to this:
def foo proc {} end foo()
In Ruby, methods are invoked with our without parentheses. All methods return values, which are always Objects. All Objects have methods. So foo() is a method call that returns a Proc object. It’s extremely consistent, with very few axioms. The fact that the axioms aren’t the same as those in a callable-oriented language doesn’t make them “weird”.
Brian Jenkins, Posted February 21, 2010, 4:51 am
I didn’t even know squarekets could be used to call procs.
I learned it from pythonistas a few minutes ago.
Fro, Posted February 21, 2010, 5:47 am
Nice article.
Indeed, the main difference lay in the way ruby coders see the implementation of Procs…
… there’s truly no need to compare Python and Ruby in that context. :-)
Tore Darell, Posted February 21, 2010, 7:10 am
I think one of the things that makes Ruby what it is is the forced encapsulation of attributes. Ruby objects don’t have attributes, they only have methods that access instance variables that are hidden from the outside. This contrasts with for example JavaScript (I don’t know Python very well), where objects only have attributes, but some of those attributes (properties in JS lingo) happen to be functions, in which context they’re called methods.
I think both implementations are elegant, but we can’t have both at the same time*. So, while Ruby can’t pass functions around as easily and elegant as JavaScript, JavaScript has no built-in encapsulation and programmers have to resort to getThis and setThat methods, exposing the difference between properties and methods. And, as all good programmers know, we should encapsulate properties that are part of the object’s interface, and writing getters and setters for everything gets tiring pretty fast. Ruby’s lack of first class functions may seem a little inelegant, but all in all I think Ruby’s choice resulted in less frustration for the programmer.
* I know JavaScript now actually has both functions-as-properties and encapsulated properties through getters and setters, but this comes at the cost of added complexity and (IMO) a less elegant implementation.
Marcelo Silveira, Posted February 21, 2010, 7:30 am
Thanks for another great post. Have been learning a lot from your ruby articles.
Erik Gävert, Posted February 21, 2010, 9:16 am
The problem with Proc is really its name which gives the impression of much more complexity than there really is.
It’s just an object storing a block for future use.
None of the complexity which would be there in most other languages.
dude, Posted February 21, 2010, 11:37 am
I’ve been waiting for a high profile Rubyist like you to comment on this. Well done!
I wasn’t shocked when Gary ‘fudged some numbers’ in the Ruby-related parts of his talk. As a Python-to-Ruby convert most of what he said didn’t sit well with me… I just don’t subscribe to the Python dogma (`import this`, etc). He’s clearly biased and only using Ruby because he has to (oh, and blocks – a missing feature in Python I don’t know you could live without).
Anyway, cheers!
Dean Wampler, Posted February 21, 2010, 11:53 am
By way of comparison, Scala is also fully object oriented, but the objects can be “callable”. You can treat any object as a “function” if it defines an apply method. In other words, myObj(“Hello World”) is sugar for myObj.apply(“Hello World”).
Rich Collins, Posted February 21, 2010, 11:56 am
Io presents an interesting alternative. There are only Blocks (functions with assignable scope). Blocks have an activatable slot. If you set it to true, the block is activated when an object receives a message that looks up the slot with the block. If false, it returns the block.
ex.
sayHello := block(“hello” println)
sayHello //returns the block
sayHello setIsActivatable(true)
sayHello //prints “hello”
getSlot(“sayHello”) //returns the block
Konstantin Haase, Posted February 21, 2010, 12:04 pm
Actually, you can make that work, somewhat, in ruby: http://gist.github.com/310410
Avdi, Posted February 21, 2010, 12:10 pm
We may not use #[] to call procs, but using Proc#call() isn’t all that uncommon. I suspect that most Rubyists have come to the same conclusion I did WRT Proc#[], which is that it’s a confusing way to say “invoke this callable”, and semantically inconsistent with other uses of #[].
James Edward Gray II, Posted February 21, 2010, 12:33 pm
I actually like Proc#[] and use it fairly often. What I like about it is how well it duck types with a Hash or anything else that supposes normal indexing.
Giles Bowkett, Posted February 21, 2010, 1:41 pm
Same here – I use it all the time. Not as often as collect &:foo, but way more often than call. “Rubyists literally never call Proc objects using []” is literally not accurate.
Also, what you showed as the more common use case with procs, that’s not a proc, dude. That’s a block. A proc is a container object for a block, a proxy that allows you to treat code as data, even though Ruby isn’t homoiconic. Code passed to a method inside a pair of curly braces is just a plain old-fashioned block. If you called block_given? in that method, you’d get true.
To get a proc in there, you’d need:
hello lambda {puts “hello world!”}
or the obviously inferior
hello proc {puts “hello world!”}
or
hi = proc {puts “hello world!”}
hello &proc
Also note that you’re passing the value of {puts “hello world!”} to puts, where you do “puts yield”, so in fact your code doesn’t say “hello world!” at all, it says “nil.”
Giles Bowkett, Posted February 21, 2010, 1:51 pm
Heh, whoops, I mean
hello &hi
wycats, Posted February 21, 2010, 2:42 pm
@giles a block is a Proc that goes into the “block” slot in a method:
def foo yield # call the Proc in the block slot end # call foo with proc { puts "hello" } in the block slot foo { puts "hello" }You could alternatively have done:
# telling ruby that the last argument should come from the block slot, # and is not a normal parameter def foo(&block) block.call # call the Proc called block end # call foo with proc { puts "hello" } in the block slot foo { puts "hello" }but these are identical. The & syntax in a method signature simply says “give a name to the Proc in the block slot”. It’s most commonly used to pass it along to another method, as I showed in my post. In that case, the & syntax in the call says “don’t pass this Proc as a parameter; instead, put it in the block slot”.
filemon, Posted February 21, 2010, 5:03 pm
I’ve got to admit I had no idea that procs can be passed like that:
def step1(&block)
puts “Step 1″
step2(&block)
end
and I find the ampersand very confusing in the step2 method invocation
My first thought was to use this instead:
def step1(&block)
puts “Step 1″
step2 {block.call}
end
filemon, Posted February 21, 2010, 5:47 pm
Sorry, cant edit previous post:
and in a way this approach, ie.
def step1(&block)
puts “Step 1″
step2 {block}
end
Allows for some flexibility, since we can in step2:
def step2
puts “Step 2″
yield.call #directly call the proc
x = yield
step_other {x} # or assign the proc to a variable and move it along again
end
Rick DeNatale, Posted February 21, 2010, 6:20 pm
@wycats no @giles is right.
Procs aren’t instantiated when a block is passed to a method which yields to it.
From Flanagan and Matsumoto “The Ruby Programming Language” p176
“Blocks, like methods, are not objects that Ruby can manipulate. But it’s possible to create an object that represents a block, and this is actually done with some frequency in Ruby programs. A Proc object represents a block. Like a Method object, we can execute the code of a block through the Proc which represents it.”
In order to create a Proc object you need to either pass a block to a method which declares a special argument with an & preceding the name in the method definition, which has the effect of creating the proc object and assigning it to that argument, or pass a block to Proc.new, or to Kernel#lambda, or to Kernel#proc, or in 1.9 by using the new -> lambda literal syntax.
Kernel#lambda and the lambda literal both produce instances of Proc with some subtle differences in how they handle control flow statements like return and break, and the semantics of how arguments are bound when the proc/lambda is called.
One major difference between blocks and procs is that procs are closures, they retain the bindings of any variables visible to them, even after the method which created them returns, and I believe that this is why procs are NOT created in a case like
def foo yield end foo {puts "Hi"}Because there’s no need to capture the bindings, the yield MUST happen by definition before the foo method returns, so there’s no need for the overhead of creating a closure.
Also, I think the connection between Smalltalk blocks and Ruby blocks/procs/lambdas is a bit more tenuous than you suggest.
Smalltalk has (or at least had at the time we developed the X3J20 standard) no yield equivalent. There is no hidden special/environmental block parameter to a Smalltalk method. Blocks were just normal method arguments. This also meant that a Smalltalk method can have zero, one, or many block arguments.
Blocks were invoked by sending them a message like value (for a block with no arguments), value: (for a block with one argument), value:value: (two arguments) etc.
The Ruby yield IS very useful in Ruby’s implementation of enumeration methods, most of which do appear to be modeled after methods in the Smalltalk collection classes. For example, in Smalltalk you might write
Ruby yield effectively optimizes this, by not creating a closure.
But another major use of blocks in Smalltalk is to implement control flow, the Ruby fragment:
if expression "True, too true!" else "Liar, liar, pants on fire" endwould be written in Smalltalk as:
The expression would normally evaluate to a boolean (i.e. a singleton instance of either TrueClass, or FalseClass) and the implementations of ifTrue:ifFalse: would look like this
"In class TrueClass" ifTrue: trueBlock ifFalse: falseBlock ^trueBlock value "In class FalseClass" ifTrue: trueBlock ifFalse: falseBlock ^falseBlock valueDoing this in Ruby is possible to a degree but requires explicitly creating procs/lambdas for the additional block arguments.
On the other hand, although Smalltalk works this way conceptually, most implementations cheated and compiled methods like ifTrue:ifFalse: to control flow bytecodes in the methods in which the ‘invocations’ appeared, and either required the receiver of these methods to actually be a boolean, or treated the case where they weren’t as an exceptional condition handled by the VM.
But getting back to the original pythonista argument. The gripe that you can’t call a block/proc with parens is just quibbling about the flavor of the syntactic sugar, and there’s more than enough similar things in Python which might be tasty to a Python fan don’t fit well with me, like the need to explicitly declare a self parameter in methods.
Rick DeNatale, Posted February 21, 2010, 6:22 pm
Oops, I was afraid that I’d lose control over the indentation in that last comment, and I did, hopefully it still makes some sense.
Dorian, Posted February 21, 2010, 8:41 pm
I don’t see anyone mentioning that the method analogue to the original example is
def five; 5 end
x = method(:five)
in which case calling it is exactly the same as a Proc
x.call
=> 5
x[]
=> 5
It looks like the presenter didn’t have a clear idea of what the code was doing and is just criticizing Ruby for not passing and calling methods and procs identically to how Python treats functions.
wycats, Posted February 21, 2010, 8:55 pm
@dorian absolutely. Ruby is *optimized* for the invoking case, providing a less elegant syntax for getting a callable. Python is *optimized* for the “passing a callable” case, providing a less elegant syntax for invoking a method. Considering the relative frequency of each, it’s clear (at least to me) that a more elegant syntax for invoking methods wins out over a more elegant syntax for retrieving an object representing the method that can be passed around.
wycats, Posted February 21, 2010, 9:07 pm
@rick thanks for that long response. I personally consider the difference between blocks and procs to be equivalent to the fact that Objects don’t get metaclasses until you ask for them. In other words, while it may be a true optimization that (one or more) implementations of Ruby perform, it has no effect on the way a Ruby programmer should think about problems.
In this case, nothing is *lost* by the Ruby programmer considering all blocks to be instances of the Proc class, and the instant he does anything to examine the situation, the illusion is preserved. In other words, I think Flanagan and Matz, in this case, are exposing internals of Ruby that are completely irrelevant to the Ruby language.
Here’s a mind game: could you imagine a Ruby implementation that created a Proc for every block. Would the difference between this implementation and CRuby be detectable in any way?
khelll, Posted February 21, 2010, 9:42 pm
I think there is also a valid reason for this which relates to syntax itself: procs with no arguments will cause an ambiguity when being called. Here is an example:
p = lambda{ ‘hello’}
if Ruby to allow direct calls on blocks, then how should it respond to this?
puts p
is that p.to_s or p() ?
So the problem is that procs with no arguments will confuse the interpreter, and that’s why the above call will default to p.to_s
=>#
If you want to call it so: either .call or [] or even .() in ruby 1.9
puts p.() #=> hello
fernando trasviña, Posted February 22, 2010, 12:36 am
Why does everyone insist that Object Oriented Nature of Ruby is the reason of all its characteristics?
There is no such thing (that i know) as a “callable oriented language”, and if it existed anyway, it didn’t have to go against Object Orientation.
Maybe a unification of Procs, Blocks and Lambdas could lead to less confusion from people coming from other languages. at the end it seems that the only difference is how to call this “executable” constructs on the language.
History has proven that simplification leads to more robust solutions.
Dhananjay Nene, Posted February 22, 2010, 3:29 am
I watched the same presentation as a pythonista and I thought it actually expressed a pythonista’s respect and yearning for code blocks quite well (http://twitter.com/dnene/status/9441446788). I am not sure what you considered unfair in the talk so will look forward to your thoughts on the same. But I guess what it boils down to is in python functions and function references are first class citizens. Isn’t that an easier way to express rather than coining a new term “Callable Oriented” ? Or did I not understand the distinction well enough.
banisterfiend, Posted February 22, 2010, 12:25 pm
here’s a link to the actual video of the ‘Python vs. Ruby: A Battle to the Death’ presentation:
http://vimeo.com/9471538
John Tantalo, Posted February 22, 2010, 6:43 pm
“with our without parentheses” should be “with or without parentheses”
http://www.emendapp.com/sites/yehudakatz.com/edits/1
Giles Bowkett, Posted February 23, 2010, 3:43 pm
nothing is *lost* by the Ruby programmer considering all blocks to be instances of the Proc class
except accurate semantics, which seem to be the whole goal of your post. aren’t you trying to correct a Pythonista’s inaccurate description of Ruby semantics? wouldn’t accuracy be useful in that context?
wycats, Posted February 23, 2010, 11:11 pm
Here are some other “accurate” things:
When you use =~, there is no actual MatchData object, $1 just behaves like it was indexing into an implicit MatchData
Objects do not have metaclasses until you ask for one. A bare object is hooked directly up to it’s class; when you modify its metaclass (like class << or extend) the class pointer is repointed at a metaclass that is only now created.
What all three of these cases have in common is that the details have nothing whatsoever to do with the semantics of Ruby and are purely internal Ruby optimizations (which may or may not exist on other implementations). In cases where a construct is not a Ruby object in MRI’s internals, but becomes one on demand, the construct is semantically equivalent to the object, and we are polluting Ruby’s semantics by exposing internal implementation details as semantically meaningful.
Nathan Youngman, Posted February 24, 2010, 11:12 am
@wycats I was also unaware of the Proc#[] syntax, thanks for the info.
It certainly makes some sense to optimize the syntax for calling methods (without parens). It is an interesting choice, to optimize for the common case rather than providing lower-level constructs that can be used in many ways.
In an unrelated example, I was translating a small Clojure snippet to Ruby and Python. In Ruby I found the standard libraries provide the exact solution to the problem, but building up lower-level constructs towards the solution was awkward.
http://gist.github.com/190489
(if there is a better way, please do fork my gist)
Python’s adoption of Haskell-style comprehensions proved far more powerful.
Ruby is a nice language, but what I am gathering is that Ruby unsurprising fits with a more OOP mindset than a mindset of functional-style composition.
wycats, Posted February 24, 2010, 12:07 pm
Hey Nathan,
I’ve forked your gist at http://gist.github.com/313605 and improved the code, leveraging Enumerators, which were added to 1.8.7+ specifically to solve this sort of problem. Also, for performance (which doesn’t seem to matter much in this case), you might want to convert the predicate Array into a Set first, which will make the subsequent #include? lookups simple Hash lookups (reducing O(m*n) to O(n)).
Giles Bowkett, Posted February 24, 2010, 1:40 pm
In cases where a construct is not a Ruby object in MRI’s internals, but becomes one on demand, the construct is semantically equivalent to the object, and we are polluting Ruby’s semantics by exposing internal implementation details as semantically meaningful.
That’s great but you didn’t answer my questions. If you’re criticizing a Pythonista for failing to understand Ruby, after that Pythonista made an over-generalized assumption about how the language worked based on how the language appeared to work, I don’t see how you can then argue in favor of generalizing away how the language works and saying it’s more important to pay attention to how the language appears to work. How do you justify this? Don’t you understand that you appear to be taking both sides of the issue simultaneously?
Nathan Youngman, Posted February 24, 2010, 2:06 pm
Thanks Yehuda, your version is much cleaner. Obviously I still have a fair bit to learn about Ruby.
Evan Phoenix, Posted February 24, 2010, 6:50 pm
Thought I’d weigh in on the difference between blocks and Procs, having implemented them a few times. There is no semantic concept of a block, there is only the syntax. All semantics related to blocks are within the Proc object.
Rick mentions something about them being different because yield can not capture locals:
One major difference between blocks and procs is that procs are closures, they retain the bindings of any variables visible to them, even after the method which created them returns…
This is just wrong. The lexical closure in both cases is exactly the same. The only difference is that the closure would have a shorter GC lifetime since there is only one reference to the closure. But thats entirely GC related, nothing to do with the closure itself.
Whether or not there is any internal optimization/laziness related to allocation of Proc objects is beside the point, because they are exposed consistently. This policy is the same as what occurs with meta classes.
For those looking to say that there is a difference between a block and a Proc, I’d ask you to demonstrate it. I’m happy to discuss any differences you put forth.
Mario Aquino, Posted February 26, 2010, 11:12 pm
Yehuda… Would you agree that:
foo = Proc.new { ‘foo’ }
foo()
is prettier than:
foo.call
or
foo.()
?
In that regard, Python clearly wins. It just does. But, so what?
banisterfiend, Posted February 27, 2010, 7:03 am
@Mario if you actually read Yehuda’s post you would know the argument was not about which was ‘prettier’ it was about ‘consistency’! :)
Mario Aquino, Posted February 27, 2010, 9:23 am
@banisterfiend, I am not sure I agree about the consistency argument, though perhaps I just need to think about it more. A proc is not exactly an unattached method and a lambda is closer to being a method, but still isn’t. Yet I would like to be able to call them like methods (foo()). I always assumed that we couldn’t because of a parser limitation (or something like that). I would find it more consistent if procs and lambdas were consistent with methods (albeit unattached ones) in how they can be called, though perhaps because they are unattached they should not “be” like methods because they really aren’t methods. That said, my preference is for one syntax over another and it’s just a preference. I probably shouldn’t have said one wins over another because now that I read it, it comes across as a flame, which I didn’t mean.
banisterfiend, Posted February 27, 2010, 11:24 am
@Mario, procs and lambdas are *objects*. methods are not objects (do not confuse methods with instances of Method). It therefore makes sense that you can only invoke a proc/lambda by sending it a message (#call or #.()).
Nonetheless in some languages (i.e Scala) you can invoke method objects by using (), but in Ruby this operator can not be overloaded, probably because parentheses are optional in Ruby.
Ochronus, Posted May 5, 2011, 8:02 am
Thanks for the clarification, a fair article. Also a nice point about distiction among OO languages, they are not all the same, there are different principles.
I’ve also written a few words with code examples about ruby blocks and closures: http://blog.mostof.it/why-ruby-part-two-blocks-and-closures/