Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

JavaScript Needs Blocks

While reading Hacker News posts about JavaScript, I often come across the misconception that Ruby’s blocks are essentially equivalent to JavaScript’s “first class functions”. Because the ability to pass functions around, especially when you can create them anonymously, is extremely powerful, the fact that both JavaScript and Ruby have a mechanism to do so makes it natural to assume equivalence.

In fact, when people talk about why Ruby’s blocks are different from Python‘s functions, they usually talk about anonymity, something that Ruby and JavaScript share, but Python does not have. At first glance, a Ruby block is an “anonymous function” (or colloquially, a “closure”) just as a JavaScript function is one.

This impression, which I admittedly shared in my early days as a Ruby/JavaScript developer, misses an important subtlety that turns out to have large implications. This subtlety is often referred to as “Tennent’s Correspondence Principle”. In short, Tennent’s Correspondence Principle says:

“For a given expression expr, lambda expr should be equivalent.”

This is also known as the principle of abstraction, because it means that it is easy to refactor common code into methods that take a block. For instance, consider the common case of file resource management. Imagine that the block form of File.open didn’t exist in Ruby, and you saw a lot of the following in your code:

begin
  f = File.open(filename, "r")
  # do something with f
ensure
  f.close
end

In general, when you see some code that has the same beginning and end, but a different middle, it is natural to refactor it into a method that takes a block. You would write a method like this:

def read_file(filename)
  f = File.open(filename, "r")
  yield f
ensure
  f.close
end

And you’d refactor instances of the pattern in your code with:

read_file(filename) do |f|
  # do something with f
end

In order for this strategy to work, it’s important that the code inside the block look the same after refactoring as before. We can restate the correspondence principle in this case as:

# do something with f

should be equivalent to:

do
  # do something with
end

At first glance, it looks like this is true in Ruby and JavaScript. For instance, let’s say that what you’re doing with the file is printing its mtime. You can easily refactor the equivalent in JavaScript:

try {
  // imaginary JS file API
  var f = File.open(filename, "r");
  sys.print(f.mtime);
} finally {
  f.close();
}

Into this:

read_file(function(f) {
  sys.print(f.mtime);
});

In fact, cases like this, which are in fact quite elegant, give people the mistaken impression that Ruby and JavaScript have a roughly equivalent ability to refactor common functionality into anonymous functions.

However, consider a slightly more complicated example, first in Ruby. We’ll write a simple class that calculates a File’s mtime and retrieves its body:

class FileInfo
  def initialize(filename)
    @name = filename
  end
 
  # calculate the File's +mtime+
  def mtime
    f = File.open(@name, "r")
    mtime = mtime_for(f)
    return "too old" if mtime < (Time.now - 1000)
    puts "recent!"
    mtime
  ensure
    f.close
  end
 
  # retrieve that file's +body+
  def body
    f = File.open(@name, "r")
    f.read
  ensure
    f.close
  end
 
  # a helper method to retrieve the mtime of a file
  def mtime_for(f)
    File.mtime(f)
  end
end

We can easily refactor this code using blocks:

class FileInfo
  def initialize(filename)
    @name = filename
  end
 
  # refactor the common file management code into a method
  # that takes a block
  def mtime
    with_file do |f|
      mtime = mtime_for(f)
      return "too old" if mtime < (Time.now - 1000)
      puts "recent!"
      mtime
    end
  end
 
  def body
    with_file { |f| f.read }
  end
 
  def mtime_for(f)
    File.mtime(f)
  end
 
private
  # this method opens a file, calls a block with it, and
  # ensures that the file is closed once the block has
  # finished executing.
  def with_file
    f = File.open(@name, "r")
    yield f
  ensure
    f.close
  end
end

Again, the important thing to note here is that we could move the code into a block without changing it. Unfortunately, this same case does not work in JavaScript. Let’s first write the equivalent FileInfo class in JavaScript.

// constructor for the FileInfo class
FileInfo = function(filename) {
  this.name = filename;
};
 
FileInfo.prototype = {
  // retrieve the file's mtime
  mtime: function() {
    try {
      var f = File.open(this.name, "r");
      var mtime = this.mtimeFor(f);
      if (mtime < new Date() - 1000) {
        return "too old";
      }
      sys.print(mtime);
    } finally {
      f.close();
    }
  },
 
  // retrieve the file's body
  body: function() {
    try {
      var f = File.open(this.name, "r");
      return f.read();
    } finally {
      f.close();
    }
  },
 
  // a helper method to retrieve the mtime of a file
  mtimeFor: function(f) {
    return File.mtime(f);
  }
};

If we try to convert the repeated code into a method that takes a function, the mtime method will look something like:

function() {
  // refactor the common file management code into a method
  // that takes a block
  this.withFile(function(f) {
    var mtime = this.mtimeFor(f);
    if (mtime < new Date() - 1000) {
      return "too old";
    }
    sys.print(mtime);
  });
}

There are two very common problems here. First, this has changed contexts. We can fix this by allowing a binding as a second parameter, but it means that we need to make sure that every time we refactor to a lambda we make sure to accept a binding parameter and pass it in. The var self = this pattern emerged in JavaScript primarily because of the lack of correspondence.

This is annoying, but not deadly. More problematic is the fact that return has changed meaning. Instead of returning from the outer function, it returns from the inner one.

This is the right time for JavaScript lovers (and I write this as a sometimes JavaScript lover myself) to argue that return behaves exactly as intended, and this behavior is simpler and more elegant than the Ruby behavior. That may be true, but it doesn’t alter the fact that this behavior breaks the correspondence principle, with very real consequences.

Instead of effortlessly refactoring code with the same start and end into a function taking a function, JavaScript library authors need to consider the fact that consumers of their APIs will often need to perform some gymnastics when dealing with nested functions. In my experience as an author and consumer of JavaScript libraries, this leads to many cases where it’s just too much bother to provide a nice block-based API.

In order to have a language with return (and possibly super and other similar keywords) that satisfies the correspondence principle, the language must, like Ruby and Smalltalk before it, have a function lambda and a block lambda. Keywords like return always return from the function lambda, even inside of block lambdas nested inside. At first glance, this appears a bit inelegant, and language partisans often accuse Ruby of unnecessarily having two types of “callables”, in my experience as an author of large libraries in both Ruby and JavaScript, it results in more elegant abstractions in the end.

Iterators and Callbacks

It’s worth noting that block lambdas only make sense for functions that take functions and invoke them immediately. In this context, keywords like return, super and Ruby’s yield make sense. These cases include iterators, mutex synchronization and resource management (like the block form of File.open).

In contrast, when functions are used as callbacks, those keywords no longer make sense. What does it mean to return from a function that has already returned? In these cases, typically involving callbacks, function lambdas make a lot of sense. In my view, this explains why JavaScript feels so elegant for evented code that involves a lot of callbacks, but somewhat clunky for the iterator case, and Ruby feels so elegant for the iterator case and somewhat more clunky for the evented case. In Ruby’s case, (again in my opinion), this clunkiness is more from the massively pervasive use of blocks for synchronous code than a real deficiency in its structures.

Because of these concerns, the ECMA working group responsible for ECMAScript, TC39, is considering adding block lambdas to the language. This would mean that the above example could be refactored to:

FileInfo = function(name) {
  this.name = name;
};
 
FileInfo.prototype = {
  mtime: function() {
    // use the proposed block syntax, `{ |args| }`.
    this.withFile { |f|
      // in block lambdas, +this+ is unchanged
      var mtime = this.mtimeFor(f);
      if (mtime < new Date() - 1000) {
        // block lambdas return from their nearest function
        return "too old";
      }
      sys.print(mtime);
    }
  },
 
  body: function() {
    this.withFile { |f| f.read(); }
  },
 
  mtimeFor: function(f) {
    return File.mtime(f);
  },
 
  withFile: function(block) {
    try {
      var f = File.open(this.name, "r");
      block(f);
    } finally {
      f.close();
    }
  }
};

Note that a parallel proposal, which replaces function-scoped var with block-scoped let, will almost certainly be accepted by TC39, which would slightly, but not substantively, change this example. Also note block lambdas automatically return their last statement.

Our experience with Smalltalk and Ruby show that people do not need to understand the SCARY correspondence principle for a language that satisfies it to yield the desired results. I love the fact that the concept of “iterator” is not built into the language, but is instead a consequence of natural block semantics. This gives Ruby a rich, broadly useful set of built-in iterators, and language users commonly build custom ones. As a JavaScript practitioner, I often run into situations where using a for loop is significantly more straight-forward than using forEach, always because of the lack of correspondence between the code inside a built-in for loop and the code inside the function passed to forEach.

For the reasons described above, I strongly approve of the block lambda proposal and hope it is adopted.

24 Responses to “JavaScript Needs Blocks”

Doesn’t function.bind() take care of the this pointer issue?

Kind if a poor example. You could just return everything up the chain and do block.call(this, f) to keep the context.

Effing fantastic post.

A propos of nothing, I think it’s “Tennent”:

http://www.amazon.com/Principles-Programming-Languages-International-Computing/dp/0137098731

Dave

JavaScript has a weak pun with block lambdas already in the form of Strict Mode lexical scopes, but it’s VERY weak in the minds of JS programmers as strict mode code is nearly non-existant in the real world. It’s fine for you to argue that JS should have this, but you’ll need to make the argument from some *other* perspective than “we need this and *not* shorter functions*. Yes, they occupy some similar head space, but I’d like for this debate to happen on the relative merits you outline, not the pent-up demand of a shorter way to write “function” (which we also need). They’re independent concepts which should be proposed (and standardized) independently.

I think there is an extra ‘that we could take the code’ phrase in the sentence: Again, the important thing to note here is that we could take the code that we can move code into a block without changing it.

You dont need this ensure thingy. If you open a file with block, ruby takes care of closing the handler after leaving the block. So you can just pass the block to your method. Just want to state this, because i nearly never use ensure.

@entropie– what if an exception is raised in the block, stopping execution? I am new to Ruby, but my understanding is that ensure will, well, ensure the execution of tasks like closing a file or an http connection, tasks that Ruby or a particular library might normally handle automatically, but could be prevented from doing so because of an exception. Am I correct?

Why are you blogging about this if you don’t know the meaning of the word “closure?”

While you’re at the proposing something like this for a language that doesn’t have it already, what I feel Ruby is lacking in this regard is uniformity in local return (ie. from the current block or from the function). You can use break to exit the local block, but you can’t use break to exit the local function if that’s the scope you’re at. Technically this violates the rule you describe, but then you could argue that ruby already does by not allowing a break at function scope at all (in 1.8 it raises, I think it parse errors in 1.9).

No coffeescript?

@withFile (f) -> f.read()

Looks like you really want some Ruby style in your JS, and I don’t think that’s going to happen. You say “it’s more elegant” but that’s purely subjective term. Lambdas and anon. functions are harder to debug, and that’s what matters at the end. Maybe the code is not as elegant as you want it to be, but still equally readable, and easier to debug.

I do most of my programming in ruby, and yet I think I prefer not to have javascript blocks, on a purely aesthetic basis. One of the parts that I like about javascript is that its syntax is kindof small.

Frankly, I don’t think I would be willing to trade that in order to satisfy Tennet’s Principle.

“JavaScript library authors need to consider the fact that consumers of their APIs will often need to perform some gymnastics when dealing with nested functions.”

Adding blocks might solve that problem, but would add another one too: *All* javascript users (not just library authors) would have to learn a new language element – with its syntax and semantics. For us ruby programmers, it would not be a significant leap. But not everyone comes from ruby.

I don’t think the benefits out-weight the costs, global-effort-in-man-hours-wise.

I don’t think this is a needed change.
The example that you provide is kinda bogus imo since mtime does 2 completely different things based on age.
It either returns something or returns nothing and does something else (sys.print).
I’d say that function is not well balanced in its returns.
Balancing the returns would pretty much void the return argument.

Aesthetically: I like the syntax {|arg| body} and consistency of this but I don’t like the paren-free syntax.

I really don’t like the different return meanings.
It would seem they just complicate things as the functions that receive the lambdas would be tightly coupled to the type of lambdas that you can pass to them.

IMHO this is a bad example, we do just fine without “magical” APIs that go and close files on the end of executing a function etc. Sure they might have their place but it’s largely a non-problem.

Yehuda, the more I think about it, I kinda like the block lambda proposal. But your example raises some superficial distractions. You need a simple *working* example to sell Javascripters on this idea. Or at least, instead of an imaginary File.open, use fs.openSync(name, “r”) in Node.js. But it’d be best to start with a non-OO example that works in both server-side and browser-side JS.

Also, you really ought to link to the block lambda proposal at the top of this post, along with a good concise summary of Ruby blocks (from a language design perspective, ideally). I don’t like blog posts littered with links, but this is necessary background reading for non-Rubyists.

OO is another whole can of worms, with “this” and “.call(this_value)” and so on in JS. The way other languages deal with OO is “interesting”. Python has the self-as-first-param kludge, which is simpler to implement but violates the correspondence principle to an irritating extent. Lua has it too, if my memory serves. C++ has classes and inheritance and all kinds of craziness, copied by other languages for no good reason. Go eschews inheritance as a Bad Idea… and I vaguely agree. Last I knew, there was a lively debate about the future of OO in Javascript… so, there are two lines of thought to pursue: (1) How would block lambdas fit into JS’s existing kludgy OO system? (2) How would they fit into the various proposals for brand-new OO systems? (And I’m not very interested in that discussion… I only use OO when it’s useful to me, which isn’t very often.)

The fact that the block must be executed immediately seems to make it needlessly complicated as opposed to functions. For example, if you were using an API, you might not know whether the method allows blocks, or requires a function.

This also confuses me: Consider a forEach function:
Array.prototype.forEach = function(block){
var i,l;
for(i=0,l=this.length;i<l;i++){
block(this[i]);
}
}
But then if you were to do the following, then the block would be called again when inactive:
function firstValue(){
[1,2,3].forEach { |x|
return x;
}
}

Right? To avoid this you would need to put a try {…} in the forEach which again makes it even more complicated. I'm new to blocks so maybe I'm missing something.

Javascript really should evolve.
+1 for blocks in Javascript!

I’m not a rubyist and I find this sytax awkward regarding the js philosophy.

I’m used to callback chaining in javascript, i.e something like

function FileInfo(name) {
this.name = name;
}

FileInfo.prototype = {
mtime: function(cbk) {
//open file ‘fname’ and call ‘this.process’ scoped with ‘this’ and with arguments [cbk]
//the method File.process opens the file, calls the cbk and performs cleaning
File.process(fname, this.process, this, [cbk])
},
process: function(cbk, f) {
var mtime = f.mtime();
cbk(mtime);
}
}

And doing this is awkward when the language does not support NIO.

I would rather write :
var mtime = File.mtime(fname) and the language platform should automatically perform NIO if necessary !

using a new syntax to perform the exact same thing

Hi Yehuda,

Thanks for putting forward the case for block lambdas so eloquently. Your post made us implement block lambda support in Stratified JavaScript – https://groups.google.com/d/topic/oni-apollo/eb3gtiwTOD4/discussion

Thanks for the article. I always wondered what is the actual difference between Ruby blocks and JS anonymous functions.

However, I really don’t think that JS would benefit from this kind of behavior because of its asynchronous nature. The example you gave would be asynchronous in JavaScript and the outer function would return before the inner one (the callback) would even start being executed.

There are really very few cases where a block would be useful. I think that the only improvement would be the execution context shift as we would not have to create variables such as self = this.

I approve of the idea for wanting to add such a feature to JavaScript, but I disapprove of the proposed syntax that was chosen. It would be a nightmare having to browse through someone else’s code not knowing whether you are staring at a Bitwise-OR(|) operation, a ternary-like-OR operation(||), a function, a switch, an if statement, a try/catch block or some actual invalid syntax that would error if executed. Its too crypric… I’m sorry, it just doesn’t look like JavaScript anymore. I would have a hard time making heads or tails of such code written with a syntax like that. The readability of it is poor and I think Javascript deserves better treatment than that. Its JavaScript, not Python, not C++, not Java, and not heaven forbid “Lisp”.

Let’s say for example a coder were to mistype an if statement and forgot the parentheses. Now all of a sudden it becomes valid lambda block syntax? How would the JavaScript interpreter know what to do with such a cryptic block? This will make parsing JavaScript much harder and probably introduce loads of bugs along the way among the browsers that try to implement such a complex and confusing syntax. Not to mention the “overhead” that an interpreter would have to deal with when trying to make sense of the unreadable code. This means slower code execution simply because the interpreter wouldn’t know what kind of block its dealing with because the JavaScript engine would need to be re-designed and have to deal with extra syntax scenarios.

I’m in favor of having lambda-like blocks in JavaScript but I dislike the purposed syntax. I would have expected a lamba block to look like a switch block, while, or try/catch block in JavaScript: e.g. lamba{} or lamba(){}

I believe if a lamba-block-like feature were added to ECMAScript, it shouldn’t smell-up the language, as a drawback, with foreign syntax that would make a scripters run up the walls in confusion and frustration. In my experience, working with the GIMP, the “lambda” expression is available in the Lisp programming language as well as “let” identities. Is that what we are essentially doing here? Smelling up JavaScript with Lisp syntax?

I can see manys advantages and disadvantages to such a change. But I think the proposal should be revised a bit more before it goes mainstream so it can better fit-in and blend-in with the already existing JavaScript syntax.

-Ulti

I liked your post. Certainly, as you said, blocks can be more elegant for the case of iterators. But I can think of no more use cases for lambdas in JS. We don’t have mutex or any other synchronization construct in JS, since the language is single threaded. Although there are some synchronous API, the IO paradigm in JS is asynchronous, so I don’t see a clear use case for blocks there. So we don’t need blocks for resource management.
Of course, the syntax for functions in JS is awful, and that’s why things like coffeescript are successful. Of course I expect some modification to the JS syntax in the future.
Good post nonetheless!

This is orthogonal to the overall argument, but the code examples have the File.open inside the exception handling—a common mistake. It should be outside:

f = File.open(“…”)
begin
# …
ensure
f.close
end

@pragdave Why is that? Is it because you don’t want to call close on nil if, for example, File.open causes Errno::ENOENT?

Leave a Reply

Archives

Categories

Meta