Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for September, 2011

A Proposal for Proposals

Over the past few years, I have occasionally expressed frustration (in public and private) about the process for approving new features to the next edition of ECMAScript. In short, the process is extremely academic in nature, and is peppered with inside baseball terms that make it nearly impossible for lay developers to provide feedback about proposed new features. In general, this frustration was met with a general assumption that the current process works the way it does for good reason, and that academic descriptions of the new features was the correct (and only) way to properly discuss them.

I have nothing against new features being described in the language of implementors, but I would like to propose some additions to the current process that would make it significantly easier for language consumers (especially framework and library implementors) to provide timely feedback about the proposal in the hope of making an impact before it’s too late.

I would like proposals for new features to have the following elements, in addition to whatever elements are already normally included (such as BNF for any new syntax).

What Problem Are We Solving?

At a high level, what can language users do now that they could not do before. In some cases, proposals may provide simpler or more convenient ways to achieve already-possible goals. These kinds of proposals are often just as important. For example, there is a current proposal to provide a new syntax for class creation. In this case, the new class syntax significantly improves the experience of building a common JavaScript construct.

Provide Non-Trivial Example Code Showing the Problem

If the proposal is solving a problem that exists in the wild, it should be possible to identify non-trivial examples of the problem rearing its head. At the very least, the process of identifying or synthesizing these examples will help language users understand what real-world problems the proposal is attempting to solve. At best, finding real-world examples will help refine the proposal at an early stage.

Show How the Example Code is Improved With the Proposal

After identifying or synthesizing example code to illustrate the problem, show how the problem would be improved if the proposal was accepted. In some cases, the problem is as simple as “needing a large library to perform this operation” and the solution is “building common functionality into the language”. In an example from a related field, the DOM library, the problem addressed by querySelectorAll was “many incompatible implementations of a CSS3 selector library”. The solution was to build the functionality into the browser.

In this case, a mistake in the querySelectorAll specification, which was resolved by the addition of queryScopedSelectorAll in the Selectors API Level 2 could have been addressed ahead of time by evaluating real-life code using selector engine libraries. Of course, the DOM library is not the same as the language specification, so the example is merely an analogy.

What are Alternate ES3 (or ES5) Solutions to the Same Problem?

Simply enumerate the ways that existing JavaScript developers have attempted to resolve the problem without language support. In small language changes, this overlaps considerably with the previous questions. By having proposal authors enumerate existing solutions to the problem, it will be easier for language users to identify the scope of the solution.

This will allow language users to provide feedback about how the solution on offer stacks up compared to existing pure-JS solutions.

Are there any restrictions that do not exist in original pure-JS solutions?

Are there any restrictions in the proposal that limit its utility as a solution to the problem in question, especially if those restrictions do not apply to solutions currently used by language users.

If the Proposal is For New Syntax, What if Anything Does it Desugar To?

Also, if the proposal desugars, why choose this particular desugaring as opposed to alternatives?

If New Syntax, Can it Be Compiled to ES3 (or ES5)?

If the proposal can desugar to an older version of the specification, can a source-to-source translator be written? If so, is there a reference implementation of a source-to-source translator written in that version?

By writing such a source-to-source translator, existing language users can experiment with the new syntax easily in a browser environment without requiring a separate compilation pass. This also allows users to build an in-browser translation UI (similar to try CoffeeScript), which can improve general understanding of the new syntax and produce important feedback.

To be more specific, what I would like to see here is a general-purpose source-to-source translation engine written in ES3 with a mechanism for plugging in translation passes for specific features. If new features come with translation passes, it would be trivial for language users to try out new features incrementally in production applications (with a nice development-time workflow). This would provide usability feedback at an early enough stage for it to be useful.

If a New Standard Library, Can it Be Polyfilled to ES3 (or ES5)?

If the proposal is for a new library whose syntax is valid in an earlier version of the specification, can it be implemented in terms of the existing primitives available in that version. If necessary, primitives not defined by the language, but provided historically by web browsers, can be used instead. The goal is to provide shims for older browsers so that a much broader group of people can experiment with the APIs and provide feedback.

In general, new libraries that parse in older versions of the specifications should also come with a well-defined mechanism to detect whether the feature is present, to make it easy for library and framework vendors, as well as the general public, to opt their users into the new features where appropriate.

Even if a fully backwards-compatible shim cannot be provided, it is still useful to provide a partial implementation together with a feature detection mechanism. At the very least, error-checking shims can be useful, so language users can easily understand the interface to the proposed library.

Does the Proposal Change any Existing Semantics?

In some cases, the proposal unavoidably changes existing semantics. For example, ES5 changed the semantics of an indirect call to eval (a call to an alias to eval, such as x = eval; x('some code') to use the global environment for the evaluated code. In ES3, indirect calls to eval behaved the same as direct calls to eval.

These cases are rare, and in most cases, require an explicit opt-in (such as the directive "use strict";).

When such changes are made, especially when they do not require an opt-in, they should be explicitly called out in the proposal to gather feedback about their likely impact on existing code. Even when they require an opt-in, information about the frequency of their use could be useful to assess the difficulty of opting in.

Since these opt-ins often enable new features as well as changing existing semantics, understanding the impact of the opt-in on existing code would help language users assess their overall utility and timeframe for adoption. This information could also help drive these decisions.

New Hope for The Ruby Specification

For a few years, a group of Japanese academics have been working on formalizing the Ruby programming language into a specification they hoped would be accepted by ISO. From time to time, I have read through it, and I had one major concern.

Because Ruby 1.9 was still in a lot of flux when they were drafting the specification, the authors left a lot of details out. Unfortunately, some of these details are extremely important. Here’s one example, from the specification of String#[]:

a) If the length of args is 0 or larger than 2, raise a
   direct instance of the class ArgumentError.
b) Let P be the first element of args. Let n be the length
   of the receiver.
c) If P is an instance of the class Integer, let b be the
   value of P.
   1) If the length of args is 1:
      i)   If b is smaller than 0, increment b by n. If b is
           still smaller than 0, return nil.
      ii)  If b >= n, return nil.
      iii) Create an instance of the class Object which
           represents the bth character of
           the receiver and return this instance.

The important bit here is c(1)(iii), which says to create “an instance of the class Object which represents the btw character of the receiver”. The reason for this ambiguity, as best as I can determine, is that Ruby 1.8 and Ruby 1.9 differ on the behavior:

1.8 >> "hello"[1]
=> 101
1.9 >> "hello"[1]
=> "e"

Of course, neither of these results in a direct instance of the class Object, but since Fixnums and Strings are both “instances of the class Object”, this is technically true. Unfortunately, any real-life Ruby code will need to know what actual object this method will return.

Another very common reason for unspecified behaviors is a failure to specify Ruby’s coercion protocol, so String#+ is unspecified if the other is not a String, even though all Ruby implementations will call to_str on the other to attempt to coerce it. The coercion protocol has been underspecified for a long time, and it’s understandable that the group punted on it, but because it is heavily relied on by real-life code, it is important that we actually describe the behavior.

This week, I am in Matsue in Japan for RubyWorld, and I was glad to learn that the group working on the ISO specification sees the current work as a first step that will continue with a more rigid specification of currently “unspecified” behavior based on Ruby 1.9.

The word “unspecified” appears 170 times in the current draft of the Ruby specification. I hope that the next version will eliminate most if not all of these unspecified behaviors in favor of explicit behavior or explicitly requiring an exception to be thrown. In cases that actually differ between implementations (for instance, Rubinius allows Class to be subclassed), I would hope that these unspecified behaviors be the subject of some discussion at the implementor level.

In any event, I am thrilled at the news that the Ruby specification will become less ambiguous in the future!