Yehuda Katz is a member of the Ruby on Rails core team, and lead developer of the Merb project. He is a member of the jQuery Core Team, and a core contributor to DataMapper. He contributes to many open source projects, like Rubinius and Johnson, and works on some he created himself, like Thor.

@andrewgleason It was pretty fun :)

Archive for January, 2010

The Irony of the iPad: A GREAT Day for Open Technologies

With the announcement of the iPad, the usual suspects have come out decrying a closed, proprietary, fully locked down system.

For instance, a story on the top of Hacker News today says:

This is what I asked in January 2007 on this site, shortly after the original iPhone was launched:

“1. Will Apple lock down the iPhone, blocking Flash, Java, custom widgets, and open development from its new platform?

2. Could Apple’s multi-touch patents actually stifle growth of new, interactive displays?”

Unfortunately, that turned out to prescient

And the FSF is out there calling this an unprecedented march of DRM:

With new tablet device, Apple’s Steve Jobs pushes unprecedented extension of DRM to a new class of general purpose computers

It’s a fair initial reaction. Apple didn’t build a general-purpose computer as its next entry into the market. Instead, they built a heavily proprietary, locked down device. In order to install an application onto the device, Apple must approve the application.

I don’t need to address the merits of the argument against how Apple handles native applications, because it’s irrelevant. A much, much larger force is at work here.

With the iPad, Apple has created two platforms. First, they have produced a heavily proprietary, native platform that requires Apple approval and has significant Apple restrictions. But ironically, with their heavy focus on improving the quality of Safari and the HTML standard, they have shipped the iPad with a platform based on open, unencumbered technologies.

If you haven’t been paying attention, over the past couple of years, the web platform has gotten offline APIs, improved caching support, local storage (on Safari, that includes an on-device SQLite database accessible through JavaScript), CSS-based animations, and custom, downloadable fonts. Mobile Safari has support for gestures, Geolocation, and hardware-accelerated graphics.

Additionally, Apple has remained at the forefront of these technologies, literally building some of them for mobile devices (hardware-accelerated animations were built for the iPhone, and by extension, the iPad). The Open Source Webkit project has remained extremely active, and in fact, has only accelerated progress since Apple first released its Native SDK, so Apple’s “locked down” strategy has a very carefully carved out intentional exception.

Apple even makes it easy to take a web app and put it on the home screen amongst normal apps. When you do this, the iPhone downloads all the assets in the HTML5 cache manifest to make the work better as an offline app. This is how I use Gmail on my iPhone (because Google knows what’s going on in this space, they leverage new tech in Safari quite well). When Apple rejected Google Voice, Google immediately built a Safari version of the app. The way they tell it:

Already, Google says it is readying a replacement for the Google Voice app that will offer exactly the same features as the rejected app—except that it will take the form of a specialized, iPhone-shaped Web page

Ironically, despite claims that not allowing Flash or Java represent a victory for proprietary technologies and a loss for open technologies, they represent quite the opposite. By restricting the web platform on the iPhone and iPad to open, patent-free, technologies, Apple has created a highly desirable market for pure-HTML5 apps. This is, frankly, a win for supporters of open technologies.

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • DZone
  • LinkedIn
  • Netvibes
  • Reddit
  • Slashdot
  • StumbleUpon
  • Technorati
  • TwitThis

jQuery 1.4 and Malformed JSON

Today, we released jQuery 1.4, a mostly backward compatible release with a few minor quirks. One of these quirks is that jQuery now uses the native JSON parser where available, rejecting malformed JSON.

This probably sounds fine, except that by malformed JSON, we mostly mean:

{foo: "bar"}

Because this is valid JavaScript, people have long assumed it was valid JSON, and a lot of hand-coded JSON is written like this. Thankfully, JSON automatically generated by Rails does not have this problem, so most of you guys should not have an issue here.

If you have server-side JSON that can’t be quickly fixed in the transition to jQuery 1.4, you can use the compatibility plugin, which tries to patch a number of small incompatibilities, or you can use this workaround:

$.ajax({url: "/url", 
  dataType: "json",
  success: function(json) {
    // do something with json
  }
});
 
// becomes
 
$.ajax({url: "/url",
  dataType: "text",
  success: function(text) {
    json = eval("(" + text + ")");
    // do something with JSON
  }
});

Essentially, this is what jQuery used to do to evaluate JSON, so with these small changes you can get the old behavior.

Again, though, you should probably be modifying your busted server-side code to stop sending malformed JSON!

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • DZone
  • LinkedIn
  • Netvibes
  • Reddit
  • Slashdot
  • StumbleUpon
  • Technorati
  • TwitThis

ActiveModel: Make Any Ruby Object Feel Like ActiveRecord

Rails 2.3 has a ton of really nice functionality locked up in monolithic components. I’ve posted quite a bit about how we’ve opened up a lot of that functionality in ActionPack, making it easier to reuse the router, dispatcher, and individual parts of ActionController. ActiveModel is another way we’ve exposed useful functionality to you in Rails 3.

Before I Begin, The ActiveModel API

Before I begin, there are two major elements to ActiveModel. The first is the ActiveModel API, the interface that models must adhere to in order to gain compatibility with ActionPack’s helpers. I’ll be talking more about that soon, but for now, the important thing about the ActiveModel API is that your models can become ActiveModel compliant without using a single line of Rails code.

In order to help you ensure that your models are compliant, ActiveModel comes with a module called ActiveModel::Lint that you can include into your test cases to test compliance with the API:

class LintTest < ActiveModel::TestCase
  include ActiveModel::Lint::Tests
 
  class CompliantModel
    extend ActiveModel::Naming
 
    def to_model
      self
    end
 
    def valid?()      true end
    def new_record?() true end
    def destroyed?()  true end
 
    def errors
      obj = Object.new
      def obj.[](key)         [] end
      def obj.full_messages() [] end
      obj
    end
  end
 
  def setup
    @model = CompliantModel.new
  end
end

The ActiveModel::Lint::Tests provide a series of tests that are run against the @model, testing for compliance.

ActiveModel Modules

The second interesting part of ActiveModel is a series of modules provided by ActiveModel that you can use to implement common model functionality on your own Ruby objects. These modules were extracted from ActiveRecord, and are now included in ActiveRecord.

Because we’re dogfooding these modules, you can be assured that APIs you bring in to your models will remain consistent with ActiveRecord, and that they’ll continue to be maintained in future releases of Rails.

The ActiveModel comes with internationalization baked in, providing an avenue for much better community sharing around translating error messages and the like.

The Validations System

This was perhaps the most frustrating coupling in ActiveRecord, because it meant that people writing libraries for, say, CouchDB had to choose between painstakingly copying the API over, allowing inconsistencies to creep in, or just inventing a whole new API.

Validations have a few different elements.

First, declaring the validations themselves. You’ve seen the usage before in ActiveRecord:

class Person < ActiveRecord::Base
  validates_presence_of :first_name, :last_name
end

To do the same thing for a plain old Ruby object, simply do the following:

class Person
  include ActiveModel::Validations
 
  validates_presence_of :first_name, :last_name
 
  attr_accessor :first_name, :last_name
  def initialize(first_name, last_name)
    @first_name, @last_name = first_name, last_name
  end
end

The validations system calls read_attribute_for_validation to get the attribute, but by default, it aliases that method to send, which supports the standard Ruby attribute system of attr_accessor.

To use a more custom attribute lookup, you can do:

class Person
  include ActiveModel::Validations
 
  validates_presence_of :first_name, :last_name
 
  def initialize(attributes = {})
    @attributes = attributes
  end
 
  def read_attribute_for_validation(key)
    @attributes[key]
  end
end

Let’s look at what a validator actually is. First of all, the validates_presence_of method:

def validates_presence_of(*attr_names)
  validates_with PresenceValidator, _merge_attributes(attr_names)
end

You can see that validates_presence_of is using the more primitive validates_with, passing it the validator class, merging in {:attributes => attribute_names} into the options passed to the validator. Next, the validator itself:

class PresenceValidator < EachValidator
  def validate(record)
    record.errors.add_on_blank(attributes, options[:message])
  end
end

The EachValidator that it inherits from validates each attribute with the validate method. In this case, it adds the error message to the record, only if the attribute is blank.

The add_on_blank method does add(attribute, :blank, :default => custom_message) if value.blank? (among other things), which is adding the localized :blank message to the object. If you take a look at the built-in locale/en.yml looks like:

en:
  errors:
    # The default format use in full error messages.
    format: "{{attribute}} {{message}}"
 
    # The values :model, :attribute and :value are always available for interpolation
    # The value :count is available when applicable. Can be used for pluralization.
    messages:
      inclusion: "is not included in the list"
      exclusion: "is reserved"
      invalid: "is invalid"
      confirmation: "doesn't match confirmation"
      accepted: "must be accepted"
      empty: "can't be empty"
      blank: "can't be blank"
      too_long: "is too long (maximum is {{count}} characters)"
      too_short: "is too short (minimum is {{count}} characters)"
      wrong_length: "is the wrong length (should be {{count}} characters)"
      not_a_number: "is not a number"
      greater_than: "must be greater than {{count}}"
      greater_than_or_equal_to: "must be greater than or equal to {{count}}"
      equal_to: "must be equal to {{count}}"
      less_than: "must be less than {{count}}"
      less_than_or_equal_to: "must be less than or equal to {{count}}"
      odd: "must be odd"
      even: "must be even"

As a result, the error message will read first_name can't be blank.

The Error object is also a part of ActiveModel.

Serialization

ActiveRecord also comes with default serialization for JSON and XML, allowing you to do things like: @person.to_json(:except => :comment).

The main important part of the serialization support is adding general support for specifying the attributes to include across all serializers. That means that you can do @person.to_xml(:except => :comment) as well.

To add serialization support to your own model, you will need to include the serialization module and implement attributes. Check it out:

class Person
  include ActiveModel::Serialization
 
  attr_accessor :attributes
  def initialize(attributes)
    @attributes = attributes
  end
end
 
p = Person.new(:first_name => "Yukihiro", :last_name => "Matsumoto")
p.to_json #=> %|{"first_name": "Yukihiro", "last_name": "Matsumoto"}|
p.to_json(:only => :first_name) #=> %|{"first_name": "Yukihiro"}|

You can also pass in a :methods option to specify methods to call for certain attributes that are determined dynamically.

Here's the Person model with validations and serialization:

class Person
  include ActiveModel::Validations
  include ActiveModel::Serialization
 
  validates_presence_of :first_name, :last_name
 
  attr_accessor :attributes
  def initialize(attributes = {})
    @attributes = attributes
  end
 
  def read_attribute_for_validation(key)
    @attributes[key]
  end
end

Others

Those are just two of the modules available in ActiveModel. Some others include:

  • AttributeMethods: Makes it easy to add attributes that are set like table_name :foo
  • Callbacks: ActiveRecord-style lifecycle callbacks.
  • Dirty: Support for dirty tracking
  • Naming: Default implementations of model.model_name, which are used by ActionPack (for instance, when you do render :partial => model
  • Observing: ActiveRecord-style observers
  • StateMachine: A simple state-machine implementation for models
  • Translation: The core translation support

This mostly reflects the first step of ActiveRecord extractions done by Josh Peek for his Google Summer of Code project last summer. Over time, I expect to see more extractions from ActiveRecord and more abstractions built up around ActiveModel.

I also expect to see a community building up around things like adding new validators, translations, serializers and more, especially now that they can be reused not only in ActiveRecord, but in MongoMapper, Cassandra Object, and other ORMs that leverage ActiveModel's built-in modules.

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • DZone
  • LinkedIn
  • Netvibes
  • Reddit
  • Slashdot
  • StumbleUpon
  • Technorati
  • TwitThis

The Maximal Usage Doctrine for Open Source

I’ve worked on a number of open source projects over the past several years (the most prominent being Merb, Ruby on Rails and jQuery) and have begun to form some thoughts about the usage (aka adoption) of open source projects and the practical effects of license styles.

The Playing Field

There are essentially two kinds of licenses popularly used in open source[1].

The type I’ve worked with most extensively in the BSD or MIT-style license. This license allows unlimited usage, modification, distribution, and commercialization of the source code, with two caveats. First, the copyright notice must be distributed with the source code. Second, to the extent legally enforceable, if you use my MIT-licensed source code, you may not sue me for the effects of using the source.

The other major license type is called copyleft, and essentially leverages copyright law to insist that any modifications to the original source must themselves be made public. The practical effect of this license could not be effected without government-enforced copyright laws.

The most popular copyleft license is GPL, which is used by Linux, and which also has viral characteristics. In short, if you link code covered by the GPL into your own code, you are required to open your code as well (if you distribute it). The GPL is not very clear about what this means for more dynamic languages (is requiring a Ruby file “linking”?).

A less viral version of the GPL, the LGPL, requires that any modifications you make to the software itself be released, but drops the “linking” requirement.

Because of the uncertainty surrounding the viral characteristics of the GPL, legal departments in big corporations are very hostile to the GPL license. Software covered by the LGPL license is more likely to get approval by corporate legal, and MIT/BSD licenses are the most well-liked (probably because they don’t confer any obligations on the corporation).

What I Want When I Write Open Source

When I work on a serious open source project, like Ruby on Rails or jQuery, I have relatively simple desires.

One. I want as many people as possible to use the software I am working on. This is probably a selfish, egoist desire, but it may also be a desire to see useful software actually used.

Two. I want some subset of the many users to be willing to report bugs, and for the varied situations the code is used in to help improve the quality of the software as a result. This is essentially “given enough eyeballs, all bugs are shallow”.

Three. I want people who are interested in improving the software to submit patches. Specifically, I want to receive patches from people who have thought through the problem the bug is trying to fix, and want to receive fewer patches from people who have hacked together a solution just to get it to work. In essence, a high signal-to-noise ratio of patches is more important than a high volume of patches.

Four. I want people who are interested in improving the software long term to become long-term contributors and eventually committers.

Of these concerns, numbers two and three are the highest priorities. I want to expose the code to as much real-world pressure as possible, and I want as many high-quality patches to fix those bugs as possible. If I had to choose between those two, I would pick number two: exposing my code to real-world stress is the best way to rapidly ferret out bugs and incorrect design assumptions that I know of.

Meeting Those Requirements

Starting out with the easiest, my first desire, to have my software used as much as possible, is most easily satisfied by an extremely liberal usage policy. Adding restrictions on the use of software I write reduces its adoption almost by definition.

Much more importantly, the same can be said about exposing code to real world stresses. By far the most important way to achieve this goal is to make it as easy as possible for as many people as possible to use the code.

If only 1% of all proprietary users of the source ever report bugs, that’s 1% of potentially thousands of users, as opposed to 100% of the zero proprietary users who were able to use the software under a more restrictive usage scheme. In practice, this number is much more than 1%, as proprietary users of software experience and report bugs just like open source users do.

The only real counter-argument to this is that by forcing users to contribute, some number of proprietary users will be forced to become open source users, and their contributions will outweigh the smaller contributions of proprietary users. In practice, proprietary users choose proprietary solutions instead when they are forced to choose between restrictive open source usage schemes and other proprietary software.

There is also much to be said for exposing open source tools into proprietary environments.

There is also much to be said for introducing proprietary developers to the open source ecosystem. Proprietary developers have access to things like paid time, access to obscure usage scenarios, and access to markets with low open source penetration that the open source community lacks.

The next major desire I have while working on open source is a steady stream of high-quality patches. This should be advantage copyleft, because all users of the software are forced to contribute back. However, since copyleft licenses are not used in proprietary environments anyway, the patches to open source projects from those environments under more permissive licenses are much more numerous. Again, even if only a few percent of proprietary users contribute back to the project, that is significantly more contributions than the 100% of zero proprietary users.

Also importantly, the patches are contributed by much more dedicated users of the software, instead of being force-contributions. I have never heard a team member on an open source project say that inadequate patches are received by permissive-license software, and that this problem would be solved by going to a more restrictive model.

Finally, I can only speak for jQuery and Rails (and other smaller open source projects I work on), but a large number of new long-term contributors became involved while working on corporate projects, where a permissive license made the decision to use the software in the first place feasible.

Lower Barrier to Entry Helps Meet Open Source Goals

Regardless of how much of the above argument you agree with, it is clear that copyleft licenses intentionally impose a higher barrier to entry for usage than more permissive licenses.

For projects that use more permissive licenses, the fact that many proprietary users of their software do not contribute back changes is a feature, not a bug.

That’s because we don’t focus on all of the people who use our software without contributing back. Instead, we focus on all the users, bug reports, patches, and long term contributors we have gained by keeping the barrier to entry as low as possible.

In the end, the world of ubiquitous open source is here. And we made it without having to resort to coercion or the formation of an entirely self-contained set of software. We made it by building great software and convincing real users that it was better than the alternative.

Postscript: Linux

Linux is a peculiar example because its license has not impeded its usage much. In part, that is because most users of Linux do not make changes to it, and the linking characteristics of the GPL license rarely come into play.

In cases like this, I would argue that copyleft licenses are close enough to usage maximization to get most of the benefits of more permissive licenses. However, it’s not clear to me what benefits the copyleft licenses provide in those (relatively rare) cases.

[1] There are other kinds of licenses, like the Affero license, which is even more restrictive than the GPL license. I would classify Affero as copyleft.

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • DZone
  • LinkedIn
  • Netvibes
  • Reddit
  • Slashdot
  • StumbleUpon
  • Technorati
  • TwitThis

The Craziest F***ing Bug I’ve Ever Seen

This afternoon, I was telling a friend about one of my exploits tracking down a pretty crazy heisenbug, and he said he thought other people would be interested in hearing about it. So let me tell you about it.

Before you continue, if you’re not interested in relatively arcane technical details, feel free to skip this post. It’s here mainly because a friend said he thought people would be interested in it.

Our Story Begins

Our story begins with a small change in the way Ruby 1.9 handles implicit coercion. The most common case of implicit coercion is in Array#flatten. The basic logic is that Ruby checks to see if an element of the Array can be itself coerced into an Array, which is then does before flattening.

There are two steps to the process:

  1. Check to see if the element can be coerced into an Array
  2. If it can, call to_ary on the element, and repeat the process recursively

In Ruby 1.8, the process is essentially the following:

if obj.respond_to?(:to_ary)
  obj.__send__(:to_ary)
else
  obj
end

In Ruby 1.9, it was changed to:

begin
  obj.__send__(:to_ary)
rescue NoMethodError
  obj
end

Of course, the internal code was implemented in C, but you get the idea. This change subtly effects objects that implement method_missing:

class MyObject
  def method_missing(meth, *)
    if meth == :testing
      puts "TESTING"
    else
      puts "Calling Super"
      super
    end
  end
end

In Ruby 1.8, #flatten will first call #respond_to? which returns false and therefore doesn’t ever trigger the #method_missing. In Ruby 1.9, #method_missing will be triggered blindly, and because the call to super should raise a NoMethodError, we will essentially get the same result.

There are some subtle differences here, but for the vast majority of cases, the behavior is identical or close enough to not matter.

A Weird Quirk

When the above change landed in Ruby 1.9.2’s head, Rails started experiencing weird, intermittent behavior. As you might know, Rails overrides method_missing on NilClass to provide a guess about what object you were expecting instead of nil. This feature is called “whiny nils” and can be enabled or disabled in Rails applications.

def method_missing(method, *args, &block)
  if klass = METHOD_CLASS_MAP[method]
    raise_nil_warning_for klass, method, caller
  else
    super
  end
end

But sometimes, when nil was inside an Array, and the Array was flattened, the flatten failed with a NameError: undefined local variable or method `to_ary' for nil:NilClass, which resulted in a failure in the flatten entirely.

The bug appeared intermittently, apparently unrelated to the code that was calling it. We worked around it by catching the NameError and reraising a NoMethodError, but the existence of the bug was rather baffling.

The Bug Reappears

When working on bundler, Carl and I saw the bug again. This time, we didn’t want to let it go. In this case, we had four virtually identical tests with four identical stack traces leading to NoMethodError or NameError seemingly at random. Something didn’t add up.

After hunting bugs for a few hours (having dug deeply into Ruby’s source), I brought in Evan Phoenix (of Rubinius). At first, Evan was baffled as well, but he correctly pointed out that the key to understanding what was going on was the difference between NameError and NoMethodError in Ruby.

x = Object.new
x.no_method #=> NoMethodError
 
class Testing
  def vcall_no_method
    no_method #=> NameError
  end
 
  def call_no_method
    self.no_method #=> NoMethodError
  end
end

In essence, when Ruby sees a method call with implicit self (which might be a typo’ed local variable), it raises a NameError, rather than a NoMethodError.

However, it still didn’t add up, because the call to to_ary was internal, and shouldn’t have been randomly interpreted as a vcall (a call to a method with implicit self) or a normal call.

Tracking it Down

Evan dug deeply into the Ruby source, and found how Ruby was determining whether the call was a vcall or a regular call.

static inline VALUE
method_missing(VALUE obj, ID id, int argc, const VALUE *argv, int call_status)
{
    VALUE *nargv, result, argv_ary = 0;
    rb_thread_t *th = GET_THREAD();
 
    th->method_missing_reason = call_status;
    th->passed_block = 0;
    // ...
}

Before calling the method_missing method itself, Ruby sets a thread-local variable called call_status that reflects whether or not the original call was a vcall or a normal call.

Upon further examination, Evan discovered that the call to method_missing in the case of type coercion did not change the thread-local method_missing_reason.

As a result, Ruby raised a NoMethodError or NameError based upon whether the last call to method_missing was a vcall or regular call.

The fix was a simple one-line patch, which allowed us to remove the workaround in ActiveSupport, but it was by far the craziest f***ing bug I’ve ever seen

Share and Enjoy:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • DZone
  • LinkedIn
  • Netvibes
  • Reddit
  • Slashdot
  • StumbleUpon
  • Technorati
  • TwitThis