Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Ruby on Rails’ Category

Status Update — A Fresh Look at Callbacks

After I finished the first stage of the ActionView refactor I was working on (I’m currently working on some small issues with Josh and getting ready to merge it into the official Rails3 branch), I decided to take a fresh look at the performance numbers. As expected, by initial refactor did not really improve performance (it wasn’t intended to — I mainly just moved things around and cleaned up the code path), but one thing that stuck out at me was how expensive the callback system is.

This is the case in both Rails and Merb, although Merb’s smaller callback feature set makes it somewhat faster than Rails’ at present (but not significantly so; both systems use the same basic approach). So I decided to spend some time today trying to make the callbacks faster. The first thing I noticed was that the ActiveSupport callbacks are fairly limited (for instance, missing around filters), and that ActionPack uses a small part of the ActiveSupport system and then layers on scads of additional functionality.

After spending a few hours trying to improve the performance of the callbacks, I remembered that Carl had considered an interesting approach for the Merb callbacks that involved dispensing with iteration in favor of a single compiled method that inlined the filters. In his original experiments (which supported before, after, and around filters, but not conditions), he was able to make filters run about an order of magnitude faster than Merb’s system.

Effectively, the approach boils down to:

before_filter :foo
after_filter :bar
around_filter :baz

def _run_hookz
  foo
  baz do
    yield
    baz
  end
end 

This completely removes the need for iteration at runtime, and compiles down the hooks into a single, speedy method. What complicated matters a bit was:

  • Rails’ support for conditions (also supported in Merb, but not in Carl’s original prototype)
  • Rails’ support for various filter types, including strings (which get evalled), procs, blocks, and arbitrary objects

After playing around a bit, I managed to support conditional filters as well as all the supported Rails filter types using the speedier callback system.I compared the ActionPack filtering system to the old ActiveSupport callbacks to the new callbacks. According to my benchmarks, the new system is 15-20x faster than the old one. Here are the actual benchmarks for 100,000 iterations (2 before filters, 1 after filter with a proc conditions).

      JRuby Results |
---------------------
actionpack    2.406 |
old           1.798 |
new           0.089 |
  MRI (1.8) Results |
---------------------
actionpack    4.190 |
old           3.063 |
new           0.276 |
  MRI (1.9) Results |
---------------------
actionpack    2.617 |
old           2.137 |
new           0.157 |

Keep in mind that I haven’t retrofitted the ActionPack filters to use the new callback system yet; I had to make ActiveSupport::Callbacks support around_filters first, but that there shouldn’t be any reason it won’t work. Also, adding around_filter support to ActiveSupport::Callbacks means that Test::Unit will now have around filters for free as part of the retrofit (this should affect ActiveRecord as well, for anyone who needs around filters in AR’s hooks).

Also, and this is a big caveat: this is most certainly not an optimization that is likely to significantly help most apps where the overhead is in database access or rendering. However, one of my personal goals for the work I’m doing is to reduce the need for things like Rack Metal by reducing the overhead of rendering a request through the full stack. Filters are used throughout the render path and we shockingly using 10-20% of the 4ms a render :text => “Hello world” was taking up. I was surprised that the action itself was only about 7% of the default render path.

So again, shaving off under 1ms is not likely to help many apps, but it is a step in the direction of making it possible to build high-performance APIs and service tiers on top of the full Rails request cycle. You can follow along this callback work on my optimize_callbacks branch.

P.S. Merb 1.0.8 should be released tomorrow. It’ll include a bunch of pending fixes (including a fix for PIDs being overwritten in production when using merb -i)

Another Dispatch: Step 1 of 2 Complete!

A couple of notes of interest for the work over the past few days:

  • Git is holding up surprisingly well. I’ve done daily merges from rails/rails/master into wycats/rails/master and then into wycats/rails/action_view, and have so far only run into a single conflict, which was easy to resolve. Moving Rails to git made the work we’re doing possible!
  • As far as I can tell, all current cases of using render() as an API now use _render_for_parts(). This is a huge milestone. Now that the four-element tuple is the standard across the board, we can push back the path lookup to the callers and unify behind something like _render_for_template(). There’s still quite a bit of work to do in general (most notably around partials and layouts), but things are coming along quite nicely.
  • I was purposely holding off keeping ActionMailer in sync as I did the refactor, because things were too much in flux for that to really be viable. As it turns out, it was relatively simple to move ActionMailer over to the new approach after everything was done. The only real irritation is the repeated use of the render() options to do mostly the same thing. I’ve moved the code that takes a partial path and converts it into the requisite paths into action_pack/common.rb so it can be used in both ActionController and ActionView. I’m not sure if this will be the final resting place but this refactor definitely pulls out some AC/AV common functionality.
  • I am more convinced than ever that a common AbstractController superclass for both ActionController and ActionMailer is the way to go. This was actually the thing that originally brought me to Merb (that firm conviction), and I’m pretty convinced it’s the right approach for Rails as well. Of course, there’s still quite a bit of work to do before it’s possible to really evaluate it, so stay tuned.
  • Lori has been working hard on the ORM adapter for Rails. Effectively, the adapter will work like: ActiveORM.for(@sequel_obj). If the object is compatible with the ActiveORM API (note that the names et al are still very much in flux), it will be passed through. For instance, ActiveORM.for(@ar_obj) will just return the ActiveRecord object. ActiveORM.for(@sequel_obj), however, will return a wrapper around the object that is compatible with Rails expectations. Example: ActiveORM.for(@sequel_obj).new_record?. Note that this paragraph is speculative and represents work that is still very much in flux.

I’m really looking forward to finishing the work that _render_for_parts was a placeholder for. As I go through the codebase, I’m definitely noticing some obvious perf opportunities (non-trivial perf that should show up very clearly on macro-benches) and I’m looking forward to finishing the initial refactor so I can get into some perf optimizations. I should have some initial numbers within a week or so.

Today’s Dispatch: Weaning ActionView off of content negotiation

I’ve been on “vacation” for the past few days and haven’t had a ton of time to work on continuing refactoring, but I did manage to make a bit more progress, and I figured I’d share.

In Rails 2.2, ActionView had a fair number of content negotiation responsibilities. In particular, ActionView::Base had a method called template_format that looked like this:

def template_format
  if defined? @template_format
    @template_format
  elsif controller && controller.respond_to?(:request)
    @template_format = controller.request.format.to_sym
  else
    @template_format = :html
  end
end

Effectively, ActionView had a small content-negotiation responsibility, that effectively entailed grabbing the first acceptable format and using that. Thankfully, this particular piece of code was usually supplanted by Rails’ respond_to code, which performed more proper content-negotiation, and set the template format directly: @response.template.template_format = mime_type.to_sym.

A number of other places, including the JavaScript helpers, also set the template format directly, which meant that for the most part, the template_format method was just a placeholder that was set by other parts of the system. However, there were more than a few cases where the default template_format method was getting used. Instead of having ActionView call back into the request object, I wanted to modify ActionView’s initialization so that ActionController could feed it the list of acceptable formats directly.

With some work, I was able to make the modification, weaning ActionView off of content negotiation. Interestingly, this is related to my overall refactoring, which currently uses a four-element tuple to represent templates (path, acceptable extensions, prefix, and a partial flag). Now that ActionView uses a list of formats internally, it can easily pass them into find_by_parts, which handles figuring out which template to use based on the ordered list of extensions.

Just a note: find_by_parts is almost certainly a temporary creation to facilitate refactoring. However, for the purposes of refactoring, it is a convenient way to make sure that all pieces of the puzzle are passing around compatible values. In the end, we will probably be passing around Template objects, and this intermediate refactoring step will help us get all of our ducks in a row so that we can get there.

A fun diff that sort of demonstrates where I’m going with this is:

   def _pick_partial_template(partial_path) #:nodoc:
-    if partial_path.include?('/')
-      path = File.join(File.dirname(partial_path), "_#{File.basename(partial_path)}")
-    elsif controller
-      path = "#{controller.class.controller_path}/_#{partial_path}"
-    else
-      path = "_#{partial_path}"
+    unless partial_path.include?('/')
+      prefix = (controller && controller.class.controller_path)
     end
-
-    self.view_paths.find_template(path, self.template_format)
+    view_paths.find_by_parts(partial_path, formats, prefix, true)
   end

As you can see, by normalizing all path lookup to the four-element tuple, which includes merging path prefixes and appending “_” to partials, I was able to remove almost all of the code of this method, now requiring only that the controller_path is specified as a prefix if no existing prefix is supplied as part of the path (that’s so render :partial => “foo” inside of the TestController looks for “test/_foo”).

Again, I probably won’t be able to do a ton of work over the next few days, but I’ll blog when I can. Happy holidays folks!

Rails Refactor Update (and Merb 1.0.7.1)

Over the past few days, I’ve been working on refactoring ActionController and ActionView to clean up the interactions between them. You can follow along on my github fork. Some principles I’m following:

  • ActionController and ActionView should work well standalone
  • All request-related information should be calculated in ActionController and passed through into ActionView
  • ActionView should be responsible for figuring out what path to render; ActionController should pass enough information for ActionView to figure it out.
  • Based on the previous, the information passed from ActionController to ActionView will likely be: template_name, list of acceptable extensions (i.e. [:html, :xml]), prefix (usually the controller name), and partial (a boolean indicating whether the template is a partial). This is not nailed down yet, but it has so far served well.
  • So far, I have unified render(:template), render(:file), and render(:action) to use this new conduit, and will be working on partials tomorrow. Partials are quite complicated so my current plan may have to change slightly when I tackle them.
  • Interesting info: ActionPack has a lot of methods that call each other (somewhat circularly), so it wasn’t really possible to just replace the existing conduit in a straight-forward manner. Instead, I created the new method (currently called find_by_parts, which finds a template based on the components I discussed above) and slowly (very slowly) moved existing callers of find_by_path over to use find_by_parts. Thankfully, the Rails test suite caught the initial errors I made, a huge saving grace of the entire effort.
  • A nice side-effect of moving to a single, clean API between AC and AV is that the final code should be easier to understand. Once I’m further along, I’ll post some details of how exactly the interaction works.

We also released Merb 1.0.7.1 today to fix a few issues that were outstanding as well as a 1.0.7 development mode regression. (In general, we will be using the x.x.x.x moniker for midweek releases and x.x.x for weekly releases). The issues that were fixed:

  • Templates should reload again in development mode
  • An issue in the bundler where gems in your local repository were getting greedily installed is fixed
  • An issue that was preventing bin/thor merb:gem:install use is fixed
  • The error message when gems could not be found was slightly improved

To upgrade:

  • gem install merb (make sure merb-gen 1.0.7.1 is installed)
  • in your app, rm -rf tasks/merb.thor
  • in your app, run merb-gen thor
  • you should receive a prompt asking you to override bin/common.rb. Accept the prompt.
  • you’re done

Happy holidays folks!

Dispatch from the Front Lines

Hey all,

It’s been a very busy few days, but I’m glad to say that the work on the Rails/Merb merge is going quite smoothly. Some things that have already happened:

  • DHH has posted an outline of our plans to bring display/provides into Rails: http://tinyurl.com/providesinrails. The new API resolves some lingering issues I personally had with the provides/display version, and is still undergoing a bit of change.
  • Michael Klishin (antares) has begun work to merge Extlib into ActiveSupport. There are already some obvious incompatibilities, so we will be doing both cleanup on the differences and selecting the fastest implementations. He has also begun porting LazyArray, the underpinnings of the DataMapper “kicker” technique, to ActiveSupport. I should note that Michael was instrumental in merging Merb and DataMapper’s extensions into Extlib in the first place, so this should work out quite well.
  • As DHH mentioned in his post, some work is also being done to work out a better API for the Merb router. While our API works fine, we’re working together on an even better API for Rails3. One of the great things about working with the Rails team is being able to work with very talented API designers on some hard problems. To be clear, we will still support both Rails2′s API and Merb1′s API, because the Merb router, which is serving as the base for this work, decouples the code generation part from the DSL (so any DSL can be used to generate the structures that the router than converts into its compiled matching method).
  • We have started discussing some other issues that could potentially break back-compat in Rails (like Merb’s BlockAwareEnhancer for helpers that take blocks), and I’m happy to report that the conversations have been very productive. Being able to revisit issues that both groups addressed suboptimally in the past with a larger group is proving to be quite enlightening.
  • I have personally begun work to unify the code in Rails that does content negotiation to always go through a single path. You can follow along at http://github.com/wycats/rails/commits/content_negotiation. I will be rebasing this remote branch against current rails master frequently (to keep it in sync with rails head) so you may need to reclone from time to time.

On the Merb front, we will be releasing 1.0.7 this weekend. Expect the first transitional release (Merb 1.1) in the next month or two. It will probably involve porting over basic configuration to Rails’ style (Merb’s Kernel#* style is pretty icky) and probably starting to create a shim for some of the more obvious differences (before/before_filter comes to mind). It’ll be the first of several releases specifically designed to help you make the transition in an incremental way.

Happy holidays folks!

Rails and Merb Merge

Today is a fairly momentous day in the history of Ruby web frameworks. You will probably find the news I’m about to share with you fairly shocking, but I will attempt to explain the situation.

Before talking tech, and even going into the details of the announcement, I want to assure everyone that the incredible members of the thriving Merb community are top priority, and that this could not have been possible without every one of them.

Merb is an open, ever-changing project, and some its best ideas have come from not-core regular-Joe community members. It’s gotten where it has because of the community, and the community will get us even further in the future. Your ideas, feedback and even complaints will be 100% welcome in the future, just as they have been in the past. I believe in the tremendous value an open community and just generally open attitude bring to the table, and am counting on those things to continue ushering in the future of Ruby.

On to the news: beginning today, the Merb team will be working with the Rails core team on a joint project. The plan is to merge in the things that made Merb different. This will make it possible to use Rails 3 for the same sorts of use-cases that were compelling for Merb users. Effectively, Merb 2 is Rails 3.

What does that mean exactly?

  • Rails will become more modular, starting with a rails-core, and including the ability to opt in or out of specific components. We will focus on reducing coupling across Rails, and making it possible to replace parts of Rails without disturbing other parts. This is exactly what Merb means when it touts “modularity”.
  • We will port all of our performance improvements into Rails. This includes architectural decisions that are big performance wins. This project will also include the creation of one or more benchmarking applications, so we can more clearly see what optimizations have real-world impact.
  • As of Rails 3, Rails will have a defined public API with a test suite for that API. This was one of the major differentiators of Merb. This will allow users and plugin developers to have a clearer, more stable API to build against. It should also significantly reduce plugin breakage from release to release.
  • Rails will be retrofitted to make it easy to start with a “core” version of Rails (like Merb’s current core generator), that starts with all modules out, and makes it easy to select just the parts that are important for your app. Of course, Rails will still ship with the “stack” version as the default (just as Merb does since 1.0), but the goal is to make it easy to do with Rails what people do with Merb today.
  • Rails will be modified to more easily support DataMapper or Sequel as first-class ORMs. While ActiveRecord will ship as the default ORM, the plan is to make it drop-dead easy to drop in other ORMs without feature degradation (to the extent possible, of course).
  • Rails will continue their recent embrace of Rack, which is a really exciting development in the Ruby community that Merb got in on early and which we believe will improve the state of modular, sharable logic between applications.
  • In general, we will take a look at features in Merb that are not in Rails (the most obvious example is the more robust router) and find a way to bring them into Rails.

How will we do this?

The plan is to start working on Rails immediately, and to continue fixing bugs and resolving other major issues in Merb in the interim. We will also release versions of Merb specifically designed to help ease the transition to Rails 3.

In particular, we will do Merb releases with deprecation notices and other transitional mechanisms to assist developers in tracking down the changes that will come between Merb 1.x and Rails 3. Expect a number of interim releases that get incrementally closer to Rails 3, and expect parts of Merb (most notably the helpers) to be ported to run on Rails 3 in order to further reduce friction.

To be perfectly clear: we are not abandoning the Merb project. There are many production applications running on Merb that are relying on both timely bug fixes and a clear path to the future. If you’re using Merb today, continue using Merb. If you’re considering using Merb for a project because it works better for your needs, use Merb. You will not be left in the cold and we’re going to do everything to make sure that your applications don’t get stuck in the past.

If you’ve already learned Merb, we will be working hard to make sure that you can parlay that knowledge into Rails 3. At Engine Yard, we fully intend to continue using Merb for our internal apps until Rails 3 is out, but we will be using those (non-trivial) applications to be sure the experience is smooth for everyone. There will be no huge jumps and you will not need to rewrite your application from scratch.

Why!?

As you have probably gathered from the above, there aren’t any clear points that the Merb and Rails team disagree on anymore. Merb has been around for roughly two years now, and we’ve proved out our ideas by use in real-world applications (like Yellow Pages, SproutCore, Powerset, Defensio, etc.). Given this philosophical convergence, it just didn’t seem like there was much to gain by continuing to duplicate effort and spend time and energy fighting each other.

I think it’s important to acknowledge the Merb community for building something super-awesome. I really hope that we’ll all stay in this together, help each other in the coming months and in the transition to Rails 3.

Rails will be putting together a new evangelism team, which will include Matt Aimonetti (Merb core team member and evangelist) and a few other people doing Rails evangelism work. This group will be responsible for, among other things, helping the community get where we’re going. Their job will be to listen to you.

This seems crazy! Has this ever happened before?

Interestingly, yes. A very similar situation confronted the Struts developers several years back. They had built a very popular framework in Struts, but a very active group of developers were really advancing the same ideas in interesting ways in a framework called Webwork. The Webwork developers saw their project as “Struts done right”, just as we believe we’ve improved on the implementation of Rails.

Eventually, the Struts guys and the Webwork guys came to the conclusion that they would be stronger together then apart, and merged the projects. What became Struts 2 was effectively also Webwork 2, but the two communities got together to build something better.

I believe that this merger will get the ideas that gave Merb momentum into the hands of many more people, and that really serves us all. I’m looking forward to the future.

Postscript

  • One of the most personally gratifying parts of working on Merb has been working with alternative Ruby implementations to make Merb run well on them. Fairly early, Merb was running faster on JRuby than MRI, and I’m looking forward to bringing a lot of that insight to Rails, and helping to make Rails to be solidly competitive on the performance front with frameworks in “faster” languages.
  • The Merb Training is very much still on! We’ve spoken with the majority of the attendees, as well as with our partners at Integrum Technologies and they’re all really excited about the merger. That makes this training basically the first training focusing on the future features of Rails 3, and a great way to get ahead of the game. There are still a few seats open, so sign up now if you’d like to join us!

Stop Watching Sophie’s Choice (And Get Some Work Done)

I woke up this morning to Kirin Dave’s cantankerous rant about how Ruby’s going down the tubes. The whole post was a giant whine-fest, with the exception of the beginning, where he heaps faux past praise on Ruby to justify his complaints.

The entire post read kind of like Joe Lieberman supporting McCain: “I used to be a Democrat, but now I think Obama’s in love with terrorists.” Dave didn’t even really attempt to be even-handed in his critique (if I can even call it a critique); he just goes after the Ruby language, interpreter, and community with the full force of his giant… rhetoric.

Let’s take a look at Dave’s claims:

Claim 1. Ruby’s interpreter is so outdated, it’s impossible to write code without it leaking memory. As evidence, he points to a case encountered by Tom Preston-Warner in his God monitor library. What Dave fails to point out is that despite his claims that Ruby 1.9 is basically useless, the last post in the thread he references informs the readers that the bug is fixed in Ruby 1.9.

He also doesn’t inform the reader that many large web applications, including YellowPages.com, scribd.com, and hulu.com, not exactly tiny web sites, are running on infrastructures that depend on this “outdated” interpreter. While I’m not going to claim that the Ruby interpreter is all rainbows, it’s hardly the problem that Dave claims it is.

Dave also dismisses JRuby with a wave of his hand, using the argument that it’s not useful for web applications because it can’t be used for scripting (which Ruby is frequently used for). This is a classic example of a red herring fallacy. The existing “terrible” Ruby interpreter works just fine for scripting, where the supposed memory leaks aren’t an issue. As a result, dismissing JRuby, which solves all of his other deployment concerns, is just pure malarky.

Claim 2. Ruby is the slowest thing imaginable. I’ve tackled this argument with some vigor before, but suffice it to say that real-life Ruby applications must be compared against real-life PHP or Django applications, and they perform quite well. Even Rails, not exactly the fastest Ruby web framework, beats out CodeIgniter in Hello World benchmarks, and is dead even with CodeIgniter on more robust benchmarks. Comparing it with CakePHP, which is a closer feature-for-feature comparison, Rails completely destroys Cake in all benchmarks.

Merb, which takes more effort to avoid being slow, does significantly better than Rails, and beats CodeIgniter on hello world benches by around 5x. The reason for this is that despite Dave’s claims that the Ruby community is stagnant, some of the worst speed offenders have been handled by native extensions that provide speedups for the community without the community having to drop into C all the time. The most recent example: ThirdBase, a library to make Ruby’s date facilities an order of magnitude faster than the built-in Date class.

Because the Ruby interpreter has a fairly good C API, developers without commit access have been able to improve the speed of everything from web servers (who uses the stdlib’s webrick!?) to XML parsing (why use REXML when you can use Nokogiri and gain orders of magnitude in speed). Real-life applications are simply not as slow as the benchmarks (which even Dave admits are not dispositive) would seem to imply. Real apps simply spend more time IO-waiting on things like databases or grabbing strings out of cache than in calculating the fibonacci sequence.

Claim 3. Rubinius didn’t happen and JRuby… these are not the droids you’re looking for… Rubinius didn’t happen. Somehow, in all his bloviating, Dave didn’t really address JRuby at all. Let’s take a look at his hand-wave real quick:

jRuby is great, but I kind of liked the illusion of a small memory footprint–a lot of Ruby use is in scripting and starting up a “java -server” instance is not really desirable for that. 

So let’s take a look: JRuby has too large a memory footprint. Actually, not so much. When you take into consideration that you only need a single JVM instead of multiple processes, JRuby starts looking very competitive. In fact, when you start factoring in the impact of real threads on IO-wait, JRuby starts killing MRI.

I already addressed the other claim: Ruby is used in scripting and JRuby is thus unsuitable for it. For those cases, MRI works just fine. Even if we were to grant Dave his claim of unbeatable memory problems, scripts can handle a slow 200k leak every 10,000 seconds (the big gotcha memory leak). As a result of all the effort around alternative implementations (and spearheaded by Rubinius), JRuby and MRI are remarkably compatible. Developing on MRI (for fast startup times) and deploying on JRuby (for a rock-solid memory footprint and great concurrency handling) is viable by design.

In essence “There are no good choices for a Ruby interpreter” is straight-up not true.

Claim 4. Ruby’s future is… yawn. Dave takes a look at the new features in Ruby, and based on the future of the Ruby language, decides that Ruby itself is booooring. He makes one tiny mistake when he claims that better FFI was only in alternative implementations: the aforementioned C API made it possible for wmeissner of the JRuby team to roll out a 100% compatible version of Rubinius’ FFI (which they have now adopted) for Ruby 1.8.

Probably the most misleading part of this claim is ignoring all the work being done on Ruby libraries. I can be competitive with Rails from time to time, but even I’m not going to claim that Rails is no longer producing good work. And Merb is fairly well acknowledged for innovation around these parts. Dave says:

And what do I have to look forward to in the Ruby world? Essentially the same promise that I heard in late 2005, “Christmas will bring a faster ruby.” 

This assumes that the only interesting things happening around a library are related to its core interpreter or the core language. While Ruby 1.9 does see some improvements to the complaints that he raised, and JRuby flat-our resolves them, Dave’s fundamental mistake is here. By assuming that the only way the community around a language can get better is by improving its core, he is ignoring reality: there are tons of exciting developments around Ruby that are being pushed by its users.

Dave closes by saying:

People will leave trying to stake out The Next Big Thing™, bleeding the Ruby talent pool dry and making it even less capable of recovering from these bad fortunes.

I predict the opposite. The Ruby talent pool will continue to grow, and we will continue to produce new, innovative, and exciting projects. JRuby will continue to improve and Rubinius will be released, sparking a new Ruby speed war that will lead to the faster Ruby that so dominates Dave’s dissatisfaction. And the Ruby community will continue to learn from other languages, like Python and Erlang. It’s only going to get better from here.

MythBusting — We Agree! Ruby is Awesome!

With all the disagreement over the past few days, it might seem like Merb and Rails are worlds apart. But David’s latest post demonstrates what I’ve been saying all along: we’re more alike than different.

In the latest installment, he takes on the myth that “Rails is hard because of Ruby”. Effectively, a bunch of people are comfortable in their language of choice (PHP, Java, Perl) and prefer to switch over to an MVC clone in their language of choice than learn the big bad scary Ruby.

As David succinctly argued, Ruby is just not that hard a language to learn. It’s better organized than PHP (with its absurdly large global namespace) and less ceremonious than Java (no IStrategyBuilderFactories here).

And what’s fantastic about Ruby is how quickly and simply new programmers are exposed to advanced concepts like lambdas. Because iteration is accomplished in Ruby almost exclusively with blocks, it’s near impossible to spend even a day in Ruby without learning what blocks are. Spend a little more time with Ruby, and the power of the closures that come along with lambdas becomes obvious. All without the need for an extensive study of the CS benefits of the construct.

Ruby is so easy, in fact, that the slim Learn to Program tome is written using Ruby as a base. It’s so straight forward that my wife, who’s hardly a programmer, was writing a program to count down “99 bottles of beer on the wall” within a few hours of picking up the book (and was even embellishing the program with a few interactive touches :-D )

Finally, implicit in the “I don’t want Rails because it’s Ruby” claim is the oft-heard myth that Ruby is slow. As I explained in my keynote at MerbCamp, Ruby does very well even compared with raw PHP. But when you compare Ruby frameworks (even Rails) against CakePHP, Rails destroys the competition. And Merb does even better. That’s because PHP’s fundamental architecture does not play very well with large frameworks, while Ruby deployment options can manage a fairly large framework with very small runtime impact.

What I showed at MerbCamp was that it was possible to squeeze up to 4,000 requests per second through the Merb stack, or around 1,500 requests per second when going through a controller and template, but that frameworks like CakePHP got only about 100 requests per second, even with a code accelerator.

Bottom line: don’t let anyone tell you that Ruby web applications need be slow. The language itself is certainly slow, but I don’t see a ton of Fibonacci web applications being built, so the real question is about where the bottlenecks are, and Ruby acquits itself very well.

MythBusting — Rails is not a monolith

Continuing his interesting train of thought, DHH posted on his blog yesterday that Rails is, in fact, not monolithic. In reality, he says, it is quite modular. In the post, he finally fully articulated the rationale behind the excessive use of alias_method_chain in Rails, which he says is to keep the code even more “modular”.

Let’s take a look at some of the claims:

Rails is not actually that large

They count all lines including comments and whitespace in Ruby files, thus punishing well-documented and formatted code

As I said yesterday, I did not actually include comments and whitespace. I specifically provided the command that I used, which removed whitespace-only lines and comment-only lines. In fact, the comment by bitsweat (a member of the Rails Core team) which started this back-and-forth erroneously included comments in Merb’s count, when merb-core has about 1 line of comment per line of code. This is why Jeremy incorrectly thought that merb-core was over 15,000 lines of code.

They count tests, thus punishing well-tested code

Neither Jeremy nor I counted tests. I’m not sure which LOC-count he’s referring to.

They count bundled dependencies, thus punishing dependency-free code

I did in fact make this mistake, but I disagree with the assertion that bundling dependencies makes code more “modular”. For example, bundling xml-simple in Rails causes a conflict with gems that require a newer version of xml-simple than the one bundled with Rails. Conversely, merb-haml has a haml dependency, which means that users can use newer versions of Haml that are released between Merb releases.

This statement by David actually teases out a fundamental difference of opinion between Rails and Merb. In effect, Rails prefers to bundle everything to reduce dependencies, while Merb prefers to use the existing Rubygems system so that applications can use different versions of the “bundled” dependencies.

Additionally, we don’t want to be maintaining bitrotted versions of things like tmail. We’d prefer to rely on gems created by experts in their niche who can maintain, and more importantly, fix bugs in, their code. We fully admit that our approach pushes the limits of Rubygems, but that has forced us to work with rubygems to improve a core piece of Ruby infrastructure.

Rails is actually pretty modular

The arguments made here almost defy reason, but let’s go through some of them.

First, Rails can include almost as much or as little of the six major pieces as you prefer.

Absolutely. I’ve referred to this in the past as the Lego vs. Duplo philosophy. Rails has added in a feature to allow you to remove entire blocks of functionality, but isn’t built on an architecture that lets you granularly opt-out. Granular opt-out allows you to reuse foundational code without buying into the full set of opinionated defaults.

One example of this is our auth system, which allows you to reuse the base auth code, which simply allows you to define strategies inside of a framework, even if you don’t want to use our built-in strategies or login views.

Granular opt-out builds a community around chunks of code that can be swapped in; having an auth core makes it easy to share small, simple authentication strategies between users of Merb.

The next part is the part that makes me incredulous. According to David, because Rails is spread across many files, it is “modular”. He describes how you would go about granularly removing certain features:

All these optional parts can actually very easily be turned off as well, if you so please. If you look at actionpack/lib/action_controller.rb, you’ll see something like the following:

ActionController::Base.class_eval do

  include ActionController::Flash
  include ActionController::Benchmarking
  include ActionController::Caching
  ...

This is where all the optional bits are being mixed into Action Pack. But they didn’t need to be. If you really wanted to, you could just edit this 1 file and remove the optional bits you didn’t need and you’d have some 3,500 lines of optional goodies to pick from.

Read that carefully. If you want to opt out of certain parts of Rails, you need to:

  1. Read the source, and figure out which files contain the features you want
  2. Figure out where the modules in question have been mixed in
  3. Fork Rails
  4. Modify your own personal version of Rails to remove modules that have been mixed in
  5. Never upgrade Rails again (or upgrade Rails and hope they haven’t made any changes to the part of the file you’ve modified)

Fundamentally, Rails’ “modular” architecture is fine for core committers, but it’s not particularly useful for consumers of the framework. Which makes sense, since 37 Signals sees Rails primarily as a library that they use to write their apps. So thinking about modularity in terms of the code of the framework itself makes perfect sense from that perspective.

On the other hand, Merb looks at modularity from the perspective of the developer using the framework.

alias_method_chain makes Rails modular

As I’ve said many times before, I don’t like alias_method_chain. But Rails’ philosophy around it is a perfect example of its problems. Superficially, it seems like it divides up responsibilities neatly into their own modules. And again, this is perfectly true from the perspective of developers working on the framework itself.

For consumers of the framework, it is simply maddening. Here’s a code snippet David provided in his post:

module Benchmarking
  def self.included(base)
    base.extend(ClassMethods)

    base.class_eval do
      alias_method_chain :perform_action, :benchmark
      alias_method_chain :render, :benchmark
    end
  end

This is an extremely common idiom in Rails. Unfortunately, it makes Rails methods extremely opaque. It is nearly impossible, without reading through a dozen files and putting together the puzzle, to figure out what the perform_action method actually does. This is evident in a Rails stack trace, which includes close to 10 frames for different parts of perform_action.

And this alias_method_chain isn’t even greppable. Effectively, it’s up to the user to divine that including a module modifies methods in the class, since it’s done via metaprogramming in an included hook. Again, this all makes perfect sense for developers on the Rails framework. But for consumers of the framework, it leads to many frustrating days trying to track down all the pieces of a particular method.

But why bother?

Again, this question makes perfect sense coming from the perspective of using Rails as an internal library. Don’t need something, like pagination? Simply move it out of the core framework. From the perspective of trying to develop something that can be used for many different purposes, in many different situations, we have different priorities.

Merb has already been used as the base for SproutCore, which generates static HTML out of a Merb base. It has been used for sinatra-like web services. And it’s being used by several large companies who don’t want to use DataMapper (and are using Sequel instead).

As people use Merb for more kinds of things, decisions like hardcoding features of Merb to a particular ORM (like Rails has done), requiring a certain directory structure, or bundling in dependencies become much harder to justify. People using Merb actually do use other gems that conflict with Rails bundled dependencies. People using Merb want to be able to upgrade their versions of Haml, ParseTree, or MailFactory without expensive surgery to the framework itself.

For the moment, these differences are the reason that Rails will continue to dominate amongst developers seeking to build apps similar in scope to apps built by 37Signals. I suspect that Merb will pick up steam amongst developers looking to build innovative apps leveraging the latest and greatest Ruby techniques and libraries.

I, for one, spent several years building large Rails applications, and am happy to leave the “just keep your own frozen, patched copy of Rails” philosophy behind.

UPDATE: Just to be clear, I know that Rails tries to use more recent versions of gems if they’re available on the system. However, that is a very naïve dependency approach, that in our experience, produced issues. It’s much better to be warned that you have dependency conflicts up front than to have them manifest in running production code.

Conferences, etc.

As some of you may know, I was at The Ajax Experience conference last week, along with the one-day mini-conference, jQueryCamp.

This was my third The Ajax Experience, and there were a lot of great people to see. Even more exciting though, was jQueryCamp, despite forgetting my t-shirt in Boston.. The accommodations were especially wonderful ;) (that’s code for… I crashed on John‘s couch. Thanks John!)..

I met most of my fellow core team members, which I’d really been looking forward to, along with a slew of jQuery geeks and gurus. Karl, Jörn, Bradley, Marc, Paul, Rey, Richard — just to name a few. I hope to post more about jQuery camp — the talks, etc. — when things get a bit calmer… if ever they do.

(As a side bonus, there were a few Merb folks around — what more could I ask for?!)

There was a pleasantly enthusiastic response to jQuery in Action; lots of people with comments, reading the Early Access version, and lots of people looking forward to it. I’m feeling good about it!

I’ll be at RubyConf in NC this weekend. There are a lot of people I’m looking forward to meeting there, and talks of some Merb hacking.

There’s nothing like putting faces to the IRC folks I talk to every day, the blogs I read, etc. There are also some old friends I’m looking forward to seeing again. Stephen Bristol, ReinH, Evan Phoenix, Jeremy McAnally, Ezra Zygmuntowicz, Dr Nic Williams, and so on. Leah (the wife) will be joining me at RubyConf too, so that’s always a bonus — and I know there are lots of people she’s looking forward to seeing too.

Do drop me a line, or comment, if you’ll be around. Would love to see everyone. Safe travels!

Archives

Categories

Meta