Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Merb’ Category

Threads (in Ruby): Enough Already

For a while now, the Ruby community has become enamored in the latest new hotness, evented programming and Node.js. It’s gone so far that I’ve heard a number of prominent Rubyists saying that JavaScript and Node.js are the only sane way to handle a number of concurrent users.

I should start by saying that I personally love writing evented JavaScript in the browser, and have been giving talks (for years) about using evented JavaScript to sanely organize client-side code. I think that for the browser environment, events are where it’s at. Further, I don’t have any major problem with Node.js or other ways of writing server-side evented code. For instance, if I needed to write a chat server, I would almost certainly write it using Node.js or EventMachine.

However, I’m pretty tired of hearing that threads (and especially Ruby threads) are completely useless, and if you don’t use evented code, you may as well be using a single process per concurrent user. To be fair, this has somewhat been the party line of the Rails team years ago, but Rails has been threadsafe since Rails 2.2, and Rails users have been taking advantage of it for some time.

Before I start, I should be clear that this post is talking about requests that spent a non-tiny amount of their time utilizing the CPU (normal web requests), even if they do spend a fair amount of time in blocking operations (disk IO, database). I am decidedly not talking about situations, like chat servers where requests sit idle for huge amounts of time with tiny amounts of intermittent CPU usage.

Threads and IO Blocking

I’ve heard a common misperception that Ruby inherently “blocks” when doing disk IO or making database queries. In reality, Ruby switches to another thread whenever it needs to block for IO. In other words, if a thread needs to wait, but isn’t using any CPU, Ruby’s built-in methods allow another waiting thread to use the CPU while the original thread waits.

If every one of your web requests uses the CPU for 30% of the time, and waits for IO for the rest of the time, you should be able to serve three requests in parallel, coming close to maxing out your CPU.

Here’s a couple of diagrams. The first shows how people imagine requests work in Ruby, even in threadsafe mode. The second is how an optimal Ruby environment will actually operate. This example is extremely simplified, showing only a few parts of the request, and assuming equal time spent in areas that are not necessarily equal.


Untitled.001.png


Untitled.002.png


I should be clear that Ruby 1.8 spends too much time context-switching between its green threads. However, if you’re not switching between threads extremely often, even Ruby 1.8′s overhead will amount to a small fraction of the total time needed to serve a request. A lot of the threading benchmarks you’ll see are testing pathological cases involve huge amounts of threads, not very similar to the profile of a web server.

(if you’re thinking that there are caveats to my “optimal Ruby environment”, keep reading)

“Threads are just HARD”

Another common gripe that pushes people to evented programming is that working with threads is just too hard. Working hard to avoid sharing state and using locks where necessary is just too tricky for the average web developer, the argument goes.

I agree with this argument in the general case. Web development, on the other hand, has an extremely clean concurrency primitive: the request. In a threadsafe Rails application, the framework manages threads and uses an environment hash (one per request) to store state. When you work inside a Rails controller, you’re working inside an object that is inherently unshared. When you instantiate a new instance of an ActiveRecord model inside the controller, it is rooted to that controller, and is therefore not used between live threads.

It is, of course, possible to use global state, but the vast majority of normal, day-to-day Rails programming (and for that matter, programming in any web framework in any language with a request model) is inherently threadsafe. This means that Ruby will transparently handle switching back and forth between active requests when you do something blocking (file, database, or memcache access, for instance), and you don’t need to personally manage the problems the arise when doing concurrent programming.

This is significantly less true about applications, like chat servers, that keep open a huge number of requests. In those cases, a lot of the application logic happens outside the individual request, so you need to personally manage shared state.

Historical Ruby Issues

What I’ve been talking about so far is how stock Ruby ought to operate. Unfortunately, a group of things have historically conspired to make Ruby’s concurrency story look much worse than it actually ought to be.

Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency. Also, even in threadsafe mode (when not using the built-in Rails support) Mongrel spins up a new thread for every request, not exactly optimal.

Second, the most common database driver, mysql is a very poorly behaved C extension. While built-in I/O (file or pipe access) correctly alerts Ruby to switch to another thread when it hits a blocking region, other C extensions don’t always do so. For safety, Ruby does not allow a context switch while in C code unless the C code explicitly tells the VM that it’s ok to do so.

All of the Data Objects drivers, which we built for DataMapper, correctly cause a context switch when entering a blocking area of their C code. The mysqlplus gem, released in March 2009, was designed to be a drop-in replacement for the mysql gem, but fix this problem. The new mysql2 gem, written by Brian Lopez, is a drop-in replacement for the old gem, also correctly handles encodings in Ruby 1.9, and is the new default MySQL driver in Rails.

Because Rails shipped with the (broken) mysql gem by default, even people running on working web servers (i.e. not mongrel) in threadsafe mode would have seen a large amount of their potential concurrency eaten away because their database driver wasn’t alerting Ruby that concurrent operation was possible. With mysql2 as the default, people should see real gains on threadsafe Rails applications.

A lot of people talk about the GIL (global interpreter lock) in Ruby 1.9 as a death knell for concurrency. For the uninitiated, the GIL disallows multiple CPU cores from running Ruby code simultaneously. That does mean that you’ll need one Ruby process (or thereabouts) per CPU core, but it also means that if your multithreaded code is running correctly, you should need only one process per CPU core. I’ve heard tales of six or more processes per core. Since it’s possible to fully utilize a CPU with a single process (even in Ruby 1.8), these applications could get a 4-6x improvement in RAM usage (depending on context-switching overhead) by switching to threadsafe mode and using modern drivers for blocking operations.

JRuby, Ruby 1.9 and Rubinius, and the Future

Finally, JRuby already runs without a global interpreter lock, allowing your code to run in true parallel, and to fully utilize all available CPUs with a single JRuby process. A future version of Rubinius will likely ship without a GIL (the work has already begun), also opening the door to utilizing all CPUs with a single Ruby process.

And all modern Ruby VMs that run Rails (Ruby 1.9′s YARV, Rubinius, and JRuby) use native threads, eliminating the annoying tax that you need to pay for using threads in Ruby 1.8. Again, though, since that tax is small relative to the time for your requests, you’d likely see a non-trivial improvement in latency in applications that spend time in the database layer.

To be honest, a big part of the reason for the poor practical concurrency story in Ruby has been that the Rails project didn’t take it seriously, which it difficult to get traction for efforts to fix a part of the problem (like the mysql driver).

We took concurrency very seriously in the Merb project, leading to the development of proper database drivers for DataMapper (Merb’s ORM), and a top-to-bottom understanding of parts of the stack that could run in parallel (even on Ruby 1.8), but which weren’t. Rails 3 doesn’t bring anything new to the threadsafety of Rails itself (Rails 2.3 was threadsafe too), but by making the mysql2 driver the default, we have eliminated a large barrier to Rails applications performing well in threadsafe mode without any additional research.

UPDATE: It’s worth pointing to Charlie Nutter’s 2008 threadsafety post, where he talked about how he expected threadsafe Rails would impact the landscape. Unfortunately, the blocking MySQL driver held back some of the promise of the improvement for the vast majority of Rails users.

Merb Vulnerability Fix (1.0.12)

Over the weekend, it was discovered that the json_pure gem is subject to a DoS attack when parsing specially formed JSON objects. This vulnerability does not affect the json gem, which uses a C extension, or ActiveSupport::JSON, which is used in Rails.

By default, Merb uses the json gem, which is not vulnerable, but falls back to json_pure if json is not available. As a result, if you have json_pure but not json on your system, you may be vulnerable. Additionally, Ruby 1.9.1 (but not Ruby 1.9 trunk) ships with json_pure, which remains vulnerable.

The easiest way to immunize yourself from this problem, no matter what Ruby version you are on, is to upgrade to the latest version of json_pure, which resolves the vulnerability. Additionally, Merb 1.0.12 has been released, which monkey-patches json_pure to remove the vulnerability, but prints a warning encouraging you to upgrade to the latest. Merb 1.0.12 only adds this patch on top of 1.0.11, so it should be perfectly safe to upgrade.

Today’s Dispatch: Weaning ActionView off of content negotiation

I’ve been on “vacation” for the past few days and haven’t had a ton of time to work on continuing refactoring, but I did manage to make a bit more progress, and I figured I’d share.

In Rails 2.2, ActionView had a fair number of content negotiation responsibilities. In particular, ActionView::Base had a method called template_format that looked like this:

def template_format
  if defined? @template_format
    @template_format
  elsif controller && controller.respond_to?(:request)
    @template_format = controller.request.format.to_sym
  else
    @template_format = :html
  end
end

Effectively, ActionView had a small content-negotiation responsibility, that effectively entailed grabbing the first acceptable format and using that. Thankfully, this particular piece of code was usually supplanted by Rails’ respond_to code, which performed more proper content-negotiation, and set the template format directly: @response.template.template_format = mime_type.to_sym.

A number of other places, including the JavaScript helpers, also set the template format directly, which meant that for the most part, the template_format method was just a placeholder that was set by other parts of the system. However, there were more than a few cases where the default template_format method was getting used. Instead of having ActionView call back into the request object, I wanted to modify ActionView’s initialization so that ActionController could feed it the list of acceptable formats directly.

With some work, I was able to make the modification, weaning ActionView off of content negotiation. Interestingly, this is related to my overall refactoring, which currently uses a four-element tuple to represent templates (path, acceptable extensions, prefix, and a partial flag). Now that ActionView uses a list of formats internally, it can easily pass them into find_by_parts, which handles figuring out which template to use based on the ordered list of extensions.

Just a note: find_by_parts is almost certainly a temporary creation to facilitate refactoring. However, for the purposes of refactoring, it is a convenient way to make sure that all pieces of the puzzle are passing around compatible values. In the end, we will probably be passing around Template objects, and this intermediate refactoring step will help us get all of our ducks in a row so that we can get there.

A fun diff that sort of demonstrates where I’m going with this is:

   def _pick_partial_template(partial_path) #:nodoc:
-    if partial_path.include?('/')
-      path = File.join(File.dirname(partial_path), "_#{File.basename(partial_path)}")
-    elsif controller
-      path = "#{controller.class.controller_path}/_#{partial_path}"
-    else
-      path = "_#{partial_path}"
+    unless partial_path.include?('/')
+      prefix = (controller && controller.class.controller_path)
     end
-
-    self.view_paths.find_template(path, self.template_format)
+    view_paths.find_by_parts(partial_path, formats, prefix, true)
   end

As you can see, by normalizing all path lookup to the four-element tuple, which includes merging path prefixes and appending “_” to partials, I was able to remove almost all of the code of this method, now requiring only that the controller_path is specified as a prefix if no existing prefix is supplied as part of the path (that’s so render :partial => “foo” inside of the TestController looks for “test/_foo”).

Again, I probably won’t be able to do a ton of work over the next few days, but I’ll blog when I can. Happy holidays folks!

Rails Refactor Update (and Merb 1.0.7.1)

Over the past few days, I’ve been working on refactoring ActionController and ActionView to clean up the interactions between them. You can follow along on my github fork. Some principles I’m following:

  • ActionController and ActionView should work well standalone
  • All request-related information should be calculated in ActionController and passed through into ActionView
  • ActionView should be responsible for figuring out what path to render; ActionController should pass enough information for ActionView to figure it out.
  • Based on the previous, the information passed from ActionController to ActionView will likely be: template_name, list of acceptable extensions (i.e. [:html, :xml]), prefix (usually the controller name), and partial (a boolean indicating whether the template is a partial). This is not nailed down yet, but it has so far served well.
  • So far, I have unified render(:template), render(:file), and render(:action) to use this new conduit, and will be working on partials tomorrow. Partials are quite complicated so my current plan may have to change slightly when I tackle them.
  • Interesting info: ActionPack has a lot of methods that call each other (somewhat circularly), so it wasn’t really possible to just replace the existing conduit in a straight-forward manner. Instead, I created the new method (currently called find_by_parts, which finds a template based on the components I discussed above) and slowly (very slowly) moved existing callers of find_by_path over to use find_by_parts. Thankfully, the Rails test suite caught the initial errors I made, a huge saving grace of the entire effort.
  • A nice side-effect of moving to a single, clean API between AC and AV is that the final code should be easier to understand. Once I’m further along, I’ll post some details of how exactly the interaction works.

We also released Merb 1.0.7.1 today to fix a few issues that were outstanding as well as a 1.0.7 development mode regression. (In general, we will be using the x.x.x.x moniker for midweek releases and x.x.x for weekly releases). The issues that were fixed:

  • Templates should reload again in development mode
  • An issue in the bundler where gems in your local repository were getting greedily installed is fixed
  • An issue that was preventing bin/thor merb:gem:install use is fixed
  • The error message when gems could not be found was slightly improved

To upgrade:

  • gem install merb (make sure merb-gen 1.0.7.1 is installed)
  • in your app, rm -rf tasks/merb.thor
  • in your app, run merb-gen thor
  • you should receive a prompt asking you to override bin/common.rb. Accept the prompt.
  • you’re done

Happy holidays folks!

Dispatch from the Front Lines

Hey all,

It’s been a very busy few days, but I’m glad to say that the work on the Rails/Merb merge is going quite smoothly. Some things that have already happened:

  • DHH has posted an outline of our plans to bring display/provides into Rails: http://tinyurl.com/providesinrails. The new API resolves some lingering issues I personally had with the provides/display version, and is still undergoing a bit of change.
  • Michael Klishin (antares) has begun work to merge Extlib into ActiveSupport. There are already some obvious incompatibilities, so we will be doing both cleanup on the differences and selecting the fastest implementations. He has also begun porting LazyArray, the underpinnings of the DataMapper “kicker” technique, to ActiveSupport. I should note that Michael was instrumental in merging Merb and DataMapper’s extensions into Extlib in the first place, so this should work out quite well.
  • As DHH mentioned in his post, some work is also being done to work out a better API for the Merb router. While our API works fine, we’re working together on an even better API for Rails3. One of the great things about working with the Rails team is being able to work with very talented API designers on some hard problems. To be clear, we will still support both Rails2′s API and Merb1′s API, because the Merb router, which is serving as the base for this work, decouples the code generation part from the DSL (so any DSL can be used to generate the structures that the router than converts into its compiled matching method).
  • We have started discussing some other issues that could potentially break back-compat in Rails (like Merb’s BlockAwareEnhancer for helpers that take blocks), and I’m happy to report that the conversations have been very productive. Being able to revisit issues that both groups addressed suboptimally in the past with a larger group is proving to be quite enlightening.
  • I have personally begun work to unify the code in Rails that does content negotiation to always go through a single path. You can follow along at http://github.com/wycats/rails/commits/content_negotiation. I will be rebasing this remote branch against current rails master frequently (to keep it in sync with rails head) so you may need to reclone from time to time.

On the Merb front, we will be releasing 1.0.7 this weekend. Expect the first transitional release (Merb 1.1) in the next month or two. It will probably involve porting over basic configuration to Rails’ style (Merb’s Kernel#* style is pretty icky) and probably starting to create a shim for some of the more obvious differences (before/before_filter comes to mind). It’ll be the first of several releases specifically designed to help you make the transition in an incremental way.

Happy holidays folks!

Rails and Merb Merge

Today is a fairly momentous day in the history of Ruby web frameworks. You will probably find the news I’m about to share with you fairly shocking, but I will attempt to explain the situation.

Before talking tech, and even going into the details of the announcement, I want to assure everyone that the incredible members of the thriving Merb community are top priority, and that this could not have been possible without every one of them.

Merb is an open, ever-changing project, and some its best ideas have come from not-core regular-Joe community members. It’s gotten where it has because of the community, and the community will get us even further in the future. Your ideas, feedback and even complaints will be 100% welcome in the future, just as they have been in the past. I believe in the tremendous value an open community and just generally open attitude bring to the table, and am counting on those things to continue ushering in the future of Ruby.

On to the news: beginning today, the Merb team will be working with the Rails core team on a joint project. The plan is to merge in the things that made Merb different. This will make it possible to use Rails 3 for the same sorts of use-cases that were compelling for Merb users. Effectively, Merb 2 is Rails 3.

What does that mean exactly?

  • Rails will become more modular, starting with a rails-core, and including the ability to opt in or out of specific components. We will focus on reducing coupling across Rails, and making it possible to replace parts of Rails without disturbing other parts. This is exactly what Merb means when it touts “modularity”.
  • We will port all of our performance improvements into Rails. This includes architectural decisions that are big performance wins. This project will also include the creation of one or more benchmarking applications, so we can more clearly see what optimizations have real-world impact.
  • As of Rails 3, Rails will have a defined public API with a test suite for that API. This was one of the major differentiators of Merb. This will allow users and plugin developers to have a clearer, more stable API to build against. It should also significantly reduce plugin breakage from release to release.
  • Rails will be retrofitted to make it easy to start with a “core” version of Rails (like Merb’s current core generator), that starts with all modules out, and makes it easy to select just the parts that are important for your app. Of course, Rails will still ship with the “stack” version as the default (just as Merb does since 1.0), but the goal is to make it easy to do with Rails what people do with Merb today.
  • Rails will be modified to more easily support DataMapper or Sequel as first-class ORMs. While ActiveRecord will ship as the default ORM, the plan is to make it drop-dead easy to drop in other ORMs without feature degradation (to the extent possible, of course).
  • Rails will continue their recent embrace of Rack, which is a really exciting development in the Ruby community that Merb got in on early and which we believe will improve the state of modular, sharable logic between applications.
  • In general, we will take a look at features in Merb that are not in Rails (the most obvious example is the more robust router) and find a way to bring them into Rails.

How will we do this?

The plan is to start working on Rails immediately, and to continue fixing bugs and resolving other major issues in Merb in the interim. We will also release versions of Merb specifically designed to help ease the transition to Rails 3.

In particular, we will do Merb releases with deprecation notices and other transitional mechanisms to assist developers in tracking down the changes that will come between Merb 1.x and Rails 3. Expect a number of interim releases that get incrementally closer to Rails 3, and expect parts of Merb (most notably the helpers) to be ported to run on Rails 3 in order to further reduce friction.

To be perfectly clear: we are not abandoning the Merb project. There are many production applications running on Merb that are relying on both timely bug fixes and a clear path to the future. If you’re using Merb today, continue using Merb. If you’re considering using Merb for a project because it works better for your needs, use Merb. You will not be left in the cold and we’re going to do everything to make sure that your applications don’t get stuck in the past.

If you’ve already learned Merb, we will be working hard to make sure that you can parlay that knowledge into Rails 3. At Engine Yard, we fully intend to continue using Merb for our internal apps until Rails 3 is out, but we will be using those (non-trivial) applications to be sure the experience is smooth for everyone. There will be no huge jumps and you will not need to rewrite your application from scratch.

Why!?

As you have probably gathered from the above, there aren’t any clear points that the Merb and Rails team disagree on anymore. Merb has been around for roughly two years now, and we’ve proved out our ideas by use in real-world applications (like Yellow Pages, SproutCore, Powerset, Defensio, etc.). Given this philosophical convergence, it just didn’t seem like there was much to gain by continuing to duplicate effort and spend time and energy fighting each other.

I think it’s important to acknowledge the Merb community for building something super-awesome. I really hope that we’ll all stay in this together, help each other in the coming months and in the transition to Rails 3.

Rails will be putting together a new evangelism team, which will include Matt Aimonetti (Merb core team member and evangelist) and a few other people doing Rails evangelism work. This group will be responsible for, among other things, helping the community get where we’re going. Their job will be to listen to you.

This seems crazy! Has this ever happened before?

Interestingly, yes. A very similar situation confronted the Struts developers several years back. They had built a very popular framework in Struts, but a very active group of developers were really advancing the same ideas in interesting ways in a framework called Webwork. The Webwork developers saw their project as “Struts done right”, just as we believe we’ve improved on the implementation of Rails.

Eventually, the Struts guys and the Webwork guys came to the conclusion that they would be stronger together then apart, and merged the projects. What became Struts 2 was effectively also Webwork 2, but the two communities got together to build something better.

I believe that this merger will get the ideas that gave Merb momentum into the hands of many more people, and that really serves us all. I’m looking forward to the future.

Postscript

  • One of the most personally gratifying parts of working on Merb has been working with alternative Ruby implementations to make Merb run well on them. Fairly early, Merb was running faster on JRuby than MRI, and I’m looking forward to bringing a lot of that insight to Rails, and helping to make Rails to be solidly competitive on the performance front with frameworks in “faster” languages.
  • The Merb Training is very much still on! We’ve spoken with the majority of the attendees, as well as with our partners at Integrum Technologies and they’re all really excited about the merger. That makes this training basically the first training focusing on the future features of Rails 3, and a great way to get ahead of the game. There are still a few seats open, so sign up now if you’d like to join us!

Stop Watching Sophie’s Choice (And Get Some Work Done)

I woke up this morning to Kirin Dave’s cantankerous rant about how Ruby’s going down the tubes. The whole post was a giant whine-fest, with the exception of the beginning, where he heaps faux past praise on Ruby to justify his complaints.

The entire post read kind of like Joe Lieberman supporting McCain: “I used to be a Democrat, but now I think Obama’s in love with terrorists.” Dave didn’t even really attempt to be even-handed in his critique (if I can even call it a critique); he just goes after the Ruby language, interpreter, and community with the full force of his giant… rhetoric.

Let’s take a look at Dave’s claims:

Claim 1. Ruby’s interpreter is so outdated, it’s impossible to write code without it leaking memory. As evidence, he points to a case encountered by Tom Preston-Warner in his God monitor library. What Dave fails to point out is that despite his claims that Ruby 1.9 is basically useless, the last post in the thread he references informs the readers that the bug is fixed in Ruby 1.9.

He also doesn’t inform the reader that many large web applications, including YellowPages.com, scribd.com, and hulu.com, not exactly tiny web sites, are running on infrastructures that depend on this “outdated” interpreter. While I’m not going to claim that the Ruby interpreter is all rainbows, it’s hardly the problem that Dave claims it is.

Dave also dismisses JRuby with a wave of his hand, using the argument that it’s not useful for web applications because it can’t be used for scripting (which Ruby is frequently used for). This is a classic example of a red herring fallacy. The existing “terrible” Ruby interpreter works just fine for scripting, where the supposed memory leaks aren’t an issue. As a result, dismissing JRuby, which solves all of his other deployment concerns, is just pure malarky.

Claim 2. Ruby is the slowest thing imaginable. I’ve tackled this argument with some vigor before, but suffice it to say that real-life Ruby applications must be compared against real-life PHP or Django applications, and they perform quite well. Even Rails, not exactly the fastest Ruby web framework, beats out CodeIgniter in Hello World benchmarks, and is dead even with CodeIgniter on more robust benchmarks. Comparing it with CakePHP, which is a closer feature-for-feature comparison, Rails completely destroys Cake in all benchmarks.

Merb, which takes more effort to avoid being slow, does significantly better than Rails, and beats CodeIgniter on hello world benches by around 5x. The reason for this is that despite Dave’s claims that the Ruby community is stagnant, some of the worst speed offenders have been handled by native extensions that provide speedups for the community without the community having to drop into C all the time. The most recent example: ThirdBase, a library to make Ruby’s date facilities an order of magnitude faster than the built-in Date class.

Because the Ruby interpreter has a fairly good C API, developers without commit access have been able to improve the speed of everything from web servers (who uses the stdlib’s webrick!?) to XML parsing (why use REXML when you can use Nokogiri and gain orders of magnitude in speed). Real-life applications are simply not as slow as the benchmarks (which even Dave admits are not dispositive) would seem to imply. Real apps simply spend more time IO-waiting on things like databases or grabbing strings out of cache than in calculating the fibonacci sequence.

Claim 3. Rubinius didn’t happen and JRuby… these are not the droids you’re looking for… Rubinius didn’t happen. Somehow, in all his bloviating, Dave didn’t really address JRuby at all. Let’s take a look at his hand-wave real quick:

jRuby is great, but I kind of liked the illusion of a small memory footprint–a lot of Ruby use is in scripting and starting up a “java -server” instance is not really desirable for that. 

So let’s take a look: JRuby has too large a memory footprint. Actually, not so much. When you take into consideration that you only need a single JVM instead of multiple processes, JRuby starts looking very competitive. In fact, when you start factoring in the impact of real threads on IO-wait, JRuby starts killing MRI.

I already addressed the other claim: Ruby is used in scripting and JRuby is thus unsuitable for it. For those cases, MRI works just fine. Even if we were to grant Dave his claim of unbeatable memory problems, scripts can handle a slow 200k leak every 10,000 seconds (the big gotcha memory leak). As a result of all the effort around alternative implementations (and spearheaded by Rubinius), JRuby and MRI are remarkably compatible. Developing on MRI (for fast startup times) and deploying on JRuby (for a rock-solid memory footprint and great concurrency handling) is viable by design.

In essence “There are no good choices for a Ruby interpreter” is straight-up not true.

Claim 4. Ruby’s future is… yawn. Dave takes a look at the new features in Ruby, and based on the future of the Ruby language, decides that Ruby itself is booooring. He makes one tiny mistake when he claims that better FFI was only in alternative implementations: the aforementioned C API made it possible for wmeissner of the JRuby team to roll out a 100% compatible version of Rubinius’ FFI (which they have now adopted) for Ruby 1.8.

Probably the most misleading part of this claim is ignoring all the work being done on Ruby libraries. I can be competitive with Rails from time to time, but even I’m not going to claim that Rails is no longer producing good work. And Merb is fairly well acknowledged for innovation around these parts. Dave says:

And what do I have to look forward to in the Ruby world? Essentially the same promise that I heard in late 2005, “Christmas will bring a faster ruby.” 

This assumes that the only interesting things happening around a library are related to its core interpreter or the core language. While Ruby 1.9 does see some improvements to the complaints that he raised, and JRuby flat-our resolves them, Dave’s fundamental mistake is here. By assuming that the only way the community around a language can get better is by improving its core, he is ignoring reality: there are tons of exciting developments around Ruby that are being pushed by its users.

Dave closes by saying:

People will leave trying to stake out The Next Big Thing™, bleeding the Ruby talent pool dry and making it even less capable of recovering from these bad fortunes.

I predict the opposite. The Ruby talent pool will continue to grow, and we will continue to produce new, innovative, and exciting projects. JRuby will continue to improve and Rubinius will be released, sparking a new Ruby speed war that will lead to the faster Ruby that so dominates Dave’s dissatisfaction. And the Ruby community will continue to learn from other languages, like Python and Erlang. It’s only going to get better from here.

Merb 1.0.2

We’re releasing Merb 1.0.2 today, which is just a small number of patches that address some of the most urgent issues:

  • Some of the built-in rake tasks were deleting some gems that were in the specs as fixtures. This only affected people building Merb from the git sources, and is now resolved.
  • The Merb source code hardcoded in certain versions in multiple places. We have removed another such case, and created Merb::DM_VERSION to specify what version of DataMapper the merb stack shipped with. We now use that variable when generating the merb gemspec.
  • Merb was truncating the log file with each start. It now opens the file in append mode.
  • Many people reported that they accidentally overrode a core Merb::Controller method in their controllers, and received only an obscure warning. Merb now raises an error when you attempt to override a non-overridable method in your controller. If you want to override something anyway, call override! :your_method before doing so.
  • merb -i was silently failing for users who did not have webrat installed. We now print a warning and allow merb -i to load. Webrat is required for some advanced features that allow you to emulate the browser in the console.
  • An issue with numeric routes (ticket 1036) is now resolved.
  • An issue with cookies (ticket 1022) is now resolved.
In case people were wondering, we are working on 1.0.x in parallel with 1.1. The 1.0.x will continue to provide bugfixes for the 1.0 release, while 1.1 will add additional features in the march toward 2.0. You can get the nightlies of both the latest 1.0.x release and the 1.1 release at the edge.merbivore.com gem server.

Merb 1.0 Spec change

When we released 1.0, we also copied the existing 1.0 specs into a directory marked spec10 and added a rake task called spec:oneoh. This allows us to make sure that new versions of Merb still run against the same API as Merb 1.0, and that we’re not accidentally breaking working 1.0 APIs (to the extent that our 1.0 specs can make such an assertion).

The rule is that those specs must not be changed, with one exception. Specifically, we’re allowed to modify the 1.0 specs if the spec itself broke for reasons unrelated to the thing it was attempting to spec, and the breakage does not indicate a breakage in the 1.0 API. Additionally, any such change requires a public announcement, to minimize the number of such changes.

As a result, I’m announcing a small change in spec10, and the rationale for that change.

Many people have accidentally overridden core Merb::Controller methods in their controllers, with unpredictable (and confusing) results. From my Rails days, I remember this happening as well. As a result, Merb, starting with 1.0.2 and 1.1, will raise an error if you attempt to override a Merb::Controller method.

One of our tests had a controller which used the “method” action. Of course, “method” is already a method on all objects, so our new code caused it to raise an error. However, the test was not asserting that it was legal to override “method”; it was the same sort of accident that caused us to add this feature in the first place. As a result, we have gone into the spec10 directory and modified the action name to “request_method” instead of “method”, allowing 1.0.2 to pass the 1.0 specs.

And that requires this public post.

I hope that these posts will be few and far between, but at the very least, they show a commitment to the public API from the Merb core team that I hope you find reassuring. Thanks!

MythBusting — We Agree! Ruby is Awesome!

With all the disagreement over the past few days, it might seem like Merb and Rails are worlds apart. But David’s latest post demonstrates what I’ve been saying all along: we’re more alike than different.

In the latest installment, he takes on the myth that “Rails is hard because of Ruby”. Effectively, a bunch of people are comfortable in their language of choice (PHP, Java, Perl) and prefer to switch over to an MVC clone in their language of choice than learn the big bad scary Ruby.

As David succinctly argued, Ruby is just not that hard a language to learn. It’s better organized than PHP (with its absurdly large global namespace) and less ceremonious than Java (no IStrategyBuilderFactories here).

And what’s fantastic about Ruby is how quickly and simply new programmers are exposed to advanced concepts like lambdas. Because iteration is accomplished in Ruby almost exclusively with blocks, it’s near impossible to spend even a day in Ruby without learning what blocks are. Spend a little more time with Ruby, and the power of the closures that come along with lambdas becomes obvious. All without the need for an extensive study of the CS benefits of the construct.

Ruby is so easy, in fact, that the slim Learn to Program tome is written using Ruby as a base. It’s so straight forward that my wife, who’s hardly a programmer, was writing a program to count down “99 bottles of beer on the wall” within a few hours of picking up the book (and was even embellishing the program with a few interactive touches :-D )

Finally, implicit in the “I don’t want Rails because it’s Ruby” claim is the oft-heard myth that Ruby is slow. As I explained in my keynote at MerbCamp, Ruby does very well even compared with raw PHP. But when you compare Ruby frameworks (even Rails) against CakePHP, Rails destroys the competition. And Merb does even better. That’s because PHP’s fundamental architecture does not play very well with large frameworks, while Ruby deployment options can manage a fairly large framework with very small runtime impact.

What I showed at MerbCamp was that it was possible to squeeze up to 4,000 requests per second through the Merb stack, or around 1,500 requests per second when going through a controller and template, but that frameworks like CakePHP got only about 100 requests per second, even with a code accelerator.

Bottom line: don’t let anyone tell you that Ruby web applications need be slow. The language itself is certainly slow, but I don’t see a ton of Fibonacci web applications being built, so the real question is about where the bottlenecks are, and Ruby acquits itself very well.

Archives

Categories

Meta