Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Archive for the ‘Merb’ Category

MythBusting — Rails is not a monolith

Continuing his interesting train of thought, DHH posted on his blog yesterday that Rails is, in fact, not monolithic. In reality, he says, it is quite modular. In the post, he finally fully articulated the rationale behind the excessive use of alias_method_chain in Rails, which he says is to keep the code even more “modular”.

Let’s take a look at some of the claims:

Rails is not actually that large

They count all lines including comments and whitespace in Ruby files, thus punishing well-documented and formatted code

As I said yesterday, I did not actually include comments and whitespace. I specifically provided the command that I used, which removed whitespace-only lines and comment-only lines. In fact, the comment by bitsweat (a member of the Rails Core team) which started this back-and-forth erroneously included comments in Merb’s count, when merb-core has about 1 line of comment per line of code. This is why Jeremy incorrectly thought that merb-core was over 15,000 lines of code.

They count tests, thus punishing well-tested code

Neither Jeremy nor I counted tests. I’m not sure which LOC-count he’s referring to.

They count bundled dependencies, thus punishing dependency-free code

I did in fact make this mistake, but I disagree with the assertion that bundling dependencies makes code more “modular”. For example, bundling xml-simple in Rails causes a conflict with gems that require a newer version of xml-simple than the one bundled with Rails. Conversely, merb-haml has a haml dependency, which means that users can use newer versions of Haml that are released between Merb releases.

This statement by David actually teases out a fundamental difference of opinion between Rails and Merb. In effect, Rails prefers to bundle everything to reduce dependencies, while Merb prefers to use the existing Rubygems system so that applications can use different versions of the “bundled” dependencies.

Additionally, we don’t want to be maintaining bitrotted versions of things like tmail. We’d prefer to rely on gems created by experts in their niche who can maintain, and more importantly, fix bugs in, their code. We fully admit that our approach pushes the limits of Rubygems, but that has forced us to work with rubygems to improve a core piece of Ruby infrastructure.

Rails is actually pretty modular

The arguments made here almost defy reason, but let’s go through some of them.

First, Rails can include almost as much or as little of the six major pieces as you prefer.

Absolutely. I’ve referred to this in the past as the Lego vs. Duplo philosophy. Rails has added in a feature to allow you to remove entire blocks of functionality, but isn’t built on an architecture that lets you granularly opt-out. Granular opt-out allows you to reuse foundational code without buying into the full set of opinionated defaults.

One example of this is our auth system, which allows you to reuse the base auth code, which simply allows you to define strategies inside of a framework, even if you don’t want to use our built-in strategies or login views.

Granular opt-out builds a community around chunks of code that can be swapped in; having an auth core makes it easy to share small, simple authentication strategies between users of Merb.

The next part is the part that makes me incredulous. According to David, because Rails is spread across many files, it is “modular”. He describes how you would go about granularly removing certain features:

All these optional parts can actually very easily be turned off as well, if you so please. If you look at actionpack/lib/action_controller.rb, you’ll see something like the following:

ActionController::Base.class_eval do

  include ActionController::Flash
  include ActionController::Benchmarking
  include ActionController::Caching
  ...

This is where all the optional bits are being mixed into Action Pack. But they didn’t need to be. If you really wanted to, you could just edit this 1 file and remove the optional bits you didn’t need and you’d have some 3,500 lines of optional goodies to pick from.

Read that carefully. If you want to opt out of certain parts of Rails, you need to:

  1. Read the source, and figure out which files contain the features you want
  2. Figure out where the modules in question have been mixed in
  3. Fork Rails
  4. Modify your own personal version of Rails to remove modules that have been mixed in
  5. Never upgrade Rails again (or upgrade Rails and hope they haven’t made any changes to the part of the file you’ve modified)

Fundamentally, Rails’ “modular” architecture is fine for core committers, but it’s not particularly useful for consumers of the framework. Which makes sense, since 37 Signals sees Rails primarily as a library that they use to write their apps. So thinking about modularity in terms of the code of the framework itself makes perfect sense from that perspective.

On the other hand, Merb looks at modularity from the perspective of the developer using the framework.

alias_method_chain makes Rails modular

As I’ve said many times before, I don’t like alias_method_chain. But Rails’ philosophy around it is a perfect example of its problems. Superficially, it seems like it divides up responsibilities neatly into their own modules. And again, this is perfectly true from the perspective of developers working on the framework itself.

For consumers of the framework, it is simply maddening. Here’s a code snippet David provided in his post:

module Benchmarking
  def self.included(base)
    base.extend(ClassMethods)

    base.class_eval do
      alias_method_chain :perform_action, :benchmark
      alias_method_chain :render, :benchmark
    end
  end

This is an extremely common idiom in Rails. Unfortunately, it makes Rails methods extremely opaque. It is nearly impossible, without reading through a dozen files and putting together the puzzle, to figure out what the perform_action method actually does. This is evident in a Rails stack trace, which includes close to 10 frames for different parts of perform_action.

And this alias_method_chain isn’t even greppable. Effectively, it’s up to the user to divine that including a module modifies methods in the class, since it’s done via metaprogramming in an included hook. Again, this all makes perfect sense for developers on the Rails framework. But for consumers of the framework, it leads to many frustrating days trying to track down all the pieces of a particular method.

But why bother?

Again, this question makes perfect sense coming from the perspective of using Rails as an internal library. Don’t need something, like pagination? Simply move it out of the core framework. From the perspective of trying to develop something that can be used for many different purposes, in many different situations, we have different priorities.

Merb has already been used as the base for SproutCore, which generates static HTML out of a Merb base. It has been used for sinatra-like web services. And it’s being used by several large companies who don’t want to use DataMapper (and are using Sequel instead).

As people use Merb for more kinds of things, decisions like hardcoding features of Merb to a particular ORM (like Rails has done), requiring a certain directory structure, or bundling in dependencies become much harder to justify. People using Merb actually do use other gems that conflict with Rails bundled dependencies. People using Merb want to be able to upgrade their versions of Haml, ParseTree, or MailFactory without expensive surgery to the framework itself.

For the moment, these differences are the reason that Rails will continue to dominate amongst developers seeking to build apps similar in scope to apps built by 37Signals. I suspect that Merb will pick up steam amongst developers looking to build innovative apps leveraging the latest and greatest Ruby techniques and libraries.

I, for one, spent several years building large Rails applications, and am happy to leave the “just keep your own frozen, patched copy of Rails” philosophy behind.

UPDATE: Just to be clear, I know that Rails tries to use more recent versions of gems if they’re available on the system. However, that is a very naïve dependency approach, that in our experience, produced issues. It’s much better to be warned that you have dependency conflicts up front than to have them manifest in running production code.

MythBusting — Merb Style

The Rails team has lately been out in full force trying to bust some myths that seem to have come from the Merb community. In a reddit comment, in attempting to rebut a claim that Merb was significantly smaller than Rails, bitsweat made this “myth-busting” claim:

The merb stack is about 30KLOC; merb-core is half that. Rails minus Active Record is 46KLOC.

That seemed wrong to me, since I’ve recently clocked in merb-core at just 7,000 lines of code, and I know that merb-more plugins are relatively small, so I decided to do some counting.

My tool of choice: find lib -name "*.rb" | xargs cat |grep -v "^[[:space:]]*$" |grep -v "^[[:space:]]*\#" |wc -l

It may be relatively coarse, but it should be close enough to the actual figure to get us some real numbers. In case you need help there, I’m getting all Ruby files in the lib directory, catting them, then removing lines that are just whitespace or begin with “#” (comments).

Drumroll please.

merb-core clocks in at just 6,944. If you add in all of merb-more but auth, exceptions, and slices (features that Rails doesn’t have), that “bloats” us up to 13,322. If you add in those plugins (which clock in at a “massive” 2,282 LOC) we’re up to 15,604. Finally, throw in extlib (our “clone” of ActiveSupport) and we’re up to 17,291.

Next stop: Rails.

As per bitsweat, we did not count ActiveRecord. We also didn’t count ActiveResource or ActiveModel, which don’t have analogues in Merb.

Starting with humble action-pack, it clocks in at a “tiny” 12,126 LOC. action-mailer clocks in at a “miniscule” 6,405 LOC (take that, merb-mailer, with your bloated 216 LOC). ActiveSupport clocks in at 23,217 LOC. I wonder what they had to leave out to keep it so small compared to the extlib “clone”.

And finally, railties, a veritable smorgasbord of Rails extras, clocks in at 5,447 LOC.

merb-core + merb-more + extlib clocks in at 17,291. Rails clocks in at 47,195.

As you can see, it’s a tie.

Oh, and I forgot to mention that Merb contains rack handlers for mongrel, thin, ebb, fcgi, cgi, webrick, etc. which clock in at over 700 LOC. In Rails-land, that code is contained in the individual gems (such as the mongrel or thin gem).

UPDATE: When you remove the LOC that’s the result of bundled gems, it’s still a tie, with Rails clocking in at 23,923 and Merb clocking in at 15,009. See my Nov. 15 post for a more exhaustive treatment of modularity.

Merb RC5 (Final RC!)

Today, we’re releasing the final RC before 1.0 final ships at RubyConf. Again, the vast majority of the changes were bugfixes, and there were some concrete improvements as well:

  • The Merb spec suite was modified to run correctly on JRuby and Windows. All specs pass :)
  • Merb action-args now has support for jruby. Unfortunately, it doesn’t work with actions defined via define_method (this will be reported to jruby for a fix).
  • Webrat support is included out of the box with Merb. It currently requires the HEAD version of webrat, but we’re told they’ll be releasing a new version at RubyConf. See below for some webrat examples.
  • We now have a nightly gem server, hosted at http://edge.merbivore.com. At the moment, we release the next version every night (for instance, last nigh we released 0.9.13), which means you need to manually remove the merb gems before updating. Soon, we’ll be releasing 0.9.13.xxxx, so you’ll be able to run gem cleanup to remove old nightlies.
  • The goal with the nightly server is for people to use nightlies for riding edge as opposed to trying to install out of git itself. Trying to install out of git has its share of issues compared with installing from rubygems. To install from the remote gem server, either do gem install merb –source http://edge.merbivore.com –source http://gems.rubyforge.org or add edge.merbivore.com to your gem sources via gem source –add.
  • We put some work into the bundling system (as I previewed last week). To use bundling, simply make sure you have a gems directory in your application, then run thor merb:gem:install merb. This will install merb and all its dependencies, as well as binary scripts to work with the gems in Merb.root/gems. To run Merb, you run bin/merb. Note that you’ll probably need to install mongrel as well, via thor merb:gem:install mongrel if you want to use the bare merb command.
  • The view helpers were converted to use nokogiri and fallback on rexml when nokogiri is not available. have_selector now will take a CSS3 compatible selector, as opposed to the somewhat buggy and non-compliant hpricot selector.
  • have_tag and match_tag convert the attributes to a CSS selector in the background. We left have_tag and match_tag in for backward compatibility, although, it is advisable to use have_selector over have_tag. 
  • It is possible that have_tag / match_tag do not behave exactly the same way as a few bugs in the logic were fixed by moving over to a nokogiri back end.
  • As a result of this work, we have now marked all test methods with their API level, which should be locked as of 1.0.
We’re still on track for 1.0 final at RubyConf. The remaining major tasks are:
  • fully vet the test suite so that it can be used moving forward for all 1.x.x releases
  • work on the merb server to handle graceful shutdown of worker threads, as well as –restart to encapsulate the idiom of startup=>graceful kill.
  • go through the remaining tickets on LightHouse and fix them or defer to 1.x

Awesome bundling in Merb 1.0 final

Since we released RC1, people have had persistent issues with bundling Merb gems. It turns out that the most common reason, by far, is conflicts between gems bundled in your app with gems available in your system.

When we first set up bundling, we thought it would be convenient to be able to bundle only some of the dependencies for your application. As it turns out, that blunts the reason for bundling in the first place: hassle-free, write-once-run-anywhere applications.

Through the course of four RCs, we’ve tried a number of things, none of which were really effective. The only full solution that guarantees that your bundled app runs correctly on development boxes and production boxes is to bundle everything in your app, and completely ignore your system gem repository.

While it is easy enough to mix and match, the number of reports of failure, combined with failures by the core team itself, leads us to believe that this is a *bad* idea, especially given some of the current assumptions made by Rubygems (which causes it to activate the latest version of a gem, which might conflict with a different dependency).

A simple example of a failure:

  • merb-haml 0.9.12 relies on merb-core 0.9.12
  • Imagine you have merb-core 0.9.13 installed in your system
  • running the merb binary loads the latest merb from all your repositories.
  • this will load merb-core 0.9.13
  • a dependency on merb-haml 0.9.12 will try to activate merb-core 0.9.12, which will conflict with the already loaded 0.9.13.

This is just one example of many issues that people had. The solution is to rely entirely on bundled gems, and remove system gems from bundled binaries.

The side-effect is that you will need to bundle gems like mongrel, rake, etc. But it’s clear to me that even bundling mongrel and rake is a good idea, as different versions of mongrel or rake can have subtle differences that lead to infuriating bugs. I personally have experienced some of these bugs in my day, and being able to package up an entire working app would have been a godsend!

RC5, which should be the final release before 1.0 final is released at RubyConf, will include bundle-everything by default, as well as more support for webrat, JRuby, and Windows.

Thanks!

Merb 1.0 RC3 Released!

We released Merb 1.0 RC3 today, inching a bit closer to a 1.0 final, which we intend to release at RubyConf. RC3 is mostly bugfixes, but there are some items of note:

  • Since we intend to support webrat as a primary mode of testing in 1.0 final, we’ve restructured the testing helpers to make that possible. No testing API changes at this time.
  • In 1.0 final, get/post/put/delete will be aliases for request(). At this time, they’re still aliased to the old pre-1.0 request()-helper. To help ease the transition, we will probably provide a use_legacy_request_helper! method.
  • There was a bug in request testing that preventing sessions from persisting for more than 2 requests. We have implemented a cookie jar, so sessions now persist properly (basically, we emulate a real browser).
  • As a result, you can now pass a :jar parameter to request(), which will allow you to test multiple sessions in the same test. If you specify no :jar parameter, all request() operations will use a :default jar. The syntax is request(“/foo”, :jar => :my_jar). request(“/foo”) is the same as request(“/foo”, :jar => :default).
  • merb-action-args works with ParseTree 2.x and 3.x.
  • merb-auth had a few bugs around redirecting back and retrying failed logins that were resolved.
  • The final, good API for dependency is solidified. A few things that you can do now: dependency “parse_tree”, :require_as => “ParseTree” { # do stuff after the gem is loaded }. You can also pass :immediate => true to have the gem load immediately, instead of being deferred until the framework is set up (the default).
  • We should now fully support Windows.
  • Seamless JRuby support is coming via Glassfish. RC3 has some fixes to make it simpler for Glassfish to hook in.
Good night and good luck!

Merb Master Process

Apologies for the delay in posting about this. I was enjoying the fantastic Ajax Experience and jQuery Camp out in Boston, where I got a chance to hang with my jQuery compatriots as the huge Microsoft and Nokia announcement broke.

Now that I’m back, I want to talk about the cool new features we’ve been adding to Merb’s server. Let’s take a look at them one at a time:

  • When a Merb server boots up and tries to bind to existing ports, it doesn’t crash. Instead, it waits for each port to become available and binds when they do. As a result, you can start up a new merb, and then gracefully kill the old one. This should make restarting clusters of merbs significantly less painful, and with almost no downtime.
  • Merb’s clusters have been rewritten to take advantage of Ruby Enterprise Edition, which makes it much easier to share memory between workers in a Merb cluster. Using Ruby Enterprise with Merb 1.0 should yield around the same 30% memory improvement that Phusion Passenger yields.
  • Merb clusters are now controlled by a master process, which can be told to gracefully kill all of its children, or reload the application code (but not init.rb or gems). 
  • Sending an INT to the master process (directly or via merb -K all) tells each of the worker processes to gracefully die. 
  • Sending a HUP to the master process tells each of the worker processes to gracefully die, but to start up a new cluster with reloaded code. This bypasses reloading Merb, gems, and init.rb, so the restart is much quicker than doing a full reload. This is also the same internal code used by development mode code-reloading, which has now been made 100% foolproof.
  • Killing a worker process (either via INT, which is graceful, or KILL, which isn’t) will cause a new process to be respawned instantly. We’re talking basically no time at all, since no code needs to be reloaded (the spawner process forks right before binding to a port)

All of these improvements are just harbingers of even more improvements, making Merb’s master process even more powerful and smart.

The most important thing to keep in mind is that Merb has been significantly tuned for Ruby Enterprise, and with regular Ruby, there is a bit of overhead for the master processes. All of the features above will work correctly with standard Ruby, but you’ll get them for free (memory-wise), as well as quite a bit of improved memory overall in a cluster by using Ruby Enterprise.

Please please please check it out.

Status Update: Merb 1.0

With Merb 1.0 right around the corner, we’re buckling down and making sure that Merb is rock-solid by October 11.

A bunch of features have landed or are landing over the weekend:

  • Taking advantage of Ruby Enterprise to reduce memory overhead
  • Simplifying deployment of Merb clusters
  • Improving the errors that Merb spits out when something goes wrong inside Merb.
  • Fixing an infrastructure problem that caused Merb’s logger to cause thread-safetiness issues in JRuby. Also resolve the comingling of logged information for multiple requests in threaded environments.
  • Fix several bugs in merb_helpers that were driving people nuts when trying to do non-standard things.
More on the first two this Monday. We’ve been doing a lot of work to make deploying Merb apps simple and efficient.
Have a great weekend!

MerbCamp

Just a reminder folks: MerbCamp is October 11-12.

The conference is going to feature a large number of short talks about various features of Merb, as well as keynotes by Ezra Zygmuntowicz, who originally wrote Merb, and me. I’ll be giving the closing keynote about the future of Merb, as well as short talks on testing and jQuery.

This is going to be the biggest gathering of Merb folks thus far, but tickets are selling out fast. Register today!

Rails is a Pre-Fab House

At RailsConf, I had the opportunity to speak to a lot of people about Merb. I also listened to DHH’s keynote, which touched on some related concepts.

People coming from a Rails background, and are looking at Merb for the first time, wonder why they should throw away the phenomenal efficiency improvements they got from switching off of error-prone, tedious configuration scripts for another set of configuration when Rails makes all the decisions for them. DHH made this point fairly explicitly in his keynote. To sum up, he argued that people only think they want to make decisions; instead, they really want to be given a fully working, homogeneous solution so they can get on with their business.

Believe it or not, I agree.

The devil, as they say, is in the details.

In the real world of software applications, the monolithic solution doesn’t always fit your needs perfectly. That’s why Rails implemented, very early, a plugin architecture, which immediately reduces homogeneity.

You see, Rails is like a pre-fab house. For people who have gone through the pain of constructing a house before, pre-fab houses are very appealing. Suddenly, there’s no need to spend time with architects, contractors, or painters. You just place an order and you get a house. It seems great. You can get the furniture you want, or even repaint it.

And when you need to bring in a contractor for routine maintenance, you can have your pick of contractors. Because your house is the same as every other house they worked with. But soon you decide you want a window. Unfortunately, your house is pre-fab, so you pick up a hacksaw and hack away. Everything seems great until the first time it rains. And the water starts coming in.

Undaunted, you pick up the phone and call all those contractors you worked with before. When they get there, they take one look and let out a sigh. They have no idea how to deal with the glaring mess you have made for yourself. After paying a hefty bill to get everything fixed, you tell yourself that it’s still worth it. After all, you’re still ahead on time compared to Andy down the block who had to do everything from scratch.

But you see, there’s another option.

At a builder’s show, you see a shiny booth run by a new company called Mer Builders. They have put together a house kit that allows you to put together your house quickly and efficiently. And when you want a new window, you just pull out one of the wall blocks and replace it with a window block. Quick and easy.

But wait, you are probably thinking. I still need to put together the house. That smacks of all the pain of building a house from scratch. It’s not as bad, but I’ll still need to decide what pieces I want, figure out how they hook up, hire someone who knows how to put everything together, and more. Who wants that. A pre-fab house will do just fine, thank you very much.

As you turn to leave, the Mer Builders salesperson calls after you. You see, it turns out that they understand that you’d rather not have to make all those decisions yourself. And they’ve put together a starter kit that looks kind of like the pre-fab house. You can order it today and have it delivered tomorrow. But when you want to replace a window, their network of certified Mer engineers will simply and easily swap out a wall block for a window block.

Heck, soon enough you’ll probably be doing all sorts of customizations on your own.

The moral of the story is… Use Mer Buil… er… Merb!

RailsConf Slides

Hey guys,

My talks at RailsConf were a ton of fun. The first night I did talks on Merb and jQuery which were completely packed, and the first day of the conference I did a talk on DataMapper which seemed to be well received as well.

While it was nice to hear that so many people wanted to see the DataMapper talk that some people were turned away, it sucks that some people got turned away. Without further ado, go check out my Merb and DataMapper slides at Slideshare:

I’ll try and get up the jQuery slides shortly as well. Enjoy!

Archives

Categories

Meta