Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Rack as a Transformative Figure

I had occasion to think about Rack a few times today. First of all, I was writing some proposals to a few conferences about Rails, which always gets me thinking about architecture. In this case, it got me thinking about how Rack is influencing Rails’ architecture moving forward. Second, a friend told me he was considering moving from an archaic web language to Python (specifically Django). I got to thinking about how the future of Ruby is intertwined with Rack, and why that might not be the case with Python and WSGI, unfortunately.

Like Ruby, Python has a single dominant web framework in Django. According to Google Trends, Django gets about eight times as many programming-related searches as the next-most-popular framework, Turbogears. And TurboGears is 1.5 to 2x the size of Pylons and CherryPy, other frameworks. Naturally, Rails dominates the Ruby space more thoroughly (in the last 12 months, Rails beat the next-most-popular framework, Merb, by more than 60 times), but the effect is roughly the same. Many more people contribute to the Django and Rails ecosystem than the Turbogears and Merb ecosystems.

Fortunately for Ruby, however, Rails (our dominant framework) has thoroughly embraced Rack, an abstraction layer and specification for the code between a web framework and a web server. This embrace of Rack has brought much-needed oxygen to the specification, and has allowed Merb, Rails, Sinatra, and others to cooperate in a shared space. In the next few months, Merb and Rails will be making their routers a shared Rack component, and the same is true for a number of smaller elements, like parameter parsing. In part, this is possible because unlike WSGI, Rack is a specification *and* a library. In the WSGI universe, the WebOb library provides similar functionality, but is not as widely shared.

In particular, while Django itself can run on WSGI servers, Django does not use WebOb, nor is its middleware WSGI-compliant. Let’s look at a specific example, a piece of middleware that wraps all requests in Django in a transaction:

from django.db import transaction
 
class TransactionMiddleware(object):
    """
    Transaction middleware. If this is enabled, each view function will be run
    with commit_on_response activated - that way a save() doesn't do a direct
    commit, the commit is done when a successful response is created. If an
    exception happens, the database is rolled back.
    """
    def process_request(self, request):
        """Enters transaction management"""
        transaction.enter_transaction_management()
        transaction.managed(True)
 
    def process_exception(self, request, exception):
        """Rolls back the database and leaves transaction management"""
        if transaction.is_dirty():
            transaction.rollback()
        transaction.leave_transaction_management()
 
    def process_response(self, request, response):
        """Commits and leaves transaction management."""
        if transaction.is_managed():
            if transaction.is_dirty():
                transaction.commit()
            transaction.leave_transaction_management()
        return response

As you can see, Django allows multiple methods in its middleware (process_request, process_response, process_exception, and process_view). As with rack, WSGI middleware uses a single method to wrap the entire request. Here’s how the above Django middleware would look in Rack:

class TransactionMiddleware
  def initialize(app) @app = app end
 
  def call(env)
    transaction.enter_transaction_management
    tranaction.managed(true)  
    begin
      @app.call(env)
    rescue
      transaction.rollback if transaction.dirty?
      transaction.leave_transaction_management
    end
    if transaction.managed?
      transaction.commit if transaction.dirty?
      transaction.leave_transaction_management
    end
  end
end

Leaving aside the differences in language, the Django middleware is incompatible with WSGI for no discernable reason. This means that Django middleware cannot be shared with the rest of the (admittedly smaller) Python web ecosystem, which makes it harder, for instance, to use Django’s ORM separately from Django (because you’d have to write your own transaction-wrapping middleware).

The big win of WSGI and Rack is that like Unix pipes, they make it easy to mix and match web framework elements to develop something that works for you. It makes it easier to develop entire frameworks, like CloudKit, that can be used as simple plugins to larger frameworks like Rails. Rails’ embrace of Rack, in the long term, is going to significantly open up experimentation in the community to non-Rails solutions that can be used WITH RAILS while they are being ironed out. In fact, being able to trivially run a Sinatra app inside a Rails app is an important design goal for Rails moving forward.

When you look inside the Rails middleware folder, you see things like this in the Rails Failsafe middleware:

def call(env)
  @app.call(env)
rescue Exception => exception
  # Reraise exception in test environment
  if env["rack.test"]
    raise exception
  else
    failsafe_response(exception)
  end
end

Instead of assuming something specific about Rails test mode, this middleware is allowing the community to come together around specifications that all Ruby frameworks can use together (in this case, a generic “test mode”). And this middleware can be used in Sinatra, Merb, or CloudKit by simply requiring “action_dispatch/middleware/failsafe” (as of Rails3) and then adding ActionDispatch::FailSafe as a middleware. No other Rails assumptions are made.

The bottom line is that this is going to be very powerful moving forward, and it’s too bad that Django’s insistence on a go-it-your-alone approach is damaging the Python community’s attempts to do the same.

On a side note, I want to be clear that I am aware that Django can run inside of a WSGI server, and can use WSGI middleware. However, Django middleware cannot be used by WSGI, and a recent post on the topic, which advocated the use of WSGI middleware in Django apps, had to be sure to include “I’m not trying to stir up any controversy, I’m not saying we should stop making Django middleware or anything like that”. I, for one, think that’s a shame.

10 Responses to “Rack as a Transformative Figure”

Django’s “middleware” is perhaps badly named — it’s basically just a system of API hooks into the internal request-processing cycle. That means it’s different from, and serves different purposes than, WSGI “middleware” or Rack “middleware”.

There’s also, IMO, good reason to have things like Django’s “middleware” API *in addition to* the gateway interface, since the gateway interface (in Python, at least) has an API that’s frankly embarrassingly anemic.

But anyway, sigh. I hadn’t realized the misinformed FUD had made it quite *this* far out of the community, and in the name of not getting dogpiled on I’ll just shut up and pretend it didn’t happen.

(and it’s not that I think you bear any ill will, by the way, just that you’ve probably seen these types of statements repeated often enough by Python folks that you assumed they were correct)

You make a good point but, to be fair, I think you could have said the same thing about Rails 6-10 months ago. In fact, a stable version of Rails has yet to be released using Rack. Rails 2.3 will be a very nice upgrade when it finally drops.

Django, to me, has always lagged behind Rails on features and has been playing catchup the entire time. Of course, I’m a bit biased because I’m a Rails/Merb dev. Give the Django and Python guys a little time. I’m sure they’ll catch up.

@james as far as I can tell, there’s nothing about Django middleware that could not be implemented as regular WSGI middleware. I provided an example in my post of a case where three of the hooks were being used, and yet it could still be converted into Rack/WSGI-style middleware.

Was my example incorrect? If so, where?

@jonathan the difference is that Rails is moving ever-closer to a full embrace of Rack, starting with 2.2, while Django has purposely gone in a different direction, fracturing the Python WSGI community.

Rack was brand-new 6-10 months ago, and Rails spared no time in moving to it, to the point that Josh Peek, a Rails core team member, was made a member of the Rack core team.

James,

“embarrassingly anemic?” Seriously? I think that’s crazily overstated, and borders on intentional disinformation.

WSGI may not be perfect, but you can do a lot with it, as proven by actual working code. WebOb provides nice clean interface for working with WSGI if you don’t want to get down and dirty with the details of the environ.

WSGI middleware provides a standard way to work with the request/response cycle in a way that transcends framework boundries. If you don’t care about that, fine, but that doesn’t mean that it’s not valuable, powerful, and flexible.

It also see the simplicity of the Rack specification, and adherence by all of the Ruby web development infrastructure as a powerful driver for better modularity. In fact I just wrote an article about it myself http://talklikeaduck.denhaven2.com/articles/2009/03/05/railsrack

I noticed, while reading code to write the article, that Rails 2.3 uses the environment keys ‘rack.session’ and ‘rack.session.options’ to allow access to the Rails session by other rack apps on the middleware stack.

By my reading of the Rack spec, this is invalid, since the spec reserves environment keys starting with ‘rack.’ to those listed in the specification.

I’m wondering if this is something which will be put into the Rack spec, and if not, should Rails change those keys to something conformant.

It is my understanding that the rack spec is undergoing change to specifically allow some conventional extensions in the rack.* namespace.

Leave a Reply

Archives

Categories

Meta