Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; he spends his daytime hours at the startup he founded, Tilde Inc.. Yehuda is co-author of best-selling jQuery in Action and Rails 3 in Action. He spends most of his time hacking on open source—his main projects, like Thor, Handlebars and Janus—or traveling the world doing evangelism work. He can be found on Twitter as @wycats and on Github.

Python Decorators in Ruby

This past week, I had a pretty long discussion on StackOverflow about Ruby and Python, that was touched off by question about whether there was anything special about Ruby that made it more amenable for Rails.

My initial response covered blocks (as a self-hosting way to improve the language) and the ability to define new “class keywords” because of the fact that class bodies are just normal executable code. The strongest argument against this was that Python decorators are as powerful as Ruby’s executable class bodies, and therefore can be used to achieve anything that Ruby can achieve.

Python decorators do in fact provide a certain amount of additional power and elegance over the Python before decorators. However, this power is simply a subset of the available Ruby functionality. This is because Ruby has always had the ability to add features like decorators to the language without needing language changes.

In other words, while Python has added a new hardcoded feature in order to achieve some of the power of Ruby, adding such a feature to Ruby would be unnecessary because of the underlying structure of the Ruby language.

To show this clearly, I have implemented a port of Python function decorators in under 70 lines of Ruby. I wrote the code test-first, porting the necessary functionality from two Python Decorator tutorials written by Bruce Eckel in October 2008.

The list of features supported:

  • Decorate any object that responds to call via: decorate SomeKlass or decorate some_lambda
  • A simpler syntax for class names that can be used as decorators: SomeKlass(arg)
  • The ability to give the decorator a name that could be used instead of the class name, if desired

Some examples:

class MyDecorator
  def initialize(klass, method)
    @method = method
  end
 
  def call(this, *args)
    puts "Before MyDecorator with #{args.inspect}"
    @method.bind(this).call(*args)
    puts "After MyDecorator with #{args.inspect}"
  end
end
 
class WithDecorator
  # we could trivially add these to all classes, but we'll keep it isolated
  extend MethodDecorators
 
  MyDecorator() # equivalent to decorate MyDecorator
  def my_function(*args)
    puts "Inside my_function with #{args.inspect}"
  end
end
 
WithDecorator.new.my_function(1)

When MyDecorator() is called inside the WithDecorator class, it registers a decorator for the next method that is defined. This is roughly equivalent to Python’s function decoration. Some other examples:

class MyDecorator < Decorator
  decorator_name :surround
 
  def initialize(klass, method, *args)
    @method, @args = method, args
  end
 
  def call(this, *args)
    puts "Before MyDecorator(#{@args.inspect}) with #{args.inspect}"
    @method.bind(this).call(*args)
    puts "After MyDecorator(#{@args.inspect}) with #{args.inspect}"
  end
end
 
class Class
  include MethodDecorators
end
 
class WithDecorator
  surround(1, 2)
  def function(*args)
    p args
  end
end
 
WithDecorator.new.function(1)

In this case, I’ve inherited from the Decorator class, which allows me to add a decorator name, which can then be used in a class with MethodDecorators mixed in. In this example, I’ve mixed MethodDecorators into Class, which makes decorators available for all classes. Again, I could have made this the default, but I like to try to make new behaviors global if I can avoid it.

This is of course a first pass and I’m sure there are subtle inconsistencies between Python’s decorator implementation and what I have here. My point is just that the feature that was added to Python to add flexibility is merely a subset of the functionality available to Ruby, because Rubyists can implement new declarative features without needing help from the language implementors, and have always been able to.

45 Responses to “Python Decorators in Ruby”

So what happens if another library relies on the method_added hook? IMHO ruby’s hooks are too simplicistic.

Wycats,

Are there any multithreading concerns with the decorator implementaions in Ruby?

I ould say “rubyist can *experiment* new declarative features, as such extension rarely grow up. I implemented your same hack when I first learned about python decorators, then never used it.

A flexible syntax and hook system makes it possible to experiment but I’ve never seen this hacks take off in ruby, and I can remember at least DBC, type checking, multimethods, coroutines, delimited continuations, interval arithmetic, units, real named args, conditions/restart etc. Mostly. I guess, because they are implemented on top of features that make them hard to understand and debug, error prone and basically impossible to get right when outside of a toy environment .

For example

>> Dummy=Class.new
=> Dummy
>> class T ; extend MethodDecorators end
=> T
>> T.C
NoMethodError: undefined method `key?’ for nil:NilClass
from ./decorators.rb:11:in `method_missing’
from (irb):3
from :0
>> T.c
NameError: wrong constant name c
from ./decorators.rb:9:in `const_defined?’
from ./decorators.rb:9:in `method_missing’
from (irb):4
from :0
>> Dummy.C
NoMethodError: undefined method `C’ for Dummy:Class
from (irb):5
from :0
>> Dummy.c
NoMethodError: undefined method `c’ for Dummy:Class
from (irb):6
from :0

you broke my ruby ;)

The single exception of some syntax hack that was successful, and that my poor memory can remember is Symbol#to_proc, and well, it wasn’t that big.

@sam there shouldn’t be any multithreaded concerns if you add the decorators during the parsing phase, before any multithreaded activity occurs. As a general rule, Ruby applications that rely on threading should not initiate threaded activity until they have loaded the application.

The decorator syntax used in Python is just that — syntax. It’s effectively fluff to make code like the following a little prettier:

def mydecorator(fn):
def wrapfn(x):
print “before”
fn(x)
print “after”
return wrapfn

def foo(x):
print x
foo = mydecorator(foo)

“foo = mydecorator(foo)” is effectively identical to “@mydecorator foo”.

Maybe I missed the point of your article (entirely possible!) but the @ syntax is largely (entirely?) sugar. It was just considered to be relatively ugly.

Excuse the shoddy formatting. Hopefully the point is still clear.

@thomas my point is that the Ruby language is malleable enough to make it reasonably easy to add things that would require syntactic elements in other languages.

From PEP 318: “The current method for transforming functions and methods (for instance, declaring them as a class or static method) is awkward and can lead to code that is difficult to understand. Ideally, these transformations should be made at the same point in the code where the declaration itself is made”.

My implementation satisfied the requirements of the PEP without requiring a new syntax element. Obviously, this was not possible in Python.

I’d also point out that it’s quite possible to support a decorator syntax like %Decorator or +Decorator in Ruby, since there are quite a number of “symbols” that can be overloaded in Ruby.

Don’t you agree the Python decorator syntax is more elegant than the ruby code you posted? It’s also more readable by both humans and machines. Xcode recognizes functions with certain decorators and automatically make them available to link to in Interface Builder.

@ahmad I posted a few syntaxes, and mentioned another potential one in the comments, all of which are possible to implement in pure-ruby, needing no new language features. Using the form where the class name itself is used (which I personally prefer and is consistent with the Python approach), editors can trivially link you to the class that created the decorator. With slightly more editor work, it would be possible to link named decorators as well.

Additionally, some of the syntaxes I proposed are identical to the Python versions, so I don’t see how it can be less elegant. Unless you think that added a “@” symbol makes it more elegant?

Wycats, please read the comments carefully. Maybe you should read Thomas Lees point again ;)

Decorators are just normal functions, no need for 70 lines of code:
class Foo(object):
@staticmethod
def bar():
pass
# Is functional wise the same as
bar = staticmethod(bar)
# Decorators are pure python too!

The compiler emits the assignment for you. That’s why it’s called syntax sugar ;)
And in python a class definition is an executable statement too!

Btw, ruby uses syntax* (sugar) for declaring static methods :P

@RandomUser seems you missed my entire point. Check out the Python source–in particular ast.c and compiler.c. The decorator feature was added to Python’s C source *itself*, and hundreds of lines of code in those two files, intermixed with other AST and Compiler code, handles decorators. This is in part because the decorator feature requires changes to ast_for_funcdef, for instance.

The underlying point is that in order to make decorators work (in the way the PEP describes as desirable), it was necessary to modify Python itself–it was not possible to implement the feature, even close, in pure-Python. Thomas is simply showing that the underlying functionality was still possible before the new syntactic feature, not that Python is powerful enough to add new syntactic features (as Ruby is).

also @RandomUser, what do you mean by “ruby uses syntax* (sugar) for declaring static methods”. There are no static methods in Ruby. There are only methods on an object. Since classes are objects, you can declare methods on them.

class Foo; def self.bar; end; end

is equivalent to

def Foo.bar; end

which is equivalent to

class << Foo; def bar; end; end

In other words, Ruby has syntax for adding methods to objects (def obj.method_name) and it works just fine for classes too.

Hi Yehuda,

There was never an issue on how to implement decorators. Before the syntax came along, pythoneers were decorating to such an extent that it was thought ‘special’ enough to highlight it with extra syntactic support in the language, which also frees old implementers to use a standardized, best in class, implementation; and allows those who previously did not decorate, access to a well documented and powerful packaged feature.

You should guard against the ‘Lispers response’ of saying “Our language is so flexible that feature XYZ, once explained, is trivial to implement – here’s an implementation, I’ve tested it against restricted examples and it works”. Such a response may leave the reader thinking what other common tasks are left up to the user to implement, or are similarly lightly tested?

- Paddy.

@paddy3118 I’m no lisper, but I strongly suspect that “Our language is so flexible that feature XYZ, once explained, is trivial to implement” is actually true about Lisp. Unfortunately, lisp has corners that can produce very incomprehensible code, so some restrictions on that power can be useful.

The history of decorators is reasonably well-documented in PEP-318. While you say that “There was never an issue on how to implement decorators”, the PEP has a section entitled “Why is this so hard” that describes some of the questions related to how to implement decorators.

My point is *not* that people should take my decorator implementation and use it. I find the decorator approach to be fairly limiting when the power to create new language features in the language itself is so freely available. My intent was to show that Python-style decorators are a small *subset* of the available functionality in Ruby.

Any of the languages functionality can be implemented in the other language. The question is just how complicated it is. It’s perfectly possible to make code that modifies the class from the class body in Python too. zope.interface does it. But it requires heavy magic, so Ruby totally wins. And Python has a small win when it comes to decorators, because it has them, and you don’t have to rely on some pretty magical code (I’m not sure I understand it anyway as the RubyNuby I am) to make them.

But then, as I pointed out in the post I wrote after the discussion (see link), you may actually think that some of these features, like decorators or whatever are BAD. :-) It’s all a matter of taste.

It would be really nice if you give a rubesque implementation of the Python descriptor API. The Python decorators were introduced to facilitate use of the descriptors.

@giorgi I can (and will), but the main reason for descriptors in Python is that attributes and methods are two separate concepts. From the author of Pro Django: “These are all things that would normally need to be done with a method, but if you’ve already started with a basic attribute, changing to a method would require changing all the code that uses the attribute to use a method call instead.”

In Ruby, this would not be a problem since “attribute access” is actually just a method call, so you can trivially change an attr_accessor to a method call without changing any of the consumer code.

People had been using decorators in Python before they were a part of the language. Your theory that is could not be done in the language is FALSE and misguided. The language modifications were to make the implementation efficient, consistent and clean as it is in the spirit of Python.

Metaprogramming is neither alien nor profound to the Python or its community. It is just highly discouraged for prudent reasons. Everything (packages, modules, classes, methods, functions, etc) in Python is an object. Every object can be modified at runtime. Python provides all kinds tools to inspect, modify, initialize objects at runtime. So you have ALL the metaprogramming features you need to go crazy in Python. And this has been the case since Python 1.5 or so.

@mystilleef “Your theory that is could not be done in the language is FALSE and misguided.” I’ll just let the PEP speak for itself:

“The current method for transforming functions and methods (for instance, declaring them as a class or static method) is awkward and can lead to code that is difficult to understand. Ideally, these transformations should be made at the same point in the code where the declaration itself is made. This PEP introduces new syntax for transformations of a function or method declaration.”

I don’t think it’s particularly controversial to say that the situation pre-decorators was awkward and could lead to hard to understand code. It is also self-evident that the functionality could not be implemented in Python itself.

Yehuda, thank you a for an illuminating insight into this Python language feature. I happen to be a long-time fan of Python and a more recent fan of Ruby. I think both languages are excellent, each with their own unique flavor, strengths and weaknesses. I totally agree with you that Ruby allows DSLs and language features to be added/extended in a very elegant way. Python is not terrible in this department, but does not hold up to Ruby in terms of flexibility.

I just don’t understand the defensive responses by some of the people here. Your post was eloquent and informational. Keep up the great work.

“I don’t think it’s particularly controversial to say that the situation pre-decorators was awkward and could lead to hard to understand code. It is also self-evident that the functionality could not be implemented in Python itself.”

You just contradicted yourself, sir. Prior to the syntatical addition of decorators to Python, implementing decorators in the language was awkward and hard to understand. That’s right. But it wasn’t IMPOSSIBLE. It was possible. And it was done! You had modules and third party library that provided this functionality. Except that it wasn’t clean enough for Pythonistas who are syntax purists.

Let me put it in simple terms. Decorators are syntatic sugar. What I’m trying to make you understand is that decorators were being used even before they became canonized in Python. Even the PEP you point to makes this glaringly clear. For almost every decorator example given, there’s a non-decorator equivalent.

So, sir, it not self-evident that this functionality could not be implemented.

I forgot to add this link. Please read this. It gives examples of how decorators where being used before syntax was added to Python.

http://www.ibm.com/developerworks/linux/library/l-cpdecor.html

@mystilleef Let me try to be clearer. Before decorators, it was indeed possible to achieve the desired effects, but the Python community decided that the mechanisms available led to awkward and hard to understand code. As a result, they introduced a new syntax to Python. Nothing similar to this syntax could be implemented in pure Python. On the other hand, it is reasonably simple to add this similar syntax to Ruby without needing a new language feature.

Hi wycats,
A guiding principal of Python is that the language should strive to be readable. It is why many, many, (many), requests to add syntax modifying features such as macros are rejected. The idea being that when you read one Python program, no one can add, for example, their own implementation of a switch statement and keyword that works in a subtly different way, to someone elses. Likewise, no one could add the @ syntax for decorators to the language, without it being officially integrated into the interpreter itself.
What Python gains from this is that readers can rely on the syntax of the language to be consistent. Importing a module cannot change the syntax of code.

Contrary to what mystileef has said, their has also been restrictions put in place on certain built-in objects – you cannot, for example change how an int works. The idea here is similar – even if you had some good idea for maybe changing the default rendering of ints to hexadecimal for example, other readers of your code would have the difficulty of working out where in your code do integers start having this non-default behaviour, and what are the implications for code re-use and maintenance? In Python certain basic functionality will not change.
This last, I know, is a bit of a turn-off to some coders who see a restriction and want it removed, or who really like working without that restriction. Unfotunately Python isn’t likely to change soon for you, but might still be your best language to use for other reasons, such as our community of Pythoneers.

I’m rambling into marketing. I’ll stop :-)

@paddy3118 thanks for your sober response ;)

If you take a look at Matz’s early US commentary, he argues for the particular feature-set of Ruby (as opposed to the more malleable syntax of Lisp) for similar reasons. It seems as though Ruby and Python both see the need to restrict power in favor of readable syntax, but Ruby tilts more in favor of power.

Hi Yehuda, independent of what is said in that book, the main reason for descriptors is to customize the way an object (and a class, as the latter is an object) interacts with the members of its dict. Ruby, I guess, also states distinction between an attribute and a method, the former being kept private; there is no way to change this behavior in Ruby, is it? (I mean, when I call attr_accessor :anattr, the class method ‘attr_accessor’ adds two methods, ‘anattr=(val)’ and ‘anattr’ to the class through metaprogramming, but, without that, there is no way to access @anattr if self is not the default receiver.)
In Python, however, one can re-implement __getattribute__ (for any class) to allow for a call of only those members of the dict, which are callables and/or properties.
BTW, one can use decorators even in Python 2.3 (i.e., before the decorators were formally introduced); see http://peak.telecommunity.com/DevCenter/DecoratorTools

If you really want to get an instance variable you can call #instance_variable_set() and #instance_variable_get(). You can set the #method_missing() method too if you want to access variables at any given time, thats how OpenStructs work in ruby.

You are right. It is not easy to add new syntax to Python. In fact, the Python community discourages it.

@giorgi

Although not explicitly in the API, you can implement __getattribute__ in Ruby as well, with method_missing. You don’t *need* method_missing, you could make it available through some global `getattr()` method the way Python does, but here:

def method_missing(sym, *args, &block)
if instance_variables.index(“@#{sym}”.to_sym)
instance_variable_get(“@#{sym}”)
else
super
end
end

You could do the same for setting attributes.

“I don’t think it’s particularly controversial to say that the situation pre-decorators was awkward and could lead to hard to understand code. It is also self-evident that the functionality could not be implemented in Python itself.”

python 2.4+
———–
@mydecorator
def foobar(arg):

python 2.3-
———–
def foobar(arg):

foobar = mydecorator(foobar)

by my count, 16 characters shorter (depends on the length of the method name) and a little clearer. it brings the decoration visually next to the method signature, which is nice, but i think PEP 318 was overstating the pre-decorator situation a bit. there’s yer controversy :)

I’m with you on the statement that both restrict power when compared to lisp but that ruby leans in favor of power next to python. my attempts at learning ruby (probably looking at all the wrong examples) have brought me down in favor of python’s balance, and that’s every bit as valid as your preference for ruby. The original post sure comes off sounding like an attempt at “A is objectively better than B” though.

That power to play fast and loose with syntax is great for a seasoned rubyist like yourself, but frankly i’m kind of glad that syntactic changes to python require an interpreter patch. I like to be able to read someone else’s code without having to read the code that creates the syntax that the rest of the code is written with.

70 lines of ruby certainly beats a few hundred lines of C any day, but wake me when the ability to decorate a function or class without a written-out assignment is a project requirement somewhere. until then we’re pretty much chasing our tails here.

@teepark “The original post sure comes off sounding like an attempt at “A is objectively better than B” though”

Probably more like “A is objectively more powerful than B”, which there’s a very real chance you’d disagree with.

nope. one can be implemented in the other and not the other way around. it’s…not particularly controversial *holds breath*

but with no discussion on the costs to readability, it’s not a big leap from “more powerful” to “better”. so i stand by my version.

@Loren Segal
Actually, the Python’s counterpart of Ruby’s ‘method_missing’ is ‘__getattr__’,'__setattr__’, ‘__delattr__’ trio, not ‘__getattribute__’. The difference is that the latter handles *any* dot call in expressions like ‘anObj.anAttr’ whereas ‘__getattr__’ comes into action only when ‘anAttr’ is not found by ‘__getattribure__’ call.

@mystilleef
You are right. The striking difference between Python and Ruby is the community culture, not the language features per se. Most of pythonisms can be easily emulated in Ruby and vice versa. The only feature of Ruby I have been unable to emulate in Python is instance_eval of an anonymous block – an excellent instrument to make DSLs…

@mystilleef
Sorry for the typo: it should be “I have *not* been able to emulate”

@Loren Segal
Just for the seek if illustration: the closest analog of Ruby for Python’s ‘__getattribute__’ is, probably, #send.
class Object
alias :old_send :send
def send(sym)
#do something with sym, e.g., get a “descriptor” class via sym and handle it
#call old_send
b=old_send sym
#do something with b and return
b
end
end
Therefore, I would like to maintain that the most significant difference between Ruby and Python is the accents the communities put on the language features (i.e., the choice of features for “daily” use) not the features themselves. The latter can be easily emulated.
Another example is re-opening classes. In Python, it’s perfectly possible, but seldom used; in Ruby, re-opening a class is pretty common. In Ruby, customization of the protocol receiver handles a message seems to be easy, but I have not seen an example of it in a real code; in Python, vice versa, customization of the protocol an object interacts with the members of the object dict is truly pythonic, and so on.

@giorgi

I know about the difference between __getattr__ and __getattribute__, the code I posted emulates this fine. The implementation will check if the call is an attribute (instance variable) only– you can continue in the else block for pure __getattr__ behaviour.

Aaaahhh…

If only more such language comparison threads were this civilised. Hats off to you all folks.

What bothers me about the Python philosophy is that it is based on fear. If the language never allowed magical meta-programming, how can anyone be sure it is such a bad thing?

The speculation that such features lead to obfuscated code makes sense in theory, but Ruby has shown that in practice, the opposite is true: a more flexible language allows the programmer to be more *expressive* and the code to communicate its meaning more *concisely*.

Of course, in the hands of a bad programmer, these features can be used to make a mess, as can any other powerful feature. But crippling languages to limit the damage done by incompetents is ultimately self-defeating. It’s much easier to just take away their keyboards.

Reading through the comments, it’s just way too hilarious to see all the defensive comments, repetitive false arguments in favor of python. It’s like Yehuda’s words are falling on deaf ears o-O

Great post.

I think that there is some disingenuousness on both sides of this argument.

If language “A” must implement a feature in C code and language “B” can implement it in language B itself, then yes language “B” is more flexible. In this case the claim is that Ruby is more flexible than Python, which is true if this decorator syntax does what is described. No, it is not “the same thing” to use the inconvenient pre-Python syntax. If that syntax were sufficient then Python would never have added decorators to the language. It is to Ruby’s credit if it can implement the modern Python syntax without changing the Python interpreter.

So that’s the disingenuousness on the Python side. If the blog post does what it does, then it is impressive and to Ruby’s credit.

On the Ruby side, I want to be sure that we are talking about something truly analogous to Pythons decorators.

* When I tried to use this construct in an ActiveRecord model, the model stopped working, even before I added any decorated methods. Is that expected? That does not happen with Python decorators.

* Based on my experience above, I believe that if I added this globally, I would break a lot of Rails or Merb code. You implied in the blog post that it was fairly discretionary whether to add it to every class in the system, but I’d appreciate it if you could be explicit about how safe this is or is not.

* It is not clear to me how I apply this to lambda or “top-level” functions

* It is not clear to me how I decorate attr_accessors. Will it work to just put the decorator before the attr_accessor line?

* Does decoration of class methods “just work” as I’d expect?

* Can I decorate classes? Ruby classes are callable, so they should be decoratable.

Once I understand these issues better I’ll be able to know whether this is truly equivalent to the Python feature or just a neat hack that emulates it in simplistic cases.

Before Python got decorator syntax, the convention for decorating was consistent across functions, methods, anonymous functions and classes. In Python 3 it is again consistent for all of those constructs. (for a while in Python 2.x classes were excluded for no good reason) Therefore I believe that Python’s syntax is much more general purpose than the one you’ve demonstrated here.

Let me re-iterate that, to use decorators, one does not need to reshape Python interpreter; use of DecoratorTools (http://peak.telecommunity.com/DevCenter/DecoratorTools) perfectly suffices:

from peak.util.decorators import decorate

class Demo1(object):
decorate(a_classmethod) # equivalent to @a_classmethod
def example(cls):
print “hello from”, cls

Good job!
Introduce decorators to ruby make sense.

question:
1. UnboundMethod#name is not available in ruby1.8.6(seems 1.8.7 is ok).
That’s annoying especially I want to record decorators’ information as metadata, method name as key.
2. how to chain decorator’s result?
ruby_decorator return the last decorator’s result.
but, unlike extlib#Hook, it doesn’t pass result to chain. sinatra use throw keyword to archieve it.

decorators may be helpful in losts situations, and Python’s @ signal makes sense.

think about alter webframework like sinatra:

Post “/articles”
Param :subject, String
Param :body, Text
Respond :html, :json
def create

end

Acctually, I didn’t see any ruby web framework treat HttpRequest well.
merb-action-args make one step.
but, HttpParamSchema is so common, that reflect HtmlForm generation, URL generation, Http Params acceptance, validation, type-cast, transformation to poro…

need ActionForm or HttpParams come to rescue?

I wrote a similar piece (but backwards) on emulating ruby that you might like:
http://nicolas-lara.blogspot.com/2009/01/emulating-ruby-blocks-in-python.html

I know I’m beating a dead horse here, but thought I’d throw another idea into the ring! After reading this post, I thought, wouldn’t [this][] be a good way to do it? :-)

[this]: https://github.com/colinta/mar

Leave a Reply

Archives

Categories

Meta