Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Threads (in Ruby): Enough Already

For a while now, the Ruby community has become enamored in the latest new hotness, evented programming and Node.js. It’s gone so far that I’ve heard a number of prominent Rubyists saying that JavaScript and Node.js are the only sane way to handle a number of concurrent users.

I should start by saying that I personally love writing evented JavaScript in the browser, and have been giving talks (for years) about using evented JavaScript to sanely organize client-side code. I think that for the browser environment, events are where it’s at. Further, I don’t have any major problem with Node.js or other ways of writing server-side evented code. For instance, if I needed to write a chat server, I would almost certainly write it using Node.js or EventMachine.

However, I’m pretty tired of hearing that threads (and especially Ruby threads) are completely useless, and if you don’t use evented code, you may as well be using a single process per concurrent user. To be fair, this has somewhat been the party line of the Rails team years ago, but Rails has been threadsafe since Rails 2.2, and Rails users have been taking advantage of it for some time.

Before I start, I should be clear that this post is talking about requests that spent a non-tiny amount of their time utilizing the CPU (normal web requests), even if they do spend a fair amount of time in blocking operations (disk IO, database). I am decidedly not talking about situations, like chat servers where requests sit idle for huge amounts of time with tiny amounts of intermittent CPU usage.

Threads and IO Blocking

I’ve heard a common misperception that Ruby inherently “blocks” when doing disk IO or making database queries. In reality, Ruby switches to another thread whenever it needs to block for IO. In other words, if a thread needs to wait, but isn’t using any CPU, Ruby’s built-in methods allow another waiting thread to use the CPU while the original thread waits.

If every one of your web requests uses the CPU for 30% of the time, and waits for IO for the rest of the time, you should be able to serve three requests in parallel, coming close to maxing out your CPU.

Here’s a couple of diagrams. The first shows how people imagine requests work in Ruby, even in threadsafe mode. The second is how an optimal Ruby environment will actually operate. This example is extremely simplified, showing only a few parts of the request, and assuming equal time spent in areas that are not necessarily equal.


Untitled.001.png


Untitled.002.png


I should be clear that Ruby 1.8 spends too much time context-switching between its green threads. However, if you’re not switching between threads extremely often, even Ruby 1.8′s overhead will amount to a small fraction of the total time needed to serve a request. A lot of the threading benchmarks you’ll see are testing pathological cases involve huge amounts of threads, not very similar to the profile of a web server.

(if you’re thinking that there are caveats to my “optimal Ruby environment”, keep reading)

“Threads are just HARD”

Another common gripe that pushes people to evented programming is that working with threads is just too hard. Working hard to avoid sharing state and using locks where necessary is just too tricky for the average web developer, the argument goes.

I agree with this argument in the general case. Web development, on the other hand, has an extremely clean concurrency primitive: the request. In a threadsafe Rails application, the framework manages threads and uses an environment hash (one per request) to store state. When you work inside a Rails controller, you’re working inside an object that is inherently unshared. When you instantiate a new instance of an ActiveRecord model inside the controller, it is rooted to that controller, and is therefore not used between live threads.

It is, of course, possible to use global state, but the vast majority of normal, day-to-day Rails programming (and for that matter, programming in any web framework in any language with a request model) is inherently threadsafe. This means that Ruby will transparently handle switching back and forth between active requests when you do something blocking (file, database, or memcache access, for instance), and you don’t need to personally manage the problems the arise when doing concurrent programming.

This is significantly less true about applications, like chat servers, that keep open a huge number of requests. In those cases, a lot of the application logic happens outside the individual request, so you need to personally manage shared state.

Historical Ruby Issues

What I’ve been talking about so far is how stock Ruby ought to operate. Unfortunately, a group of things have historically conspired to make Ruby’s concurrency story look much worse than it actually ought to be.

Most obviously, early versions of Rails were not threadsafe. As a result, all Rails users were operating with a mutex around the entire request, forcing Rails to behave like the first “Imagined” diagram above. Annoyingly, Mongrel, the most common Ruby web server for a few years, hardcoded this mutex into its Rails handler. As a result, if you spun up Rails in “threadsafe” mode a year ago using Mongrel, you would have gotten exactly zero concurrency. Also, even in threadsafe mode (when not using the built-in Rails support) Mongrel spins up a new thread for every request, not exactly optimal.

Second, the most common database driver, mysql is a very poorly behaved C extension. While built-in I/O (file or pipe access) correctly alerts Ruby to switch to another thread when it hits a blocking region, other C extensions don’t always do so. For safety, Ruby does not allow a context switch while in C code unless the C code explicitly tells the VM that it’s ok to do so.

All of the Data Objects drivers, which we built for DataMapper, correctly cause a context switch when entering a blocking area of their C code. The mysqlplus gem, released in March 2009, was designed to be a drop-in replacement for the mysql gem, but fix this problem. The new mysql2 gem, written by Brian Lopez, is a drop-in replacement for the old gem, also correctly handles encodings in Ruby 1.9, and is the new default MySQL driver in Rails.

Because Rails shipped with the (broken) mysql gem by default, even people running on working web servers (i.e. not mongrel) in threadsafe mode would have seen a large amount of their potential concurrency eaten away because their database driver wasn’t alerting Ruby that concurrent operation was possible. With mysql2 as the default, people should see real gains on threadsafe Rails applications.

A lot of people talk about the GIL (global interpreter lock) in Ruby 1.9 as a death knell for concurrency. For the uninitiated, the GIL disallows multiple CPU cores from running Ruby code simultaneously. That does mean that you’ll need one Ruby process (or thereabouts) per CPU core, but it also means that if your multithreaded code is running correctly, you should need only one process per CPU core. I’ve heard tales of six or more processes per core. Since it’s possible to fully utilize a CPU with a single process (even in Ruby 1.8), these applications could get a 4-6x improvement in RAM usage (depending on context-switching overhead) by switching to threadsafe mode and using modern drivers for blocking operations.

JRuby, Ruby 1.9 and Rubinius, and the Future

Finally, JRuby already runs without a global interpreter lock, allowing your code to run in true parallel, and to fully utilize all available CPUs with a single JRuby process. A future version of Rubinius will likely ship without a GIL (the work has already begun), also opening the door to utilizing all CPUs with a single Ruby process.

And all modern Ruby VMs that run Rails (Ruby 1.9′s YARV, Rubinius, and JRuby) use native threads, eliminating the annoying tax that you need to pay for using threads in Ruby 1.8. Again, though, since that tax is small relative to the time for your requests, you’d likely see a non-trivial improvement in latency in applications that spend time in the database layer.

To be honest, a big part of the reason for the poor practical concurrency story in Ruby has been that the Rails project didn’t take it seriously, which it difficult to get traction for efforts to fix a part of the problem (like the mysql driver).

We took concurrency very seriously in the Merb project, leading to the development of proper database drivers for DataMapper (Merb’s ORM), and a top-to-bottom understanding of parts of the stack that could run in parallel (even on Ruby 1.8), but which weren’t. Rails 3 doesn’t bring anything new to the threadsafety of Rails itself (Rails 2.3 was threadsafe too), but by making the mysql2 driver the default, we have eliminated a large barrier to Rails applications performing well in threadsafe mode without any additional research.

UPDATE: It’s worth pointing to Charlie Nutter’s 2008 threadsafety post, where he talked about how he expected threadsafe Rails would impact the landscape. Unfortunately, the blocking MySQL driver held back some of the promise of the improvement for the vast majority of Rails users.

25 Responses to “Threads (in Ruby): Enough Already”

The removal of Rubinius’ GIL is already underway, in the hydra branch:

http://github.com/evanphx/rubinius/tree/hydra

@chris post updated!

What about PostgreSQL? Does the pg gem let Ruby context switch while it’s talking to the DB? If not, do any other Postgres drivers do it right?

the pg gem using rb_thread_select, which works similarly to POSIX’s select, but is Ruby-aware. So yes, the pg driver does let Ruby know that it’s ok to run something else while it’s waiting to get its response from the Postgres DB, making it properly thread-aware.

Very nicely explained post – the diagrams make it easy to understand too!

How about the psql driver?

Excellent post.
I must confess that i’ve not even tried the thread safe mode of Rails 2.2+ yet because of massive “FUD” spread in the community.

For anyone who had questions about the mysql2 gem (When it became the default Rails mysql handler? How to use it in your current rails project? etc). Read this stack overflow post.
http://stackoverflow.com/questions/3001243/ruby-rails-mysql2-gem-does-somebody-use-this-gem-is-it-stable

Great Post Yehuda.
I think this explains a problem I was having with Passenger a few months ago. Every now and then I would find Passenger was completely unresponsive because its threads were all frozen.

In debugging I found that they would get stuck on a mysql server I had hosted on an unreliable shared host (was too lazy) … so everytime the server went down or got unresponsive the threads would freeze instead of returning 500 errors.

Your explanation for the mysql being a badly behaved C extension immediately set off a lightbulb in my head. Thanks!

Hey Yehuda,

You kind of have a nasty habit of rewriting history and it sort of borders on slander. I’m really big on guys like you saying my code sucked when it didn’t, so let’s just address your assertions right here.

I suggest you change your history of Mongrel and Rails history. The truth is that Rails wasn’t thread safe until much much later. In fact, DHH and all of rails core crew swore it could never be made thread safe. Yes, they actually said it could *never* be made thread safe. The only recourse was to have Mongrel wrap all of rails in a mutex. Notice how none of the other web frameworks had this mutex, you know like Merb? Merb had to come along and prove you could make a “thread safe rails” before they even thought about it being possible. Mongrel was just trying to protect poor people who had to try to run Rails and keep it from crashing.

I guess that’s why 37Signals got with Engine Yard and forced the Rails and Merb projects to merge (unlike the claim that it was a whole meeting of the minds). That way rails could finally be thread safe.

And BTW, Mongrel worked perfectly fine. Your whole (i.e. not mongrel) is a load of crap.

Zed

Zed has replied to the Mongrel mutex bit:
http://dpaste.de/5xyG/raw/

Hey Zed,

I don’t disagree with your comment, except for one thing: Rails was threadsafe in 2.2, well after Merb was in full swing, but also well before the merge.

> Hey Yehuda,

> You kind of have a nasty habit of
> rewriting history and it sort of
> borders on slander. I’m really big on
> guys like you saying my code sucked
> when it didn’t, so let’s just address
> your assertions right here.

Mongrel is a great Ruby web server that we used to good effect in Merb and I don’t have any gripes with it. I don’t even disagree with your decision to wrap a lock around requests in mongrel_rails.

> I suggest you change your history of
> Mongrel and Rails history. The truth
> is that Rails wasn’t thread safe until
> much much later.

I’m sorry if I was unclear in my post. I specifically referenced Rails 2.2 and the attitude of the core team toward threadsafe Rails.

> In fact, DHH and all of rails core crew
> swore it could never be made thread
> safe. Yes, they actually said it could
> *never* be made thread safe.

I can totally believe this, as it was a source of much frustration in the Merb days. We had to put up with a lot of arguments that threads were basically useless, especially in 1.8. Parts of the argument in this post are arguments we made on behalf of Merb being threadsafe by default.

> The only recourse was to have
> Mongrel wrap all of rails in a mutex.

I completely agree that this was the correct course of action prior to Rails 2.2. The problem is that Mongrel 1.1.5 (the last stable release of mongrel, even until today) was released in May 2008, while Rails 2.2 was released in November 2008.

In other words, my comments were *not* a personal attack against you; you (and other mongrel maintainers) personally did the right thing at the time of the release of every version of mongrel that was ever publicly released.

> Notice how none of the other web
> frameworks had this mutex, you
> know like Merb? Merb had to come
> along and prove you could make a
> “thread safe rails” before they even
> thought about it being possible.

Yep. Again, this FUD was a source of frustration for us back in 2008.

> Mongrel was just trying to protect
> poor people who had to try to run
> Rails and keep it from crashing.

And you did the right thing.

> I guess that’s why 37Signals got
> with Engine Yard and forced the
> Rails and Merb projects to merge
> (unlike the claim that it was a whole
> meeting of the minds).

I can tell you with a clear conscience that the merge was not a plot between Engine Yard and 37 Signals. We spent significant amounts of time in technical discussions (with the core team, not 37S) before everyone got on board, and the core team was frankly skeptical at first.

> That way rails could finally be
> thread safe.

This is just not true. Rails became threadsafe in November 2008, after a Google Summer of Code project in Summer 2008.

> And BTW, Mongrel worked perfectly
> fine. Your whole (i.e. not mongrel) is a load of crap.

I apologize for the wording; I was (in context) specifically referencing the default behavior of mongrel_rails, which I had covered earlier in the post. As I said earlier, mongrel itself worked just fine, but the default Rails handler (in the latest release) hardcoded non-threaded behavior.

It was the right decision at the time, but it did have an impact on people’s perception of Rails’ threaded performance.

> Zed

– Yehuda

Yehuda, thanks for this post. When you say that Mysql2 is the new default MySQL driver for Rails, does that mean that with gem install rails –pre, I would expect to see the mysql2gem installed as one of the dependencies? (MySQL2 doesn’t install with Rails3 for me).

Beyond that, my takeaway is: a Rails2.3+ app running on Ruby 1.9.x A) should have as many processes spun up as you have cores (e.g., four Ruby processes on a quad-core box); B) will enjoy non-blocking I/O with MySQL as long as you have the mysql2 gem installed. Do have I that mostly right?

> Yehuda, thanks for this post. When you say that Mysql2 is the new default MySQL driver for Rails, does that mean that with gem install rails –pre, I would expect to see the mysql2gem installed as one of the dependencies? (MySQL2 doesn’t install with Rails3 for me).

Even if it doesn’t, just add the gem to gemfile.

MySQL2 doesn’t install (I understand this as “present in Gemfile”) as default because the mysql2 driver (rails -d mysql adds the “old” mysql gem to Gemfile) is still under heavy development and not considered fully stable, for instance only recently it has solved the issue of losing DB connection:
http://github.com/brianmario/mysql2/issues/issue/31

I’m interested in answer to the “takeaway” question as well. How many Unicorn processes should I now spawn having Rails3 app with mysql2 gem?

Great article, thanks.

One remark “Threads are just HARD”: Unfortunately it’s not only the framework (Rails) which needs to be threadsafe, all the myriad of gems and plugins need to be threadsafe as well. And I’ve seen too many plugins just using class variables (for instance) in a non threadsafe manner. I have the feeling that threadsafety still just isn’t something the majority of the Ruby community is concerned about. Just look at the small number of threadafe plugins at railsplugins.org:

http://railsplugins.org/plugins?criteria=2

I never would dare to turn on the threadsafe! option in a Rails project without a thorough review of all the libraries I’m using. Right now that is simply not worth it for me.

About servers: Which servers do actually support threadsafe operation?

In his post ‘Rails Performance Needs an Overhaul’ http://www.igvita.com/2010/06/07/rails-performance-needs-an-overhaul Ilya Grigorik talks about Fibers vs Threads. What’s your take on these ideas ?

So mongrel and thin don’t currently play nice with threasafe rails 2.3.x?

Does passenger? If I don’t like passenger, is there an app server that does work well with threadsafe 2.3.x?

Hey Yehuda, thanks so much for writing this post. I was so tired of hearing all the nonsense arguments against threads. Can you can comment on the status of the old autoload/require/constant issue from this thread? http://www.ruby-forum.com/topic/172385

Yehuda, I’m really excited to read this. If I was to summarize all of my thoughts on this post in one statement then its: “Enough already” is not enough – we need many, many more conversations like this one. And the fact that we are, finally having it, is exciting to me.

First, let’s take Threads and events off the table – to me, that’s not the actual meat of the discussion. If I’ve advocated for event driven style of programming in the past, then its much more so by necessity, than anything else. Both models can coexist peacefully, and in fact, require each other – but that’s a matter for a different post & discussion.

The more interesting part of this conversation are the “historical ruby issues” you mentioned. The vast majority of Rubyists run MRI, so while JRuby is a great VM with no GIL, I always make a point of talking about parallelism as something we need to pay *very careful* attention to. Problem is, really, we haven’t been for the most part. As you pointed out yourself, defaults matter, and when the defaults result in the “imaginary” diagram you drew – then, well, it’s not really imaginary is it?

Thing is, all of this is addressable, but it requires a level of vigilance and attention that we just haven’t had in the community as a whole. My belief is: Rails can do it, and by doing so, shift the entire Ruby community with it (because it *does* carry such a big influence).

So, from my perspective (and I admit to frequently using my soapbox to advocate for this), I think parallelism, concurrency & good defaults need to be first class citizens in the priority list for Rails, and every other Ruby project.

P.S. I should say, we’ve made major leaps on all of these fronts recently (Rails 3 addressed a lot of issues in one swoop), and I love the fact that as a community we’re finally talking about it, and more importantly, doing something about it. We just need more of it. Concurrency & parralelism is not a bug you can close once and for all, this is something you need to always keep in the back of your head.

A couple notes on threadsafe Rails and JRuby.

1. We have started recommending that anyone running Rails on JRuby try to run in threadsafe mode. JRuby + threadsafe Rails is by far the best way to run moderate to large sites; you can have a single JRuby process handle all concurrent requests and saturate even large machines. Threadsafety is therefore very important to us, and we’ve been fighting the good fight for threadsafe Rails and Ruby libraries for years.

2. All implementations are trending toward concurrent threads. After JRuby, IronRuby was probably the first implementation that could handle concurrent Rails requests, with MacRuby (already concurrent, but a little tweaky…and no Rails yet) and Rubinius (working on concurrency, and Rails works now) following. It’s only a matter of time before Ruby 1.9 (or 2.0?) also bites the bullet and goes to concurrent native threads, so effort spent now to make things threadsafe will pay massive dividends in the future.

3. Threadsafety is not that hard. Don’t share data across threads – recreated it per request, or cache it only in thread-local storage with appropriate aging out. If you must share data across threads, only share immutable data. Freeze those arrays or hashes before stuffing them into constants or globals, and you’re golden. If you must share mutable data, just wrap them in mutex.synchronize calls and keep the synchronized body of code as light and simple as possible (like swapping an existing Array for the new copy with modifications, a la copy-on-write). Follow some simple rules and the vast majority of threading pain just disappears.

This really is the way the world works; threads are not going to disappear in favor of fully evented systems, nor are they going to disappear in favor of only ever using heavyweight processes. It’s time the Ruby community got together to form better threaded coding standards for Ruby and better utilities for doing so safely. And I’m standing by to help in any way I can :)

Thanks for describing the cooperative design of Ruby 1.8′s green threads and how that actually does help applications use the CPU. I’ve been getting tired of people throwing green threads under the bus unequivocally.

Great post. Here’s a summary of what I’ve learnt from this and other articles:

Ways to deploy a Rails app to make efficient use of servers
———————————————————–

- Wasteful memory usage reduces requests handled per second per dollar by using more RAM and by reducing CPU performance due to cache contention. (Exception: Some memory duplication can be favourable on servers with a non-uniform memory architecture.)

- A simple Rails deployment (e.g. Webrick, Mongrel-1, unthreaded Thin) requires one Rails handler operating system process for each concurrently-handled web request. Inefficient.

Better solutions:

1. Phusion Passenger + Ruby Enterprise Edition:

Allows part of the memory of a set of Rails handler processes to be shared between them, plus dynamically changes the number of processes according to load.

Disadvantages:
– To share memory the entire app must be loaded before forking (no load-on-demand).
– There’s still a great deal of un-shared memory.
– Processes are blocked when doing disk and network I/O (a patch to allow Rails processes under Passenger to run multi-threaded has been created, but is not currently in the main Passenger distribution).

2. Use async-event-loop programming (made more natural by using Ruby Fibres) to allow single-threaded handler processes to work on other requests while a request is blocked on I/O.

Disadvantages:
– Need Ruby 1.9+, an EventMachine-based web-server, and fibre-aware I/O components.
– To properly use all the server CPUs, still need a separate process for each logical CPU (CPU-core + SMT shard), wasting memory, and overloading CPU caches.
– Programming is more difficult, especially exceptions.

3. Enable Rails multi-threading (ensure all gems and plugins used are thread-safe), use a non-blocking database adaptor (e.g. mysql2), and use a webserver that can start requests in a new thread, while still receiving & parsing requests, and sending responses, concurrently in its own thread (e.g. thin –threaded, Rainbows!, and Mongrel-2). Saves much memory by only needing at most one process per logical CPU (less any allocated to DB servers and front-end web servers) — A little less than one for Ruby 1.8 due to the concurrency possible in reading requests and sending responses, even less in Ruby 1.9+ because of the concurrency in I/O code, and only a single process for JRuby. Extra processes can be used to create redundancy, speed work on NUMA architectures, and avoid Ruby thread limits.

Disadvantages:
– Ruby 1.8 thread-switching is reasonably slow and memory heavy.
– Ruby (except JRuby) cannot interpret code concurrently in the same process (other threads must be doing I/O).
– Non-Ruby extension code and OS kernel I/O code only run multi-threaded in Ruby 1.9+.
– Entire app must load on start to avoid non-thread-safe Ruby autoloading of modules and classes.

Best is #3?
– JRuby, Ruby 1.9, or Ruby 1.8 Enterprise Edition (for the faster memory allocator).
– Threadsafe Rails app + multi-threading web-server.
– Non-blocking database adaptor.

Benchmarks?

Nice article, thanks Yehuda. I’m just wondering how threadsafe Rails are, if every time I spawn new thread to do some background processing (safely, nothing shared), I can’t even use Rails.logger ?

The problem with this article,even though I think it’s relatively useful to a lot of people, is it’s based more on theory then on practice.

In practice, threading is hard at higher concurrencies. Any rails app can run in multithreaded mode without much work at low concurrencies. At higher concurrencies, it all changes.

We run jruby + rails + tomcat on around 100 servers doing right at 700 rps each. These are complicated cpu intensive apps that also make 4-6 http calls to backend services per request, plus scribe, rsyslog, and memcache calls. The difference between 200 and 700 rps was almost entirely from thread contention. There were bugs in various gems, bugs in jruby, and a whole lot of side effects that made rails non performant in jruby until they were fixed.

What happens at around this point is that you get capped. It’s really difficult to get more concurrency because of the thread synchronization that goes on in tomcat and java itself. blocking io starts chewing up most of your cpu, and context switching really kicks in. Your rps curve pretty much flattens out at this point, and the effort required to get more rps is not worth it.

If we had used something like eventmachine as the base for the app, with all non blocking io, I’m pretty sure we could have more then tripled our rps without having to tune much of anything.

As for running multithreaded under cruby, that’s a bad joke. Even at fairly low concurrencies you have terrible performance with very high standard deviations on request times due to how the GIL works. Languages built on the GIL just have bad threading performance at even moderate concurrencies, it’s just the way it is.

Chris

One more thing… Another issue with running large numbers of threads is your connections to backend services. Say you are making http or redis calls. You need one connection per thread, because in most cases 80% or more of your threads are going to be in those network calls all at the same time, so sharing them in any way between threads leads to too much lock contention. Now multiple 400 threads by say 1000 servers. That’s 40,000 networks connections to the lb for your backend service.

And most load balancers only support multiplexing http connections. If you want to connect to redis or mysql, you get to roll your own multiplexing load balancing solution in most cases. (mysql has some options, none of them just work out of the box)

With evented IO, you can usually just use one connection per cpu/core, if that.

Leave a Reply

Archives

Categories

Meta