August Tokaido Update

It's been a while since I posted anything on my blog, and I figured I'd catch everyone up on the work I've been doing on Tokaido.

Components

Tokaido itself is made up of a number of components, which I am working on in parallel:

• Ruby binary build, statically compiled
• A logging and alerting UI
• Remote Notifications for Rails 3+
• Integration with Puma Express
• Integration with code quality tools
• Resolve bundler issue related to binary builds and deployment to Heroku
• Work with community members to start shipping binary builds of popular gems (gates on fixing the bundler bug)

A number of people are doing parts of the work to make Tokaido a reality. I specifically want to thank:

• Michal Papis of the rvm team for using the sm framework to make the Tokaido build maintainable over time and doing the heavy lifting to take the initial spike I did and get a reproducible binary build
• Terence Lee of Heroku for packaging up these early binary builds for use at several Rails Girls events, with great success!

Ruby Binary Build, Statically Compiled

This is the first work I did, a few months ago, with help from Michal Papis. I detailed the hard parts of making Ruby statically compiled in June's status update. Since then, Terence Lee (@hone02) has used the binary build at several Rails Girls events, dramatically reducing the time needed for Rails installation on OSX. In addition to the Tokaido binary build, Terence also precompiled a number of gems, and put built a script to download the entire thing as a zip, install it into the user's home directory, and modify their ~/.profile to include it.

This strategy works well for trainings, but has a number of limitations:

• Since it relies on precompiled gems already existing in the gem home, the gems cannot be removed or upgraded without requiring a C compiler. The correct solution is what gems authors do for Windows: ship versions of binary gems with specific OSX designations. Currently, binary gems do not interact well with Heroku deploys (or anyone using bundle install --deployment), so we have been working to resolve those issues before foisting a bunch of new binary gems on the world. See below.
• It relies on modifying all instances of the Terminal, which means that some system edge-cases leak into this solution. It also relies on modifying ~/.profile, which may not work depending on what shell the user is using and what other startup scripts the user is running. For Tokaido, we will have a way to launch a Terminal that reliably injects itself without these problems, and without polluting all instances of Terminal.
• It is difficult to upgrade components, like patch levels of Ruby or versions of C dependencies like libyaml. Tokaido.app will store its copy of Ruby and gems in a sandbox, loadable from the UI (as I described in the second bullet), which makes it easy for the .app to upgrade patch levels of Ruby or whatever components it needs.

Because I understand that many people WANT the ability to have their Ruby take over their terminal, Tokaido will integrate with rvm (and rbenv, if possible) to mount Tokaido.app as the default version of Ruby in your shell.

A Logging and Alerting UI

The primary UI for Tokaido will be a logging and alerting UI for your Rails application. I expect that you will spend the most time in the "Requests" UI, which will show you a list of the previous requests, and the log of the current request. My goal is to use the logging UI to improve on the typical "tail the log" experience. I've been working with Austin Bales, a designer at do.com to develop some mockups, and hope to have something for people to look at in the next few weeks.

In addition to an improved logging experience, Tokaido will alert you when something has gone wrong in your request. This may include exceptions (5xx) or information provided by plugins (bullet can let you know when your code is triggering an N+1 query; see the README for more information). The list of prior requests will highlight requests with issues, and problems in the notifications tray will directly link you to the request where the error occurred.

If everything goes well, the Tokaido UI will replace your logging workflow, without impacting the rest of the tasks you perform from the command-line.

Remote Notifications for Rails 3+

Rails 3 shipped with a new instrumentation API, which provides detailed information about many events that happen inside of Rails. This system is used by Rails' own logging system, and I would like to use it in Tokaido as well.

The instrumentation API provides all of the information we need, but no built-in way to communicate those notifications across processes. Because Tokaido.app will not run in the same process as the Rails app, I have been working on a gem (remote_notifications) that provides a standard way to send these notifications to another process.

This gem also backports Aaron's Rails instrumentation work (see his talk at Railsberry for more information) to Rails 3.0 and 3.1, which makes it possible to build a reliable tree from notifications, even on systems with low clock resolution.

It also includes a mechanism for subscribing to notifications sent from another process using the regular notifications API, and pluggable serializers and deserializers. It isn't quite done yet, but I should have an initial release soon.

Integration with Puma Express

Puma is a new threaded web server built by Evan Phoenix of Rubinius fame. Puma Express manages Puma servers, automatically setting up the DNS (appname.dev) for you. This is similar to the approach used by Pow, but without a Node dependency.

Tokaido will integrate with Puma Express, so you will not need to manually boot and shut down your server. Because Tokaido also takes care of logging, there shouldn't be any need for dedicated tabs with persistent running processes when you use Tokaido.

You will also be able to install an executable into your system (a la GitX and Textmate) to make it easy to boot up a Tokaido in the context of the current application:

$ tokaido .

This will be especially useful for people using the rvm "mounted" Tokaido.

Integration with Code Quality Tools

By default, Tokaido will integrate with flog, flay and rails-best-practices. It will periodically scan your Rails app for problems and notify you via the app's notification tray. This mechanism will be extensible, so a future version of Tokaido might integrate with Code Climate or support code quality plugins.

Binary Gems and Heroku

Bundler provides a mechanism for deployment (bundle install --deployment) that specifically rejects deploys that require changes to the Gemfile.lock. When deploying a Gemfile.lock that was built using binary gems from a different platform, this mechanism rejects the deploy. At present, only Windows makes heavy use of binary gems, and Heroku's solution to date has been to simply remove the Gemfile.lock and re-resolve dependencies.

Unfortunately, this solution eliminates the most important guarantee made by bundler, and is untenable in the long-term. If OS X users started to use binary gems more broadly, this would bring this problem into much wider circulation. Also, because platform-specific gems can contain alternate dependencies (see, for example Nokogiri 1.4.2), it is important that bundle install support alternate platforms more extensively than with a naïve solution.

There are a few possible solutions:

• Do not use --deployment on Heroku. This would allow bundler to update the Gemfile.lock for the new platform. It would also mean that if a developer didn't update their Gemfile.lock before deploying, they might run unexpected code. This is a short-term fix, but would probably alleviate most of the symptoms without introducing a lot of new caveats.
• Fix --deployment to allow changes to the Gemfile.lock, but only in response to a genuine platform change. Unfortunately, because of cases like Nokogiri 1.4.2, this could theoretically result in totally different code running in production.
• Provide a mechanism for developers to explicitly add a deployment environment: bundle platform add x86_64-linux. This would pre-resolve gems across all available platforms and ensure that the same gem versions could be used everywhere.
• Improve the bundler resolution mechanism to allow gems with the same name, version and dependencies to be treated the same. Because dependencies can also exhibit this problem (a dependency of some JRuby gem could be another JRuby gem with its own non-standard dependencies), this would need to ensure that the entire subtree was the same.

The last solution is most promising, because the platform-specific gems Tokaido is concerned with are simply precompiled variants of the same gem. In the vast majority of cases, this simply means that the gem author is saving the end-developer the step of using a C compiler, but doesn't actually change a lot else.

Unfortunately, Rubygems itself doesn't distinguish between precompiled variants of gems and gems with different dependencies, so we will have to add smarts into bundler to detect the difference. My guess is that we will use a short-term fix like the first option while working on a longer-term fix like the last option.

Work Time

As I said in the original Kickstarter, I planned to take time off from work to work on the project. I have structured that time by working mornings on Tokaido, and shifting client work to the afternoons and evenings.

I don't have a specific ship-date in mind for Tokaido yet, but you should start seeing more work-product as the project progresses, starting over the next few weeks.

Thanks for your patience. It shall be rewarded!