Yehuda Katz is a member of the Ember.js, Ruby on Rails and jQuery Core Teams; his 9-to-5 home is at the startup he founded, Tilde Inc.. There he works on Skylight, the smart profiler for Rails, and does Ember.js consulting. He is best known for his open source work, which also includes Thor and Handlebars. He travels the world doing open source evangelism and web standards work.

Using .gemspecs as Intended

When you clone a repository containing a Unix tool (or download a tarball), there’s a standard way to install it. This is expected to work without any other dependencies, on all machines where the tool is supported.

$ autoconf
$ ./configure
$ make
$ sudo make install

This provides a standard way to download, build and install Unix tools. In Ruby, we have a similar (little-known) standard:

$ gem build gem_name.gemspec
$ gem install gem_name-version.gem

If you opt-into this convention, not only will it simplify the install process for your users, but it will make it possible for bundler (and other future automated tools) to build and install your gem (including binaries, proper load path handling and compilation of C extensions) from a local path or git repository.

What to Do

Create a .gemspec in the root of your repository and check it in.

Feel free to use dynamic code in here. When your gem is built, Rubygems will run that code and create a static representation. This means it’s fine to pull your gem’s version or other shared details out of your library itself. Do not, however, use other libraries or dependencies.

You can also use Dir[] in your .gemspec to get a list of files (and remove files you don’t want with -; see the example below).

Here’s bundler’s .gemspec:

# -*- encoding: utf-8 -*-
lib = File.expand_path('../lib/', __FILE__)
$:.unshift lib unless $:.include?(lib)
 
require 'bundler/version'
 
Gem::Specification.new do |s|
  s.name        = "bundler"
  s.version     = Bundler::VERSION
  s.platform    = Gem::Platform::RUBY
  s.authors     = ["Carl Lerche", "Yehuda Katz", "André Arko"]
  s.email       = ["carlhuda@engineyard.com"]
  s.homepage    = "http://github.com/carlhuda/bundler"
  s.summary     = "The best way to manage your application's dependencies"
  s.description = "Bundler manages an application's dependencies through its entire life, across many machines, systematically and repeatably"
 
  s.required_rubygems_version = ">= 1.3.6"
  s.rubyforge_project         = "bundler"
 
  s.add_development_dependency "rspec"
 
  s.files        = Dir.glob("{bin,lib}/**/*") + %w(LICENSE README.md ROADMAP.md CHANGELOG.md)
  s.executables  = ['bundle']
  s.require_path = 'lib'
end

If you didn’t already know this, the DSL for gem specifications is already pretty clean and straight-forward, there is no need to generate your gemspec using alternative tools.

Your gemspec should run standalone, ideally with no additional dependencies. You can assume its __FILE__ is located in the root of your project.

When it comes time to build your gem, use gem build.

$ gem build bundler.gemspec

This will spit out a .gem file properly named with a static version of the gem specification inside, having resolved things like Dir.glob calls and the version you might have pulled in from your library.

Next, you can push your gem to Rubygems.org quickly and painlessly:

$ gem push bundler-0.9.15.gem

If you’ve already provided credentials, you’ve now published your gem. If not, you will be asked for your credentials (once per machine).

You can easily automate this process using Rake:

$LOAD_PATH.unshift File.expand_path("../lib", __FILE__)
require "bundler/version"
 
task :build do
  system "gem build bundler.gemspec"
end
 
task :release => :build do
  system "gem push bundler-#{Bunder::VERSION}"
end

Using tools that are built into Ruby and Rubygems creates a more streamlined, conventional experience for all involved. Instead of trying to figure out what command to run to create a gem, expect to be able to run gem build mygem.gemspec.

A nice side-effect of this is that those who check in valid .gemspec files can take advantage of tools like bundler that allow git repositories to stand in for gems. By using the gem build convention, bundler is able to generate binaries and compile C extensions from local paths or git repositories in a conventional, repeatable way.

Try it. You’ll like it.

33 Responses to “Using .gemspecs as Intended”

Thank you for this post. I was saying to a friend the other day that there wasn’t a good write-up on the process of making a ruby gem.

I completely agree – some the completely extraneous build tools drive me batty. gemspec’s aren’t that complicated or verbose. Either way, though, there should be a physical .gemspec in the gem, wether it is generated or hand-written.

You point out that it lets tools like bundler do their thing, but it would also allow possible extending of rubygems itself for functionality like pointing `gem install` at a gemspec directly (or even an online gemspec – of course that might need a full manifest to find all the files) and having the gem command handle all the intermediary steps in /tmp (for instance). Even for people who don’t want that, it’s just one instance of what becomes possibly if gem authors follow the standard of having a real gemspec checked into their project and/or included in the gem (included in the gem is nice for being able to rebuild any ext for an installed gem, directly).

As always, wycats deftly removes the best-practice scales from our eyes. Nice!

thanks for this post. i was hoping to see something like this, since the “.gitignore your *.gemspec” article is just wrong. educate contributors submitting patches that change your gemspec. also i don’t see any reason to still use jeweler. it’s just so easy to maintain a gemspec and publish your gems on gemcutter.

I’m confused. I had learned all along that gem installation only needed one command: gem install gem_name. When did this become two commands, and why? It appears the rest of the article is about how to make gems that fit the two command approach. (I sorta like the old one command approach better unless I feel I’m getting something valuable out of the extra typing.)

I like to use a rake task to generate the gemspec. It allows me to use the project’s Gemfile to fill in the gem dependencies in the spec. Does this mean making sure you regenerate the gemspec each time anything changes in your Gemfile? This is an unfortunate duplication of gem dependencies and it could easily get out of sync, but it’s presently the only way to use bundler to create the gemspec.

Is this the best way? Is it just “keep them in sync so people don’t have to type $ rake gemspec or equivalent”? It’s much less common to want to download and immediately install a gem from source. It’s much more likely you want to work on it (bundle install) or just get the latest release version (gem install gemname). The gemspec just seems like an extra thing you only generate when you need to bundle with an unreleased gem or when you’re ready to release the actual gem (which is usually another rake task anyway).

It would be nice if the gemspec automatically use a Gemfile but I know this means a rubygems upgrade and so will be much harder to make happen.

Martin Emde couldn’t be more right!

Step one of Yehuda’s instructions (“Create a .gemspec in the root of your repository and check it in.”) is completely broken. You shouldn’t be checking in generated files to your repro. That goes against _every_ SCM best practice out there.

Another thing that is dead wrong is Yehuda’s myopic fixation on dirglobs. He’s allergic to manifests for some irrational reason (direct quote: “They’re too hard to maintain”) and yet he won’t acknowledge that his own example has flaws:

s.files = Dir.glob(“{bin,lib}/**/*”) + %w(LICENSE README.md ROADMAP.md CHANGELOG.md)

Tell me how to keep _every_ editor’s temp and backup files out of there.

Tell me how this avoids crap like OSX’s .DS_Files.

Tell me how this avoids _every_ SCM’s dotfiles.

Tell me how this avoids shipping things you’re not ready to ship.

It doesn’t.

**It can’t**.

All of these complications are further compounded by the fact that Yehuda wrote this out of frustration caused by a ridiculous feature for bundler: installing directly via git repo. Who says that the git repo you’re going to install from has a policy of always checking in working code? It is a failure prone idea to begin with.

I would have to disagree with you Yehuda on this.

For some gems, the scenario you’re presenting will work, for others, it wouldn’t.

Let’s ignore for a second that bundler doesn’t like gems with native extensions or specific platform gems that got bundled, let’s talk about the gemspec.

Gem::Specification get serialized inside the gem as YAML, and I’ve seen lot of people using things like Rake::FileList to build gemspecs, which fails. A concrete example can be found here:

http://github.com/technicalpickles/jeweler/issuesearch?state=open&q=rake-compiler#issue/73

Also, gem build will not take in consideration special and external dependencies your gem might require that are not bundled with a simple gem build but that are pulled and generated when you do a “rake gem” command.

For example, a ragel parser based extension.

The whole idea around RubyGems and solutions like GemCutter are to provide an unified approach to gem installation.

Suggesting usage of gem build and gem install and comparing them with unix tools means that my gem pulled from your GitHub repository will not be the same gem as John pulled from your repository two weeks ago, even they use the same version.

I believe that rely on gemspec to actually track dependencies and having the gemspec in your code repository is not a good approach at all.

But, I might be wrong.

One small point – I don’t understand why the convention is to name the gemspec file GEMNAME.gemspec. The gemspec already contains the name of the gem.

If you just called it gemspec (or Gemspec if you were feeling adventurous), you can create a completely generic set of rake tasks such as http://github.com/markryall/gemesis/blob/master/lib/gemesis/rake.rb

Generating gemfiles seems like overkill to me – editing the version number with each release using a text editor isn’t so onerous.

I don’t quite understand why you say “Do not, however, use other libraries or dependencies.”. Do you mean you should make them explicit in the gemspec or a Gemfile?

By “Generating gemfiles”, I meant “Generating gemspec files”.

Thanks for post Yehuda. I wrote how to work with gemspecs a while ago: http://blog.101ideas.cz/post/353002256/the-truth-about-gemspecs

@Ryan Davis: i think “Create a .gemspec in the root of your repository and check it in.” refers to not using jeweler but creating a gemspec yourself. so it’s not a generated file.

I wholeheartly agree.

As for the “easily automate this process using rake” hint I hate rake and tend to avoid it if possible. I’ve been using a rubygems plugin that does the same thing and put it up to github after reading this post: http://github.com/svenfuchs/gem-release

I may be old fashioned, but I still like hoe and its manifests and rake…

@Sven…what is it you dislike so much about rake? Not saying you are wrong…or right…just curious.

Hey, I thought Ruby’s similar (little-known) standard was:

ruby setup.rb

:-)

I suspect the proliferation of tools to generate gemspecs was a direct result of http://gems.github.com/ which would use a gemspec to generate a gem, but was sandboxed so you couldn’t have much in the way of logic (particularly anything that touched other files) in there.

Ryan, I don’t think Yehuda is advocating checking in a generated file (though I know db/schema.rb does!), he’s talking about maintaining the file by hand. Which isn’t a big deal, is it?

I think the point is that you don’t really need to generate the gemspec since you can put dynamic code in it. I can’t think of any reason why you couldn’t read the Gemfile or a manifest from the gemspec?

It’s a good point about what to do about the version field though. Unfortunately git commit shas aren’t valid version strings according to rubygems. (Though you could hack around this. e.g.: version = “00.#{`git rev-parse HEAD`}”).

Thanks for this post, you made me think about the whole thing in a completely different way. I still think it’s annoying to have people touch your .gemspec every time they send a patch, but that’s probably a problem with having generated .gemspecs.

Also, the idea of having my gem information in the Rakefile _and_ the .gemspec in my repository doesn’t feel right (like you do with Jeweler).

I’ve wrote a follow-up to my “.gitignore your *.gemspec” post called “Don’t put your *.gemspec in your Rakefile” (http://jeffkreeftmeijer.com/2010/dont-put-your-gemspec-in-your-rakefile/) in which I explain myself more thoroughly.

I agree with Yehuda here, I’ve been doing this for a long time and I’ve always failed to see the usefulness of virus-like gems like Hoe, an overly complex self reproducing piece of code Ryan Davis and his minions are way too proud of.

About globbing: that is of course optional, and I use more restrictive patterns, but maintaining a manifest can lead to some omissions too.

I couldn’ t agree more with some topics but for me jeweler still valid for things like bootstrap the gem folders, git repo local and remote, manage release versions and so on Jeweler.

Another great example of an incorrect assumption:

http://github.com/luislavena/mysql-gem/issues/closed#issue/3

GemSpec files for “gem build” command can be provided in YAML format:

http://github.com/luislavena/mysql-gem/blob/master/mysql.gemspec

But seems bundler fails because it thinks is a Ruby file.

I concur with Ryan Davis here and would like to add that while the gemspec is an acceptably nice DSL, it doubles the work in some areas. Most information that is in the gemspec (Summary, Description, Authors, Version etc.) is already written in other places. Keeping them in sync is tedious. For that reason I generate the gemspec in my gems. This allows me to keep the information in one place only and not having to maintain some checklist (be it mental or written down) on where to change all the relevant information once it changes.

I agree with Yehuda. Don’t try to provide abstractions around two very simple RubyGems commands(build & install). I hate trying to guess what rake task will build and install a package as it seems to vary between projects. Use what exists, works, and will be uniform between projects.

The proper *nix package installation procedure is to install a package. The make install dance, while ubiquitous, is not actually done by apt or rpm or any other package installation tool. It is only used to install libraries from source. His comparison is not apt.

Gems should be installed as packages. Unpackaged source should be installed from source. We already have a tool for installing Ruby libraries from source: setup.rb. If Yehuda wants to bundler to be able to do so, he should insist that they include a setup.rb, not a gemspec.

@Robert Gleeson:

That’s why you don’t build the gem from source and install it, that’s why you use the existing and published gem, not the source repository.

Only use the repository and the developer’s toolbox when you want to contribute.

Read my comment about doing gem build from a git clone will not always result in the same scenario for two different users through time unless is carefully planned.

You shouldn’t be hating guess which task to run to generate the gem, use the existing gem.

If the exiting gem doesn’t work for you, then fork it, contribute back the patches, get them in, talk to the developer… don’t just rant about it.

@Luis Lavena

I agree with you, of course your first call of port to install a RubyGem should be from GemCutter but it doesn’t cover every use case out there. I don’t want to hold back people from cloning my repository and installing and managing a project(which might have been released yet) easily.

Providing an abstraction layer around what is already super simple is completely unnecessary. Don’t break it if its not broken, and don’t make a simple task into a complicated one.

And yeah, I read your previous post. There might be special circumstances for some projects(ie, outside dependencies handled by a third-party package manager) but for the great majority of Ruby projects a GemSpec is all you need. Fit your projects needs as needed, but don’t add what you don’t need. Don’t confuse the user either, if i need to rummage through your rake tasks to figure out how to build a package then i’d call that a failure.

I’m all about making the experience for the user easy and pleasant as i’m sure you are too.

@Robert Gleeson:

Indeed, you don’t need to dig into my projects to figure out that I’ve clear separation of tasks:

http://github.com/luislavena/rake-compiler/tree/master/tasks/
http://github.com/luislavena/mysql-gem/tree/master/tasks/
http://github.com/luislavena/sqlite3-ruby/tree/master/tasks/

Instead of a monolithic Rakefile. Even when using hoe, a simple “rake gem” will be enough.

But of course, if you want to test what you’re about to manually install, then “rake test” or “rake spec” will ensure to bring all the vendored libraries and test it properly.

The problem with most of Ruby developers is they work under “assume everything” type of development, and because of that, they impose something on others as “this is the way of doing it, and if not, you’re doing it wrong”

I’m not against the gemspec, as long the project is small and simple enough to deal with it. I’m against the assumptions around it and people telling me that I’m doing it wrong maybe because I use rake and own RubyGems’ Gem::PackageTask

@ Luis Lavena

Thats partly my point. Its impossible to “assume everything” because human beings are the most dynamic creatures around. They’ll create and encounter situations that you won’t even think about.

Thats why I don’t agree with the notion that “GemCutter for packages, GitHub for source code” will always be true. It should be true for the majority of cases but it won’t be true for all the cases and you can’t impose your ideology on the behavior & habits of other people.

And yeah, every project might have its own special needs but I think all that most ruby projects need is a GemSpec.

Thanks for bringing this up, Yehuda. It’s been a major pet peeve of mine as well. My own feelings on the matter: http://plusrw.wordpress.com/2010/04/09/gemspecs/

@Martin Emde

I also like to manage my dependencies in one place, but I manage them in the gemspec and generate the Gemfile from there. The code to do so is almost nothing and leads to a nice setup. See this gist http://gist.github.com/277349

It’s worthing noting that Bundler now supports a few extras since(?) this post was written:

`bundle gem my-awesome-gem` will create a directory called my-awesome-gem and give it a skeleton to write your gem inside, including rake tasks and Gemfile as above.

You can also `require ‘bundler/gem_tasks` in your Rakefile to add three common gem rake tasks:

rake build # Build example-0.0.1.gem into the pkg directory
rake install # Build and install example-0.0.1.gem into system gems
rake release # Create tag v0.0.1 and build and push example-0.0.1.gem to Rubygems

Thanks for this article!

In your release task, change the line:

system “gem push bundler-#{Bunder::VERSION}”

by adding “.gem”, as follows:

system “gem push bundler-#{Bunder::VERSION}.gem”

In the Rake code:

“`
task :release => :build do
system “gem push bundler-#{Bunder::VERSION}” # *cough* “Bunder” *cough*
end
“`

Leave a Reply

Archives

Categories

Meta