So after my announcement about moneta yesterday, a very common question was “what’s the point of this?”

There are basically two targets:

• If you are needing a cache store that might potentially need to scale, you can use Moneta::Memory to start, and then scale up to Moneta::Memcache or Moneta::TokyoCabinet, etc. It’s certainly possible to do that right now, but you’d have to retool some of your code in order to do that. With Moneta, key/value stores are literally just drop-in-replacements for each other.
• More interestingly, libraries that want to provide sugar around key/value stores (e.g. Rails and Merb’s caching support) can use Moneta as their backend in much the same way as they use Rack to connect to web servers or use DataObjects to connect to backend databases. By providing a simple API, libraries can say “we support moneta stores”, and then any stores created by the community will just work. One interesting ideas that I heard from dkubb last night was moving DataMapper’s IdentityMap to be compatible with Moneta (pretty darn easy since Moneta’s API is a subset of the Hash API), and support plugging in other moneta stores. This would allow processes or even multiple servers to share an IdentityMap, providing a simple write-through cache. When potentially combined with something like Memcache, this could be a powerful combination.
• The bottom line is that by making key/value stores an abstracted idea, the community can build a slew of stores for Moneta, which will then work with whatever front-ends use them. The simplicity of the idea belies the potential power that comes from creating easy to understand APIs and letting the community at large take care of the actual implementaion.

I’m happy to announce the (very) initial release of a new library called moneta, which aims to provide a unified interface for key/value stores. The interface mirrors the key/value API of hash, but not the iteration API, since iteration is not possible across all key/value stores. It is likely that a future release will add support for enumeration by keeping a list of keys in a key (very meta :P )

In order to prove out the API, I created five highly experimental stores for the initial release:

• A basic file store: It uses the file name as the key, the file contents as the value, and xattrs to hold expiration information. The file store requires the xattr gem
• An xattrs-only store: This uses a single file’s xattrs for keys and values, and a second file’s xattrs to hold an identical hash with expiration information
• An in-memory store: This was the first store I wrote, purely to prove out the API. It uses a Hash internally, and a second hash for expiration information
• A DataMapper store: Uses any DataMapper-supported storage with three columns: the_key, value, and expiration
• A memcached store: Uses memcached and its native expiration

Note that the initial release also does not do any kind of locking; a Moneta Store should be treated as a standard hash, and should be locked appropriately by the consumer. Also note that the stores themselves do not perform any locking, so they should probably not be used between processes at this time (i.e. they are only experimental). It should be pretty straight-forward to add locking to most of the stores.

Likely the only locking issue in the memcached store is in #delete, which does:

def delete(key)
  value = self[key]
  @cache.delete(key) if value
  value
end

As a result, it is possible (at the moment, again) for delete to return a key that has been modified before it was actually deleted. For most use-cases, this is rather unlikely to matter, but keep in mind that pretty much all of the stores are unoptimized and not concurrent-safe across processes. Treat the stores, at the moment, as proofs of concept for the overall API.

Some details

The Moneta API is purposely extremely similar to the Hash API. In order so support an
identical API across stores, it does not support iteration or partial matches, but that
might come in a future release.

The API:

#initialize(options)
options differs per-store, and is used to set up the store

#[](key)
retrieve a key. if the key is not available, return nil

#[]=(key, value)
set a value for a key. if the key is already used, clobber it.
keys set using []= will never expire

#delete(key)
delete the key from the store and return the current value

#key?(key)
true if the key exists, false if it does not

#has_key?(key)
alias for key?

#store(key, value, options)
same as []=, but you can supply an :expires_in option,
which will specify a number of seconds before the key
should expire. In order to support the same features
across all stores, only full seconds are supported

#update_key(key, options)
updates an existing key with a new :expires_in option.
if the key has already expired, it will not be updated.

#clear
clear all keys in this store

I started today with a (I thought) pretty simple task that I expected to be able to dispense with in an hour or two. The task was to modify param parsing to correctly handle foo=bar&foo=baz. In Rails, that returns {“foo” => “bar”}. In Merb, it returns {“foo” => “baz”}. In Rack, it returns {“foo” => ["bar", "baz"]}. In other words, a crazy state of affairs.

Last night, we agreed to attempt something like the following:

>> q = Rack::Util.parse_query("foo=bar&foo=baz")
=> {"foo" => "baz"}
>> q["foo"]
=> "baz"
>> q.all("foo")
=> ["bar", "baz"]
>> q.each {|key, value| p [key, value] }
["foo", "baz"]
=> nil

The idea was cribbed from WebOb, a popular Python library built for WSGI (the inspiration for Rack). WebOb uses their own MultiDict, and the patch I wrote introduces a MultiHash for Ruby with similar semantics.

Next, I needed to modify the Rack tests because they initially tested the actual resulting Hash from parsing parameters, and since the structure was internally different, I needed to make some changes to the tests to make them pass. Here’s a sample line from the diff:

-      should.equal "foo" => "1", "bar" => "2"
+      should.match_hash "foo" => "1", "bar" => "2"

Rack is using test/spec, so the matcher looks like:

class Test::Spec::Should
  def match_hash(hash)
    matches = true
    hash, obj = hash.dup, @object.dup
    while !hash.empty?
      key, value = hash.shift
      assert_equal value, obj.delete(key), "#{value.inspect} expected for key #{key.inspect}"
    end
    assert obj.empty?, "Some additional keys were found: #{obj.inspect}"
  end
end

I’d imagine that it could use some work, but this did the trick for the moment. I also added a new context for the new behavior:

context "when the same value is supplied twice and it does not end in []" do
  before do
    @hash = Rack::Utils.parse_query("foo=bar&foo=quux")
  end
 
  specify "should return the last parsed value when indexed" do
    @hash.should.match_hash "foo" => "quux"
  end
 
  specify "should return all of the values with .all" do
    @hash.all("foo").should.equal ["bar", "quux"]
  end
end

Now that all the tests passed, I moved on to Rails, and then proceeded to spend more hours than I’d like to admit trying to figure out how a seemingly unrelated bug was triggered by this change. As it turned out, I was running the latest version of Rack before the repo was moved, and there was a change to part of the same file (utils.rb) that broke Rails. A few more tweaks later and all tests pass.

The proposal should be hitting the rack mailing list in the next day or so, and I’ll be sure to keep everyone up to date on how that’s going.

During the hackfest in Chicago, we finally merged my ActionView changes into 3-0-unstable, the official Rails next branch. Josh’s ActionDispatch reorg (moving dispatching related things to their own top-level directory under ActionPack), which continues to leverage Rack, was also merged in.

Unfortunately, the merge produced a fair number of test failures, and I spent today fixing them. A few of the failures were legitimate, but going to be corrected in Rails3 significantly differently than they were implemented (for Rails 2.3) in the past. I wanted a “pending” classification, as in rspec, but unfortunately Test::Unit 1 (which is currently used by Rails) has no such thing. I searched github for and found a quick hack by Jeremy McAnally that half fit the bill. It printed a “P” when a test was pending, but never printed a list of pending specs. I modified Jeremy’s code to add support for printing the pending specs at the end. It is available as part of ActiveSupport::TestCase in 3-0-unstable.

The syntax is:

def test_should_implicitly_render_html_template_from_xhr_request
  pending do
    get :render_implicit_html_template_from_xhr_request, :format => :js
    assert_equal "Hello HTML!", @response.body
  end
end

When combined with the new declarative testing style in Rails 2.3, you can do:

test "implicitly renders an html template from XHR requests" do
  pending do
    get :render_implicit_html_template_from_xhr_request, :format => :js
    assert_equal "Hello HTML!", @response.body
  end
end

Anyhow, after adding support for pending, 3-0-unstable is green again!