before(:all) doesn’t do what you’d expect

Like many before me, last week I was bitten by RSpec’s interesting implementation of before(:all). Although the documentation (http://rspec.info/documentation/) clearly states that before(:all) ‘is run once and only once, before all of the examples and before any before(:each) blocks’, it in fact runs once for every context in the scope. This might be useful, but in every situation I’ve seen it isn’t. The documentation goes on to say

Warning: The use of before(:all) and after(:all) is generally discouraged because it introduces dependencies between the Examples. Still, it might prove useful for very expensive operations if you know what you are doing.

which implies that it really does only get invoked once – you wouldn’t want an expensive operation to be executed for every context.

RSpec’s author has explained that the behaviour is actually as he intended it and proposed to change the documentation, but so far that hasn’t happened.

So, why did I need before(:all) to operate as advertised? I have written a test that collects some metadata and then iterates over it to generate common Examples for every attribute in the metadata. This is great because as the metadata is expanded, the test suite expands automatically ensuring consistent implementation of all of my API. The metadata is nested, so it makes sense to take advantage of RSpec’s contexts to nest the Examples so that the generated test results are self-documenting. Unfortunately, the setup for this test involves one expensive operation: login to the API to get the metadata and prepare to probe every attribute. Logging in takes several seconds. It should be faster, but that’s a different story – my test expands to nearly 5000 examples so even if logging in only took half a second we would still waste over 40 minutes. As it is, with RSpec running my expensive operation for every context, the test was taking over 4 hours!

A blog post helped me on my way, but it focussed on resetting the database between tests and didn’t work with Rails RSpec. I modified it to work with Rails RSpec and refactored it to make it useable in any test. Using it is simple; add the following to your spec_helper.rb file:

# HORRIBLE HACK to work around RSpec's broken before(:all) which does not get run
# once as per the documentation - it runs once at the level defined and once per
# sub-context!
# Extend ActiveSupport::TestCase with a 'before_once' that takes a block just like
# before(:all), but only gets called once at the top-level.
# Inspired by http://sickill.net/blog/2009/11/23/quick-and-dirty-hack-for-rspec-before-all.html
# but recast so that it can be used in any test.
class ActiveSupport::TestCase
  # Like before(:all), but only runs once at the top context and not for every
  # sub-context. It keeps track of the classes that are registered and does not
  # invoke the block in subclasses.
  def self.before_once
    _before_once_class_parents = []
    self.instance_variable_set(:"@_before_once_class_parents", _before_once_class_parents)
    before(:all) do
      unless _before_once_class_parents.select{|g| self.class.to_s =~ /^#{g}/}.any?
        _before_once_class_parents << self.class.to_s
        yield
      end
    end
  end
end

It adds a before_once method to ActiveSupport::TestCase that you can use just like before(:all), but it guarantees that it will only run once per scope.

So now, using before_once, I can ensure that my test only logs into the API once and not once per attribute. The net effect is a test that runs in about 90 seconds rather than taking half a day and climbing!

Using RCov with a remote server or mechanize

Here at workbooks, we use a number of different testing techniques including server-based unit and rspec testing and client-based selenium testing.

To power our ExtJS interface, requests to our RESTFul API with a specific extension reply with an ExtJS compatible JSON object. In order to test if the JSON coming back is what we expect, we wrote a test helper for rspec which uses mechanize to make HTTP requests and a JSON parser to analyse the response against known key/value pairs.

Like many, we wanted to be able to determine the effectiveness of our tests in various areas of our code base and also integrate this into our continuous integration system so we have a constant indicator of whether we are testing new code. The best way to do this is using RCov.

We soon found that when we ran our rspec tests using RCov, that the code paths were not being marked as executed. After a little thinking; we realised that our rspec tests were not invoking the ruby code directly because of our mechanize implementation. A quick google search later revealed that you can also run script/server through rcov – an ideal solution:


rcov script/server -o log/coverage --rails

We modified our cruise control rake task to run up the server using the snippet above and ensured that the coverage report gets put in the log directory (as our custom build artifacts are published from the contents of the log directory). The end result being, that for every build; we now have a rcov report for our tests.

It’s also worth noting that we use the report aggregation feature of rcov (not shown in the example for clarity), as the server is only started for the mechanize/client-side based tests. Model and controller unit testing etc is run without a need for the server.