before(:all) doesn’t do what you’d expect

Like many before me, last week I was bitten by RSpec’s interesting implementation of before(:all). Although the documentation (http://rspec.info/documentation/) clearly states that before(:all) ‘is run once and only once, before all of the examples and before any before(:each) blocks’, it in fact runs once for every context in the scope. This might be useful, but in every situation I’ve seen it isn’t. The documentation goes on to say

Warning: The use of before(:all) and after(:all) is generally discouraged because it introduces dependencies between the Examples. Still, it might prove useful for very expensive operations if you know what you are doing.

which implies that it really does only get invoked once – you wouldn’t want an expensive operation to be executed for every context.

RSpec’s author has explained that the behaviour is actually as he intended it and proposed to change the documentation, but so far that hasn’t happened.

So, why did I need before(:all) to operate as advertised? I have written a test that collects some metadata and then iterates over it to generate common Examples for every attribute in the metadata. This is great because as the metadata is expanded, the test suite expands automatically ensuring consistent implementation of all of my API. The metadata is nested, so it makes sense to take advantage of RSpec’s contexts to nest the Examples so that the generated test results are self-documenting. Unfortunately, the setup for this test involves one expensive operation: login to the API to get the metadata and prepare to probe every attribute. Logging in takes several seconds. It should be faster, but that’s a different story – my test expands to nearly 5000 examples so even if logging in only took half a second we would still waste over 40 minutes. As it is, with RSpec running my expensive operation for every context, the test was taking over 4 hours!

A blog post helped me on my way, but it focussed on resetting the database between tests and didn’t work with Rails RSpec. I modified it to work with Rails RSpec and refactored it to make it useable in any test. Using it is simple; add the following to your spec_helper.rb file:

# HORRIBLE HACK to work around RSpec's broken before(:all) which does not get run
# once as per the documentation - it runs once at the level defined and once per
# sub-context!
# Extend ActiveSupport::TestCase with a 'before_once' that takes a block just like
# before(:all), but only gets called once at the top-level.
# Inspired by http://sickill.net/blog/2009/11/23/quick-and-dirty-hack-for-rspec-before-all.html
# but recast so that it can be used in any test.
class ActiveSupport::TestCase
  # Like before(:all), but only runs once at the top context and not for every
  # sub-context. It keeps track of the classes that are registered and does not
  # invoke the block in subclasses.
  def self.before_once
    _before_once_class_parents = []
    self.instance_variable_set(:"@_before_once_class_parents", _before_once_class_parents)
    before(:all) do
      unless _before_once_class_parents.select{|g| self.class.to_s =~ /^#{g}/}.any?
        _before_once_class_parents << self.class.to_s
        yield
      end
    end
  end
end

It adds a before_once method to ActiveSupport::TestCase that you can use just like before(:all), but it guarantees that it will only run once per scope.

So now, using before_once, I can ensure that my test only logs into the API once and not once per attribute. The net effect is a test that runs in about 90 seconds rather than taking half a day and climbing!

Advertisements

Does your rabbit tell you about your build?

[tweetmeme]

A rabbit is for life…

A French company called Violet sells a number of “Things” – internet-enabled devices that interact with you and/or your environment.  (Actually you can buy them from lots of resellers).  They include mirrors, stamps, lamps and strangely rabbits!  The first generation rabbit was called Nabaztag (Nabaztag is Armenian for “rabbit”), whilst the second  generation is called a Nabaztag:tag, and amongst other enhancements, has been blessed with the ability to “sniff” RFID tags and play MP3 files.  You can identify a Nabaztag:tag by it’s belly-button microphone which enables the rabbit to hear a small number of single word commands, e.g. “weather” and respond, e.g. by telling you what the weather will be.  Wikipedia can tell you more about Nabatztags.

I recently got a Nabaztag:tag.  He (or is it a she?) comes in clean pure white, although Violet also sells coloured replacement ears, and many people customise their rabbit’s look.  Of course, the geek in me didn’t care too much about style and rather wanted to fiddle with how it works and get it to do different things…

Cruisecontrol.rb is for peace of mind…

As was alluded to in a previous blog post, we use CruiseControl.rb to do continuous integration of our code.  Within ten minutes of checking some code into Subversion (yes we use that too), our CruiseControl.rb server will automatically

  • check the updated code out
  • update the application’s database using Rails migrations
  • start the application
  • run our unit tests, our integration tests and our performance tests
  • do some profiling of our code
  • shut down the application again
  • build our documentation
  • and finally trawl through our code picking out the TODOs and FIXMEs etc.

This is great, but how do we know whether it worked or not?  Well, we use CCMenu which sits on our menu bars and quietly shows a coloured blob giving the current state of the build.  Some of us integrate it with Growl to actively tell us whether it succeeded or failed when the build completes.  But that wasn’t enough for me!

Tell me, tell me now!

Yep, I had to integrate CruiseControl.rb and my rabbit.  Unfortunately, CruiseControl’s documentation on writing plugins is a little sparse (I couldn’t find any truly complete documentation), and Violet’s documentation isn’t ideal either (and they keep moving it although there is some general consensus between them all).

So, I’ve tried to simplify it for you.  First I wrote a CruiseControl.rb plugin using the sparse documentation and got it working, but it didn’t do much because I’d only learnt about a couple of hooks.  Thinking that there must be more, I trawled the CruiseControl.rb source code to discover all the possible hooks… and there were lots more!  So I added methods for all of the possible CruiseControlRB hooks to my plugin.

Finally, I used that complete template plugin to integrate with my rabbit.  And here’s the code:

#!/usr/bin/ruby

require 'net/http'
require 'uri'

class NabaztagNotifier

  # Configure your Nabaztag's details here.
  NABAZTAG_SERIAL_NUMBER = 'xxxxxxxxxxxx'
  NABAZTAG_TOKEN = 'yyyyyyyyyy'

  COLOURS = {
    :black    => "0,0,0",
    :red      => "255,0,0",
    :green    => "0,255,0",
    :blue     => "0,0,255",
    :magenta  => "255,0,255",
    :cyan     => "0,255,255",
    :yellow   => "255,255,0",
    :white    => "255,255,255",
  }

  def initialize(project)
#    puts "initialize"
  end

  def no_new_revisions_detected
#    puts "no_new_revisions_detected"
  end

  def new_revisions_detected(revisions)
#    puts "new_revisions_detected"
  end

  def sleeping
#    puts "sleeping"
  end

  def build_requested
#    puts "build_requested"
  end

  def configuration_modified
#    puts "configuration_modified"
  end

  def build_initiated
#    puts "build_initiated"
  end

  def build_started(build)
#    puts "build_started"
    rabbit(:colour => :yellow, :message => "Build started for #{committer(build)}", :status => :building)
  end

  def build_finished(build)
#    puts "build_finished"
    if build.successful?
      rabbit(:colour => :green, :message => "Build fixed by #{committer(build)}", :status => :success)
    elsif build.failed?
      rabbit(:colour => :red, :message => "Build broken by #{committer(build)}", :status => :failure)
    elsif build.incomplete?
      rabbit(:colour => :yellow, :message => "Build incomplete", :status => :incomplete)
    else
      rabbit(:colour => :yellow, :message => "Build status unknown", :status => :unknown)
    end
  end

  def build_loop_failed(error)
#    puts "Build loop failed: #{error.class}: #{error.message}"
    rabbit(:colour => :red, :message => "Build failed with an error", :status => :failure)
  end

  def build_broken(build, previous_build)
#    puts "build_broken"
#    rabbit(:colour => :red, :message => "Build broken by #{committer(build)}", :status => :failure)
  end

  def build_fixed(build, previous_build)
#    puts "build_fixed"
#    rabbit(:colour => :green, :message => "Build fixed by #{committer(build)}", :status => :success)
  end

  private

  #
  def committer(build)
    match = build.changeset.match(/ committed by (\w+) /)
    if match.nil?
      return "unknown"
    elsif match[1].length > 1
      return "#{match[1][0,1].upcase} #{match[1][1..-1].capitalize}"
    else
      return match[1]
    end
  end

  def rabbit(options)
    url = "http://api.nabaztag.com/vl/FR/api.jsp?sn=#{NABAZTAG_SERIAL_NUMBER}&token=#{NABAZTAG_TOKEN}"

    # Quarter second tempo
    url += "&chor=4"

    # Ears up => success
    # Ears down => failure
    # Ears askew => unknown
    if options.has_key?(:status)
      angle = case options[:status]
        when :success
          [0, 0]
        when :failure
          [180, 180]
        else
          [90, 90]
        end
      angle.each_index do |ear|
        url += ",0,motor,#{ear},#{angle[ear]},0,#{ear}"
      end
    end

    # Colour:
    #   Symbol => named colour
    #   String => "R,G,B"  (0-255 in each value)
    if options.has_key?(:colour)
      colour = case options[:colour]
      when nil
        COLOURS[:blue]
      when Symbol
        COLOURS[options[:colour]]
      when String
        options[:colour]
      else
        COLOURS[:white]
      end

      (0..12).each do |n|
        url += ",#{n    },led,#{n % 4 + 1},#{colour}"
        url += ",#{n + 1},led,#{n % 4 + 1},#{COLOURS[:black]}"
      end
      url += ",14,led,1,#{colour},14,led,2,#{colour},14,led,3,#{colour}"
    end

    # Message
    url += "&tts=#{URI.escape(options[:message])}&ttlive=10" if options[:message]

    puts "Sending Nabaztag: #{url}"
    $stdout.flush
    Net::HTTP.get_print URI.parse(url)
  end

  public

  def test
    puts "testing"
    rabbit(:colour => :blue, :message => "Boo", :status => :abc)
    build_started(nil)
  end

end

# Uncomment the following line to test:
# NabaztagNotifier.new(nil).test

Project.plugin :nabaztag_notifier

As it stands, it is a drop in CruiseControl.rb/Nabaztag:tag plugin that announces the start of a build and who started it, and the completion of a build and who broke it or fixed it.  However, it is ripe for extension to make it do what you need – make the rabbit more chatty, or rip out the Nabaztag bits and reuse it as the starting place for a standard CruiseControl.rb plugin.

Installing it

There are 3 steps to installing the Nabaztag:tag CruiseControl.rb plugin.

  1. Installing the plugin depends somewhat on your CruiseControl.rb version.  So far we’ve used two different versions:
    • in the first you had to put the above file in builder_plugins/installed directory in CruiseControl.rb, making sure that the file has the right owner and permissions.
    • in our current CruiseControl.rb version, you need to put it in the builder_plugins directory, again making sure that the file has the right owner and permissions.
  2. Then, you’ll need to edit the first few lines to set the
        NABAZTAG_SERIAL_NUMBER = 'xxxxxxxxxxxx'
        NABAZTAG_TOKEN = 'yyyyyyyyyy'

    You can get the serial number from your rabbit – follow the Violet instructions for registering. Once you have registered with Violet and opened your violet.net account, then you can configure your rabbit to join the “Violet ecosystem”, i.e. generate the token.  Now you have the serial number and the token, update the appropriate lines in the plugin.

  3. Finally, simply restart CruiseControl.rb to pick up the plugin.

To test that it is working, start a build – if you start one from CruiseControl.rb’s web interface, the rabbit should announce “Build started for unknown” whilst flashing yellow LEDs and cocking his ears forward expectantly.  Later, hopefully your build will succeed and the rabbit will announce “Build fixed by unknown”, flash green LEDs and perk his ears up, but if it fails then the rabbit will announce “Build broken by unknown”, flash red LEDs and put his ears down flat.

How to tweak it to suit you

Out of the box, the plugin will announce the start and completion of the builds, along with the Subversion committer of the latest checkin.  To help the rabbit pronounce our usernames, the plugin splits the first character off the username, e.g. “jsmith” becomes “J Smith”.  This is done in the private committer() method – modify it to suit the pronunciation of your Subversion usernames, or even to cope with a different source code control system.

By default, the plugin uses the very boring, but universal colours of green meaning good, red meaning bad and yellow meaning warning.  Hence, the rabbit flashes yellow when announcing the start of a build, red if the build fails and green if it succeeds.  You can modify the COLOURS constant in the script to define more colours, and then refer to them in calls to the rabbit() method.  The colours are defined as red/green/blue combinations between 0 and 255.

As I just alluded to, the private rabbit() method does the hard work of putting together a message and a choreography (what Violet calls a sequence of ear movements and LED flashes).  So, this is the method you would modify if you want to change the rabbit’s behaviour.  The Violet documentation (or wherever it is today) isn’t particularly clear, but hopefully is a bit easier to follow with this example.  One thing you could do is to use the voice API parameter to use something other than the default voice.

Finally, as I mentioned earlier, the plugin includes methods for all the hooks I could find in CruiseControl.rb.  You could use any of them to make your rabbit more chatty, or to integrate CruiseControl.rb with other services such as sending emails, writing to logs, etc.

Good luck; have fun.

Using RCov with a remote server or mechanize

Here at workbooks, we use a number of different testing techniques including server-based unit and rspec testing and client-based selenium testing.

To power our ExtJS interface, requests to our RESTFul API with a specific extension reply with an ExtJS compatible JSON object. In order to test if the JSON coming back is what we expect, we wrote a test helper for rspec which uses mechanize to make HTTP requests and a JSON parser to analyse the response against known key/value pairs.

Like many, we wanted to be able to determine the effectiveness of our tests in various areas of our code base and also integrate this into our continuous integration system so we have a constant indicator of whether we are testing new code. The best way to do this is using RCov.

We soon found that when we ran our rspec tests using RCov, that the code paths were not being marked as executed. After a little thinking; we realised that our rspec tests were not invoking the ruby code directly because of our mechanize implementation. A quick google search later revealed that you can also run script/server through rcov – an ideal solution:


rcov script/server -o log/coverage --rails

We modified our cruise control rake task to run up the server using the snippet above and ensured that the coverage report gets put in the log directory (as our custom build artifacts are published from the contents of the log directory). The end result being, that for every build; we now have a rcov report for our tests.

It’s also worth noting that we use the report aggregation feature of rcov (not shown in the example for clarity), as the server is only started for the mechanize/client-side based tests. Model and controller unit testing etc is run without a need for the server.