RSpec 2.6 introduced a deprecation warning when using
... } after defining an example group. In RSpec 2.7, this warning was removed, and now an error is raised when particular configuration settings (
mock_with) are set after defining an example group.
I’m the one who made those changes to RSpec, and people are rightfully annoyed with these changes…but there’s a lot more to the story. I’m not sure what the right solution is to the problem I was trying to solve by making those changes, but I’m hoping that by blogging about it, we can get some good ideas from the community.
One of the primary goals of RSpec 2 was to decouple the spec runner (rspec-core) from the mocking framework (rspec-mocks) and the expectation framework (rspec-expectations). Besides the fact that decoupling is A Good Thing™, this separation opened up new possibilities for people to pick and choose which parts of RSpec they want to use.
In particular, it allows people to use rspec-core to define and run their tests using RSpec’s example definition DSL (
let, etc) while sticking with the
assert_foo assertion methods from Test::Unit or minitest, rather than using RSpec’s
object.should whatever syntax. For better or worse, some people really dislike rspec-expectations but love the runner.
RSpec 2 allows you to configure which you want to use:
Both rspec-expectations and the standard library assertions are available as modules–
Test::Unit::Assertions, respectively. RSpec 2.0 to 2.5 included the appropriate module into
RSpec::Core::ExampleGroup just prior to running the examples (that is, after all of them have been defined) to allow people to configure this whenever they want.
Unfortunately, this triggered an unfortunate bug in ruby 1.9…which I’ll get to below.
Infinite Recursion Issues
Shortly after RSpec 2 was released, we began to get some intermittent reports that users were occasionally getting a
SystemStackError from RSpec, indicating infinite recursion was occurring. I myself saw this error when working on the rspec-core specs at one point.
The recursion always happened in rspec-expectations’ method_missing hook. In particular, the call to
super triggered infinite recursion.
I found this very, very puzzling, and spent many hours troubleshooting trying to figure out why
super would infinitely recurse on itself.
It’s a Bug in Ruby 1.9
I eventually managed to boil the problem down to a simple rspec-less example:
If you run this on ruby 1.8, you will (correctly) get a
NoMethodError. On ruby 1.9, you get infinite recursion and a
SystemStackError. Here’s my short explanation of the conditions that trigger this bug:
- Define a module that has a method that uses
- Include that module in a class after it has already been included in one of its subclasses.
- Create an instance of said subclass and call the method.
Note that this error does not occur if the module is included in the superclass before it is included in the subclass.
How this Bug Manifested Itself in RSpec
Here’s how this bug manifested itself in RSpec:
- Every time you define an example group (using
context), RSpec creates a new subclass of
RSpec::Core::ExampleGroup, or a subclass-of-a-subclass, if you have nested your example group within another one.
- Some users included
RSpec::Matchersinto their example groups. Here’s one example.
- As I explained above, in 2.0 to 2.5, RSpec included
RSpec::Core::ExampleGroupright before running the examples, after all examples have been defined (and hence, after the user may have already included
RSpec::Matchersin their example groups).
- When a user called an undefined or misspelled method from an example, it would trigger this error.
Fixing the Issue…by Introducing Other Problems :(
To prevent users from getting infinite recursion, we need to prevent
RSpec::Matchers (and really, any other module that may use
super) from being included in an example group before it is included in
RSpec::Core::ExampleGroup. At one point, I considered adding a hook to either
RSpec::Matchers to detect when a user is including it, and somehow prevent or warn them. However, I quickly realized that the extreme flexibility of Ruby’s module system makes this very complicated. A user may not be including
RSpec::Matchers directly; they may be including a module from some library or plugin, that itself includes
RSpec::Matchers, or includes a module that includes
RSpec::Matchers. I realized this wasn’t going to be a simple solution to get right.
Instead, after talking with David Chelimsky and some other members of the RSpec core team, we decided it was best to change when
RSpec::Matchers gets included. By including
RSpec::Core::ExampleGroup before it has been subclassed the issue goes away entirely.
We still wanted to let people configure
expect_with :stdlib, though. the best solution we could come up with is to delay the inclusion of the expectation module until the moment when
RSpec::Core::ExampleGroup is being subclassed for the first time. This gives the user a chance to configure
expect_with :stdlib as long as they do so before defining any example groups. Once an example group has been defined, we automatically default to
expect_with :rspec–which means that any future
expect_with configuration would be effectively ignored.
Here’s the commit, if you’re interested. You’ll notice that it also changes when the mocking framework adapter module gets included. Since we don’t want to deal with this issue again if/when one of the mocking framework adapter modules uses
super, I thought it best to change it as well.
This whole issue suggested to me that it’s problematic to allow users to configure RSpec after defining examples. In my mind, since the configuration affects how RSpec works, it’s best to set it before you start to use RSpec (i.e. by defining examples). I made a change to cause RSpec to print a deprecation warning when you use
RSpec.configure after defining an example group.
In retrospect, I should have realized that this would affect a lot of users. At the time, it didn’t occur to me that it would. I do all of my RSpec configuration before defining any example groups and I assumed that’s what everyone else did, too. The deprecation warning was mostly just put there on the off chance that someone might configure RSpec after defining an example.
RSpec 2.6 was released with these changes and we immediately began getting questions and complaints about this warning. People like Gary Bernhardt and Corey Haines have a technique of speeding up their tests by loading as little as possible–and this usually involves not loading
spec_helper from each spec file. This can trigger the deprecation warning when one spec file (say,
spec/lib/my_class_spec.rb) does not require
spec_helper, but another file in the same suite (say
spec/models/user_spec.rb) does. If
spec/lib/my_class_spec.rb is loaded before
spec/models/user_spec.rb (which usually happens–they tend to get loaded in alphabetical order), it will trigger the warning since examples are defined in
spec/lib/my_class_spec.rb before the configuration happens in
I’m a big fan of the “don’t load spec_helper” approach now, but at the time I made the changes, I had never heard or thought of doing it that way.
We had a conversation about this on an rspec-rails issue. The best suggestion to come out of that (and one that no one argued against) was to remove the warning, and instead raise an error on the specific, problematic configs (
mock_with), if they get set after defining an example group.
I made this change and it was released in RSpec 2.7.
After RSpec 2.7 came out, there were yet more complaints about this change. Now some users were unable to run their specs because of the error. Effectively, this had made the problem worse–the previous warning could be ignored, but not the error.
I committed a change a few days ago that should improve things here: instead of raising an error anytime
mock_with are called after an example group is defined, the error is only raised if the method call is changing the setting. No error will be raised if you’re simply explicitly setting the default (i.e.
mock_with :rspec) or re-setting the existing config value.
This is certainly an important, needed change that I simply didn’t consider when I made the previous changes. I apologize.
Note that this does not remove the error entirely: if you are configuring RSpec to use a different expectation/assertion framework or mocking framework, this must still be done before an example group is defined so that RSpec can include the appropriate module in
RSpec::Core::ExampleGroup before it has been subclassed.
Avoiding these warnings/errors
If you’re on RSpec 2.6 or 2.7 and have gotten these warnings or errors, there are some very simple changes you can make to avoid them.
First, make sure all of your spec_helper requires are simply
'spec_helper'. If you use a path relative to
__FILE__, as people often do, spec_helper.rb can be loaded multiple times (since ruby will happily re-require a file if it is specified as a different file path). RSpec puts the spec directoy on the load path for you, so you can (and generally should) just require
Second, if you follow the “don’t load spec_helper” approach, and you need to configure either
mock_with, you’ll need to create a secondary helper file for your isolated, fast specs.
Here’s one way to do it:
I know this goes against the “don’t load spec_helper” approach a bit, but the important thing is that the rails/sinatra/whatever environment is not fully loaded in the isolated specs. We’re using fast_spec_helper.rb to only configure the bare minimum–the specific
mock_with settings that must be set before an example group is defined. The one extra require isn’t going to make a noticeable difference in the speed of your isolated tests.
Of course, if you are just setting
expect_with to the default (
:rspec) then you should just remove that configuration entirely and the error should go away–no need for a separate
Alternatively, you could use ruby’s
-r flag to force a helper file to be loaded before any isolated specs, rather than having to require a helper file from each.
Is There a Better Way to Work Around this Bug?
So there you have it…the full story behind the recent configuration warnings and errors in RSpec. I apologize if this has caused upgrade pain for you. I solved the issues in the best way I could figure out.
If you think I totally screwed up, or if you can think of a better way to deal with the ruby 1.9 bug, please let me know in the comments!
Also, I filed a bug on the ruby issue tracker. Please comment there if you would like to see it fixed.