Faster Test Boot Times with Bundler Standalone

TL;DR

bundle install --standalone generates a file that allows you to avoid loading bundler at runtime (while still having your load path set up) when booting your test environment. With a fast test suite, this can make a noticeable difference in how long it takes to run.

I recently started a new project, and inspired by what I’ve been learning from Destroy All Software, I want the tests to run as fast as possible. Getting the wall clock times of my test runs down under a second makes me so much more productive!

One of the keys to achieving this, as people like Corey Haines talk about, is loading as few things as possible. This has an important design benefit–it forces you to think carefully everytime you couple your code-under-test to a new dependency–and is essential for achieving fast tests.

If you use bundler for your project (as you should; it’s awesome), it’s an additional thing that you may not have to load when you run your tests. On this project, the tests are fast enough that running the tests through bundler makes a significant, noticeable difference.

Without bundle exec:

➜  time rspec spec/unit/models
................................

Finished in 0.05011 seconds
32 examples, 0 failures

rspec spec/unit/models  0.52s user 0.06s system 97% cpu 0.595 total

With bundle exec:

➜  time bundle exec rspec spec/unit/models
................................

Finished in 0.05819 seconds
32 examples, 0 failures

bundle exec rspec spec/unit/models  1.20s user 0.18s system 98% cpu 1.398 total

That’s 0.6 seconds vs. 1.4 seconds–more than twice as slow to run the tests through bundle exec.

When I noticed this, I tried running other specs without bundle exec…and immediately ran into a problem:

➜  rspec spec/unit/apps

.../rubygems/custom_require.rb:36:in `require': cannot load such file -- shardonnay (LoadError)
...the full backtrace here...

Lame. I had forgotten that I had a few :git gems in my Gemfile (such as shardonnay, a private gem we’re actively developing internally). As the bundler documentation states, :git gems are not available to rubygems. You have to use Bundler.setup to make them available on the load path. But Bundler.setup is equivalent to bundle exec¹, and I still wanted to skip it if I could find way.

The Solution

One of my co-workers, Evan Battaglia, suggested there might be a way to make a record of all the load paths that need to be setup (including the paths for :git gems) and re-use that. I started looking into doing this when I remembered a new bundler 1.1 option I’ve been meaning to try out: bundle install --standalone. Running this generates a handy file at bundle/bundler/setup.rb:

path = File.expand_path('..', __FILE__)
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/gems/rake-0.9.2.2/lib")
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/gems/addressable-2.2.7/lib")
$:.unshift File.expand_path("#{path}/../ruby/1.9.1/bundler/gems/shardonnay-f2cf0d585593/lib")

The real file in my project is much longer, with many more gems, but that should give you the idea. Notice that it sets up the load path both for installed released gems and :git gems. This makes it perfect for what I want to do.

To use this, I’ve git-ignored the bundle directory (since it contains artifacts generated by bundler) and I have this bit of code that runs at the start of booting my test environment:

begin
  # use `bundle install --standalone' to get this...
  require_relative '../bundle/bundler/setup'
rescue LoadError
  # fall back to regular bundler if the developer hasn't bundled standalone
  require 'bundler'
  Bundler.setup
end

With this in place, I’m able to run my specs without bundle exec, with no slowdown from bundler at runtime (since bundler isn’t even being loaded at runtime!).

Caveats

This is working great for me, and I plan to keep using this strategy, but it does have a few caveats:

• There’s no guarantee that the right version of rspec will be used (since it is the binary I invoke from my shell, before the load paths get setup). I’m using an RVM gemset for this project so it’s not much of an issue.
• Bundler doesn’t keep any persistent state about the fact that I’m using --standalone. If I forget the --standalone flag the next time I run bundle install, the generated setup file will not be updated with the new load paths–which means the wrong version of a gem may be used the next time I run my tests.
• A full Bundler.setup gives you a sandbox guarantee: the only gems that can be loaded are those that included in the locked bundle. There’s no such guarantee when using --standalone. I can install a gem and immediately require it. I’m not too concerned about this since the normal way I install gems now is via Bundler; plus, our CI server will catch any problems here since it is running bundle install (with no --standalone).

I’m certainly willing to live with these tradeoffs in exchange for faster test environment boot times, but you may not be.