Ruby's Hidden Gems: Bullet

A database is the heart of many applications, and having problems with it may result in serious performance issues.

ORMs such as ActiveRecord and Mongoid help us abstract implementation and deliver code faster, but sometimes, we forget to check what queries are running under the hood.

The bullet gem helps us identify some well-known database-related problems:

"N+1 Queries": when the application runs a query to load each item of a list
"Unused Eager Loading": when the application loads data, usually to avoid N+1 queries, but doesn't use it
"Missing Counter Cache": when the application needs to execute count queries to get the number of associated items

In this post, I'm going to show:

how to configure the bullet gem in a Ruby project,
examples of each problem mentioned before,
how bullet detects each,
how to fix each problem, and
how to integrate bullet with AppSignal.

I will use some examples from a project that I created for this post.

How to Configure Bullet in a Ruby Project

First, add the gem to Gemfile.

We can add it to all environments given, we can enable or disable it and use a different approach on each one:

1gem 'bullet'

Next, it's necessary to configure it.

If you are in a Rails project, you can run the following command to generate the configuration code automatically:

1bundle exec rails g bullet:install

If you are in a non Rails project, you can add it manually, for example, by adding the following code in spec_helper.rb after loading the application's code:

1Bullet.enable        = true
2Bullet.bullet_logger = true
3Bullet.raise         = true

And adding the following code in the main file after loading the application's code:

1Bullet.enable = true

I'm going to share more details on configurations in this post. If you want to see them all, go to bullet's README page.

Using bullet In Tests

With the previously suggested configuration, Bullet will detect bad queries executed in tests and raise exceptions for them.

Now, let's see some examples.

Detecting N+1 Queries

Given an index action as follows:

1# app/controllers/posts_controller.rb
2class PostsController < ApplicationController
3  def index
4    @posts = Post.all
5  end
6end

And a view like this:

1# app/views/posts/index.html.erb
2
3<h1>Posts</h1>
4
5<table>
6  <thead>
7    <tr>
8      <th>Name</th>
9      <th>Comments</th>
10    </tr>
11  </thead>
12
13  <tbody>
14    <% @posts.each do |post| %>
15    <tr>
16      <td><%= post.name %></td>
17      <td><%= post.comments.map(&:name) %></td>
18    </tr>
19    <% end %>
20  </tbody>
21</table>

bullet will raise an error detecting an "N+1" when running an integrated test that executes code from the view and the controller, for example, using a request spec as follows:

1# spec/requests/posts_request_spec.rb
2require 'rails_helper'
3
4RSpec.describe "Posts", type: :request do
5  describe "GET /index" do
6    it 'lists all posts' do
7      post1 = Post.create!
8      post2 = Post.create!
9
10      get '/posts'
11
12      expect(response.status).to eq(200)
13    end
14  end
15end

In this case, it will raise this exception:

1Failures:
2
3  1) Posts GET /index lists all posts
4     Failure/Error: get '/posts'
5
6     Bullet::Notification::UnoptimizedQueryError:
7       user: fabioperrella
8       GET /posts
9       USE eager loading detected
10         Post => [:comments]
11         Add to your query: .includes([:comments])
12       Call stack
13         /Users/fabioperrella/projects/bullet-test/app/views/posts/index.html.erb:17:in `map'
14         ...
15     # ./spec/requests/posts_controller_spec.rb:9:in `block (3 levels) in <top (required)>'

This happens because the view is executing one query to load each comment name in post.comments.map(&:name):

1Processing by PostsController#index as HTML
2  Post Load (0.4ms)  SELECT "posts".* FROM "posts"
3  ↳ app/views/posts/index.html.erb:14
4  Comment Load (0.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 1]]
5  ↳ app/views/posts/index.html.erb:17:in `map'
6  Comment Load (0.1ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 2]]

To fix it, we can simply follow the instruction in the error message and add .includes([:comments]) to the query:

1-@posts = Post.all
2+@posts = Post.all.includes([:comments])

This will instruct ActiveRecord to load all the comments with only 1 query.

1Processing by PostsController#index as HTML
2  Post Load (0.2ms)  SELECT "posts".* FROM "posts"
3  ↳ app/views/posts/index.html.erb:14
4  Comment Load (0.0ms)  SELECT "comments".* FROM "comments" WHERE "comments"."post_id" IN (?, ?)  [["post_id", 1], ["post_id", 2]]
5  ↳ app/views/posts/index.html.erb:14

However, bullet will not raise an exception in a controller test like the following, because controller tests don't render views by default, so the N+1 query will not be triggered.

Note: controller tests are discouraged since Rails 5:

1# spec/controllers/posts_controller_spec.rb
2require 'rails_helper'
3
4RSpec.describe PostsController do
5  describe 'GET index' do
6    it 'lists all posts' do
7      post1 = Post.create!
8      post2 = Post.create!
9
10      get :index
11
12      expect(response.status).to eq(200)
13    end
14  end
15end

Another example of a test that Bullet will not detect an "N+1" is a view test because, in this case, it will not run the N+1 queries in the database:

1# spec/views/posts/index.html.erb_spec.rb
2require 'rails_helper'
3
4describe "posts/index.html.erb" do
5  it 'lists all posts' do
6    post1 = Post.create!(name: 'post1')
7    post2 = Post.create!(name: 'post2')
8
9    assign(:posts, [post1, post2])
10
11    render
12
13    expect(rendered).to include('post1')
14    expect(rendered).to include('post2')
15  end
16end

A Tip to Have More Chances to Detect an N+1 in Tests

I recommend creating at least 1 request spec for each controller action, just to test if it returns the correct HTTP status, then bullet will be watching the queries when rendering these views.

Detecting Unused Eager Loading

Given the following basic_index action:

1# app/controllers/posts_controller.rb
2class PostsController < ApplicationController
3  def basic_index
4    @posts = Post.all.includes(:comments)
5  end
6end

And the following basic_index view:

1# app/views/posts/basic_index.html.erb
2
3<h1>Posts</h1>
4
5<table>
6  <thead>
7    <tr>
8      <th>Name</th>
9    </tr>
10  </thead>
11
12  <tbody>
13    <% @posts.each do |post| %>
14    <tr>
15      <td><%= post.name %></td>
16    </tr>
17    <% end %>
18  </tbody>
19</table>

When we run the following test:

1# spec/requests/posts_request_spec.rb
2require 'rails_helper'
3
4RSpec.describe "Posts", type: :request do
5  describe "GET /basic_index" do
6    it 'lists all posts' do
7      post1 = Post.create!
8      post2 = Post.create!
9
10      get '/posts/basic_index'
11
12      expect(response.status).to eq(200)
13    end
14  end
15end

Bullet will raise the following error:

1  1) Posts GET /basic_index lists all posts
2     Failure/Error: get '/posts/basic_index'
3
4     Bullet::Notification::UnoptimizedQueryError:
5       user: fabioperrella
6       GET /posts/basic_index
7       AVOID eager loading detected
8         Post => [:comments]
9         Remove from your query: .includes([:comments])
10       Call stack
11         /Users/fabioperrella/projects/bullet-test/spec/requests/posts_request_spec.rb:20:in `block (3 levels) in <top (required)>'

This happens because it's not necessary to load the list of comments for this view.

To fix the problem, we can simply follow the instruction in the error above and remove the query .includes([:comments]):

1-@posts = Post.all.includes(:comments)
2+@posts = Post.all

It's worth saying that it will not raise the same error if we run only a controller test, without render_views, as shown before.

Detecting Missing Counter Cache

Given a controller like this:

1# app/controllers/posts_controller.rb
2class PostsController < ApplicationController
3  def index_with_counter
4    @posts = Post.all
5  end
6end

And a view like this:

1# app/views/posts/index_with_counter.html.erb
2
3<h1>Posts</h1>
4
5<table>
6  <thead>
7    <tr>
8      <th>Name</th>
9      <th>Number of comments</th>
10    </tr>
11  </thead>
12
13  <tbody>
14    <% @posts.each do |post| %>
15    <tr>
16      <td><%= post.name %></td>
17      <td><%= post.comments.size %></td>
18    </tr>
19    <% end %>
20  </tbody>
21</table>

If we run the following request spec:

1describe "GET /index_with_counter" do
2  it 'lists all posts' do
3    post1 = Post.create!
4    post2 = Post.create!
5
6    get '/posts/index_with_counter'
7
8    expect(response.status).to eq(200)
9  end
10end

bullet will raise the following error:

11) Posts GET /index_with_counter lists all posts
2  Failure/Error: get '/posts/index_with_counter'
3
4  Bullet::Notification::UnoptimizedQueryError:
5    user: fabioperrella
6    GET /posts/index_with_counter
7    Need Counter Cache
8      Post => [:comments]
9  # ./spec/requests/posts_request_spec.rb:31:in `block (3 levels) in <top (required)>'

This happens because this view is executing 1 query to count the number of comments in post.comments.size for each post.

1Processing by PostsController#index_with_counter as HTML
2  ↳ app/views/posts/index_with_counter.html.erb:14
3  Post Load (0.4ms)  SELECT "posts".* FROM "posts"
4  ↳ app/views/posts/index_with_counter.html.erb:14
5   (0.4ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 1]]
6  ↳ app/views/posts/index_with_counter.html.erb:17
7   (0.1ms)  SELECT COUNT(*) FROM "comments" WHERE "comments"."post_id" = ?  [["post_id", 2]]

To fix this, we can create a counter cache, which can be a bit complex, especially if there is data in the production database.

A counter cache is a column that we can add to a table, that ActiveRecord will update automatically when we insert and delete associated models. There are more details in this post. I suggest reading it to know how to create and sync the counter cache.

Using Bullet in Development

Sometimes, tests might not detect the problems previously mentioned, for example, if test coverage is low, so it's possible to enable bullet in other environments using different approaches.

In the development environment, we can enable the following configurations:

1Bullet.alert         = true

Then, it will show alerts like this in the browser:

1Bullet.add_footer    = true

It will add a footer on the page with the error:

It's also possible to enable errors to be logged in the browser's console:

1Bullet.console    = true

It will add an error like this:

Using Bullet in Staging with Appsignal

In the staging environment, we don't want these error messages to be shown to end-users, but it would be great to know if the application starts to have one of the problems mentioned previously.

At the same time, bullet may degrade performance and increase memory consumption in the application, so it's better to enable it only temporarily in staging, but don't enable it in production.

Assuming the staging environment is using the same configuration file as the production environment, which is a good practice to reduce the difference between them, we can use an environment variable to enable or disable bullet as follows:

1# config/environments/production.rb
2config.after_initialize do
3  Bullet.enabled   = ENV.fetch('BULLET_ENABLED', false)
4  Bullet.appsignal = true
5end

To receive notifications about issues Bullet has found in your staging environment, you can use AppSignal to report those notifications as errors. You'll need to have the appsignal gem installed and configured in your project. You can see more details in the Ruby gem docs.

Then, if a problem is detected by bullet, it will create an error incident like this:

This error is raised by the uniform_notifier gem which was extracted from bullet.

Unfortunately, the error message doesn't show enough information, but I sent in a Pull Request to improve this!

Conclusion

The bullet gem is a great tool that can help us detect problems that will degrade performance in applications.

Try to keep good test coverage, as previously mentioned, to have greater chances of detecting these problems before going to production.

As an extra tip, if you want to be even more protected against performance problems related to the database, take a look at the wt-activerecord-index-spy gem, which helps to detect queries that are not using proper indexes.

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!