Testing your Rails 2.3.9 Gem plugin with rspec

Environment Notes:
OS: OS X 10.6.4
Ruby: ree-1.8.7-2010.01 [ x86_64 ]
Terminal: bash

In this post I’m going to cover how to make a Rails specific gem using jeweler, how to embed a test rails app in the gem, and how to test your gem code with rspec.

I have a gem that I’ve helped build. It’s called Surveyor. We’ve used it on several of our apps here at NUBIC and it’s been working out pretty well.

I’ve never been happy with our test suite. It’s very temperamental. For the longest time it had to run it inside of an app that had been built just for testing and development purposes which is annoying. Also, it is another barrier to having people in the budding Surveyor user group help out, add features, etc, etc… Not having a proper test suite is somewhat irresponsible, time to correct that problem so development can move forward at full speed.

Surveyor now is working fine as a standalone gem and the specs run inside the gem repo and use the Rails app context.

Step 1 - Jeweler

We started using Jeweler for Surveyor some time ago. It seems to work well. We’re going to use it here.

Get Jeweler from Rubygems. Here’s the github page with instructions on how to install the gem. The version I use in this example is version 1.4.0. Follow the instructions in the Readme on the github page or the downloaded gem. Also remember to run jeweler --help to see all the params you can pass in. The gist of the process is:

jeweler [options] <name_of_gem>

For this example I’ll pretend I’m creating a gem to help us have customized Rails log reporting. I’m going to call my gem ‘bclog’ (bc is for bio-computing, though coincidentally ‘bc’ are also my initials). This gem will be tested with cucumber and rspec so I’ve included those options when running the jeweler command. See below:

brian$ jeweler --rspec --cucumber bclog
create  .gitignore
create  Rakefile
create  LICENSE
create  README.rdoc
create  .document
create  lib
create  lib/bclog.rb
create  spec
create  spec/spec_helper.rb
create  spec/bclog_spec.rb
create  spec/spec.opts
create  features
create  features/bclog.feature
create  features/support
create  features/support/env.rb
create  features/step_definitions
create  features/step_definitions/bclog_steps.rb
Jeweler has prepared your gem in bclog

Great! If you cd into your new gem dir you’ll be able to run $ rake -T and see all the neat things Jeweler can do for you. We won’t be covering any of those tasks here but it’s good to know that Jeweler has a lot of helpful features for your gem development.

Step 2 - Our dev gemset with RVM

Before we get going into modifying our gem we’ll need to set up our development gemset. To do this you’ll need RVM installed. I highly recommend using RVM for development. Even if you don’t have need for multiple ruby versions the ability to easily setup gemsets on a per-directory level is VERY helpful.

I’m assuming you’ve installed RVM, follow these instructions to install RVM if you don’t have it. To create a gemset stand in our new gem directory and run the command below. For more info on RVM Gemsets see this page.

rvm gemset create bclog

We’ll want to have this gemset being used every time we work on this project. Create an .rvmrc file in the project so that everytime you change to this directory the gemset is auto-loaded. One less thing to worry about. Here’s some more information on the RVM site about the workflow.

echo 'rvm_gemset_create_on_use_flag=1; rvm gemset use bclog' > .rvmrc

Now when we cd into the directory we’ll automatically start using this gemset. Give it a try now to make sure it’s working. cd up one dir and back into the project dir. You’ll see a warning message the first time you do this (say ‘yes’ to the message).

Step 3 - Bundler and Gemfile

In the gemset for the gem, install Bundler (1.0.3 as of writing).

Running gem list should show two gems installed; bundler and rake. See below:

brian$ gem list

*** LOCAL GEMS ***

bundler (1.0.3)
rake (0.8.7)

Create a file called Gemfile in your project directory. Load it up with the Gems we need at this point.

source :rubygems

gem 'jeweler'
gem 'rspec', '~> 1.3'
gem 'rspec-rails', '1.3.3'
gem 'cucumber', '~> 0.8.0'
gem 'rails', '2.3.10'

Save this file and then run bundle install

This should install some gems into your local gem bundle.

Step 4 - First gem build and dev setup

Now that your gems are installed (specifically jeweler at this point). Run rake -T to see the Jeweler tasks available to you. What we need to do at this point is setup the gem we’re building so when we include it in our Gemfile as a local, path specific gem, bundler will recognize it as a gem. This means, having a version number and a .gemspec file.

I opened up my Jeweler generated file lib/bclog.rb and added a module definition… it’s for fun at this point. We’ll be adding more to this file later.

module Bclog
# Awesome code goes here...
end

Now run some commands!

brian$ rake version:write
Updated version: 0.0.0

brian$ rake gemspec
Generated: bclog.gemspec
bclog.gemspec is valid.

Pop over to your Gemfile and add the line below. Substituting ‘bclog’ with the name of your gem. This tells bundler to load up the gem your building into the gemset for the project.

plugin_root = File.dirname(__FILE__)
gem 'bclog', :path => plugin_root

Run the install command for bundler

bundle install

Look for your gem in what is printed to the command line. You should see something like this: Using bclog (0.0.0) from source at /Users/brian/bclog

Step 5 - Rake tasks for building test app

Now we are going to create some rake tasks to handle the setting up and tearing down of the test Rails app we’re going to embed in our spec folder. In the Rakefile for this new gem add the following lines from this gist

If you run Rake -T now you should see these new tasks:

rake testbed:build_app               # Install rails base app in spec dir
rake testbed:copy_files              # Copy in setup files to test app
rake testbed:install_surveyor        # Install surveyor in rails base app, runs migrations, preps for testing 
rake testbed:remove_app              # Remove rails base app in spec dir
rake testbed:setup                   # Setup for the test app (create)
rake testbed:teardown                # Teardown for the test app (remove)

There are a few files you’ll need to put in your gems spec directory. These are files that are specific to your development and bundler environment. There is a rake task called testbed:copy_files that will copy the files from the base of the spec folder to where they should belong inside your test rails app. Here are the files as gists: testboot.rb, testpreinitializer.rb, test_Gemfile

After you have created these files in your spec/ folder in the gem you can run the rake testbed:setup task to install the rails app in your spec folder.

Running this task should put a lot of output to the console, it’s running a Rails generator, installing a bundle, copying files, etc… When it’s done you should see a test_app folder in your spec directory.

Step 6 - Start your testing!!

At this point I’ve covered the basics and you should be ready to start writing and running tests on your gem as you would in a Rails app (ie rake spec, rake features).

Note: For a good overview of Rakefiles and their usage see this blog post by Jason Seifer.

Bunder problem with a Rails gem that has generators

I ran into this problem recently so I thought I’d share.

I have a gem that I am working on. For development I use bundler and have a hard coded :path to where the gem source code is located. This allows me to develop in the context of a Rails app and have any change I make to the gem immediately available it the app. This works fine except for the generators in the gem.

With a Gemfile like this source :rubygems

gem 'rails', '2.3.10'
gem 'surveyor', :path => '/Users/brian/Rails/surveyor_for_dev'

I get “Couldn’t find ‘surveyor’ generator” when I run bundle exec script/generate surveyor from the cmd line.

If I change that line in the Gemfile and remove everything after the “gem ‘surveyor’” part, bundle install, and run the same command, everything works fine.

What I’ve had to do is use the gem for running the generators, then remove the gem from the system (or gemset) and rebundle. This way I can work on the gem despite the issue bundler is causing.

This problem has something to do with how Rails looks up generators, bundler is not providing the gem generators for local gems to Rails in the proper way.

I don’t have a good solution at this time for the problem. It’s just something to keep in mind when doing Rails gem development using bundler.

Setting up multiple Rails apps on OS X using Apache and Passenger

This is a post I thought could benefit people who develop on a Mac and want to use the OS Apache and Passenger to host their Rails projects. You can use passenger pref pane but for some reason this kept crashing my OS pref pane. I couldn’t figure out why this was happening so I removed it and set up everything manually (old school). Turns out it’s not that hard. In general I don’t create new rails apps too often so configuring by hand is not a big deal. Also, it’s likely I’ll forget how to do this by the time I need to add another Rails app so this post will hopefully be beneficial to my future self!

Step 1) Make sure you have installed Ruby, Rubygems, Passenger.

Also, make sure you have installed the Passenger apache module. Instructions are on the Phusion webpage. The instructions that are generated in the terminal are not entirely correct with regards to how you should configure Apache for multiple apps. Don’t let this throw you off, do everything else except the Rails app configuration stuff. That part is described below.

NOTE* Before we get to the configuration stuff; just a reminder to create copies of the files before you start editing them. Name them http-conf.old and hosts-old (for example). Just make sure you don’t have the old conf file end in “.conf” Apache will read any .conf file and load it.

Step 2) Configure Apache to handle named hosts and the one, or more apps your going to have it run.

To do this you’ll need to add a few configuration lines to your Apache httpd.conf file. You should find it under /etc/apache2/http.conf I always add the lines to the end of the file so it’s easy to locate. The file is quite long.

#This allows apache to have name based virtual hosts. can listen on the same Ip. by defualt it does not do this 
NameVirtualHost *:80

<VirtualHost *:80>
  ServerName registar.dev
  DocumentRoot "/Users/brian/Rails/registar/public"
  RailsEnv development
  RailsAllowModRewrite off
  <directory "/Users/brian/Rails/registar/public">
    Order allow,deny
    Allow from all
  </directory>
</VirtualHost>

<VirtualHost *:80>
  ServerName enotis.dev
  DocumentRoot "/Users/brian/Rails/enotis/public"
  RailsEnv development
  RailsAllowModRewrite off
  <directory "/Users/brian/Rails/enotis/public">
    Order allow,deny
    Allow from all
  </directory>
</VirtualHost>

The first line tells Apache to accept named based virtual hosts. This will allow your apps to be accessible at http://myapp.dev instead of the usual http://localhost:3000. Much shorter, and more descriptive. You can see from the code above that I’ve got two apps I’m serving from here, one called Registar and the other called eNOTIS. The ServerName in the VirtualHost block configures how I refer to the app in the browser, the DocumentRoot sets where the apps public folder lives on my filesystem, RailsEnv sets the running environment, and I’m not too sure what the other lines do. I will have to do some more research and update the post. I’m guessing it’s got something to do with access permissions and such and where the request can originate from. Here’s the Apache docs on the config settings.

Step 3) Modify your host file

The hosts file is located in /etc/hosts on your OS X file system. This file is what is used by the OS to figure out where to find http://enotis.dev (in my case) on the network. There isn’t a DNS server out there telling the computer to that this is on my computer (nor should there be). The hosts file is a DNS shortcut to resolve the location (i.e. the IP) of named hosts. Here’s what mine looks like after editing it for the two apps I want to run.

##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1 localhost enotis.dev registar.dev
255.255.255.255 broadcasthost
::1             localhost 
fe80::1%lo0 localhost

You can keep adding apps to the end of the 127.0.0.1 line, just follow the formatting above. I think the other lines are used by BonJour and such. More can be read about this file on the Apple webpage or by typing ‘man hosts’ in the terminal

Step 4) Restart Apache

We made some pretty big changes so you’ll need to restart Apache before they’ll take effect. To do that from the terminal type

sudo /usr/sbin/apachectl restart

note: see the update below if this causes an error when you run it

Step 5) View your Rails apps

At this point you’ve restarted Apache, and told it where to find your Rails apps. If you go to http://.dev you should see your app. If not, see below.

HELP it doesn’t work!!!!

Well, this does happen from time-to-time. When I was figuring all this stuff out I ran into lots of problems. For a while Apache was serving up one app instead of two, Passenger started crashing, Apache started crashing and failing to restart, etc… etc… Use the logs on the system to help you figure out what is going on. Tail them as you make http requests in the browser to your apps URLs. There are two logs, accesslog and errorlog. Use them well.

tail -n 100 -f /var/log/apache2/error_log

Also take a look at any console message if you’ve hosed your config so bad it’s not doing anything at all. The app is called “Console”, spotlight search for it or look in your Apps/Utilities folder.

Update on Tuesday, December 28, 2010 at 4:42PM by Brian Chamberlain

An update to OS X to 10.6.5 breaks the apachectl script. For example, the command sudo /usr/sbin/apachectl restart to give this error:

/usr/sbin/apachectl: line 82: ulimit: open files: cannot modify limit: Invalid argument

See this Apple support thread for more details: http://discussions.apple.com/thread.jspa?threadID=2644342&tstart=0

I fixed it on my machine by editing the apachectl file and changing the ULIMIT_MAX_FILES entry to

ULIMIT_MAX_FILES=""

PGError: ERROR: permission denied: "RI_ConstraintTrigger_xxxxxxx" is a system trigger

This was a fun one.

Today I set out to get one of my apps setup on the CI server we have use at work when I ran into this problem. It occurred only for a few of the remaining specs in the test suite that still use fixtures. The error looked something like this:

ActiveRecord::StatementInvalid in 'RegistryController should show the current registry for the current user - no params sent'
PGError: ERROR:  permission denied: "RI_ConstraintTrigger_2681229" is a system trigger: ALTER TABLE "schema_migrations" ENABLE TRIGGER ALL;ALTER TABLE "users" ENABLE TRIGGER ALL;ALTER
......
ALL;ALTER TABLE "master_answers" ENABLE TRIGGER ALL;ALTER TABLE "questions" ENABLE TRIGGER ALL;ALTER TABLE "answers" ENABLE TRIGGER ALL;ALTER TABLE "people" ENABLE TRIGGER ALL

Turns out this is a permission issue with the how Postgres handles triggers. Triggers belong to the superuser and if you’ve set up your DB permissions properly your test user on your DB should NOT be a super user.

This author correctly identifies the problem but suggests the solution is to change the permissions on your test user:

Here is a better approach to addressing the actual problem (and thorough explanation):

I deviated from this approach slightly in that I only added the ‘require’ statement to pull in the hack in my CI environment (not for all the environments as the author suggests). I thoughts are that this change only needs to solve a problem in the CI env, so that is where it should live. This could bite me in the butt because it introduces an inconsistency in the adapter behavior across envs. To be honest, I’m not sure which is worse. We’ll see how it plays out.

Update on Tuesday, December 8, 2009 at 2:19PM by Brian Chamberlain

So, some additional things to add to this:

For the monkey patch in the active_record_fk_hack.rb file you created. The author of that article forgot to mention that in order to make sure when Rails loads it doesn’t clobber your patch you need to require the file you’re patching. So your file should look like this

# Hack taken from http://kopongo.com/2008/7/25/postgres-ri_constrainttrigger-error
require 'active_record/connection_adapters/postgresql_adapter'
module ActiveRecord
  module ConnectionAdapters
    class PostgreSQLAdapter < AbstractAdapter
      def disable_referential_integrity(&block)
         transaction {
           begin
             execute "SET CONSTRAINTS ALL DEFERRED"
             yield
           ensure
             execute "SET CONSTRAINTS ALL IMMEDIATE"
           end
         }
      end
    end
  end
end

Update on Tuesday, December 8, 2009 at 2:23PM by Brian Chamberlain

Two other things: If you’re not seeing this error it might be because you are creating your test env from schema.rb (which does NOT preserve foreign keys), using migrations or development_structure.sql to build your test db DOES preserve the foreign keys, or perhaps you’re not using foreign keys in your db (which for the record, I think is a bad idea… despite what DHH says (via secondary source)).

To check to see if you’ve got the triggers that will cause this you can use psql from the command line. Once you’ve connected to your db you can see all the triggers you have by doing select * from pg_trigger; To see if you’ll get the same permission based error on your specs you can run the command manually from psql ALTER TABLE people DISABLE TRIGGER ALL; (just make sure, if you don’t get a permission error, that you revert this with ALTER TABLE people ENABLE TRIGGER ALL;

Trouble with the Postgres Ruby gem on OSX 10.6 (pg gem on Snow Leopard)

I recently (i.e. earlier today) decided to stop cheating and using the Postgres pure Ruby (postgres-pr) adapter and switch to the real-deal compiled version (formerly known as ‘postgres’ now known as just ‘pg’). I had been putting this off because all prior attempts to sudo gem install postgres had failed with esoteric messages which I will likely never understand.

It also happens that I just upgraded to OS X 10.6 (AKA Snow Leopard). What better time to try again?

Step 1 - Out with the old

gem uninstall postgres-pr

Select gem to uninstall:
1. postgres-pr-0.4.0
2. postgres-pr-0.5.1
3. postgres-pr-0.6.1
4. postgres-pr-0.6.1
5. All versions

I chose option 5

Successfully uninstalled postgres-pr-0.4.0
Successfully uninstalled postgres-pr-0.5.1
Successfully uninstalled postgres-pr-0.6.1
Successfully uninstalled postgres-pr-0.6.1

I don’t know why there were two postgres-pr-0.6.1 perhaps one had been installed in my local .gem directory and the other in the system gem dir.

Step 2 - Installing the newness

sudo gem install pg

Wait for it…

FAIL! `ERROR: Error installing pg:
ERROR: Failed to build gem native extension.

/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
extconf.rb:1: command not found: pgconfig —version
ERROR: can’t find pgconfig.
HINT: Make sure pg_config is in your PATH `

Crap, that sucks. But there is a glimmer of hope. Note the “HINT”…hmm… thanks fellas!

Step 3? - Where is that pg_config?!

Through the magic of the command line we can search for it. Drop this on the command line.

mdfind pg_config|grep bin|uniq

Your results will probably be different but mine are: /Library/PostgreSQL/8.3/bin/pg_config
/usr/local (from old Mac)/bin/pg_config
/usr/local (from old Mac)/pgsql/bin/pg_config
/usr/local (from old Mac)/src/postgresql-8.2.5/src/bin/pg_config
/usr/local (from old Mac)/src/postgresql-8.2.5/src/bin/pg_config/pg_config
/usr/local (from old Mac)/src/postgresql-8.2.5/src/bin/pg_config/pg_config.c
/usr/local (from old Mac)/src/postgresql-8.2.5/src/bin/pg_config/pg_config.o

Pay attention to the first line that’s where the elusive ‘pg_config’ is hiding. (For the record, I have no idea why this file is needed or what it does. Perhaps some sort of pg config?) See the comments below for an explanation by my friend Rhett.

Note to reader: At this point I got distracted and tried to delete that local (from old mac) folder and almost ran an rf with recursive force on my usr/local directory (by almost… I mean I did run it but thankfully a sudo is required for this operation. For all my complaining about having to use Sudo it totally saved my ass. I will henceforth never complain about having to type it in all the time.)

Step… to hell with the steps because we have derailed.

Time to try the install again. Drop the path on the command as part of the install like so.

PATH=/Library/PostgreSQL/8.3/bin:$PATH sudo gem install pg

This yields a new, somewhat more confusing and more verbose error. The meat of it is:
In file included from compat.c:16:
compat.h:38:2: error: #error PostgreSQL client version too old, requires 7.3 or later.

I have even less of an idea what to do with this… thankfully, after some googling it turns out this is a common issue compiling the pg gem on Snow Leopard and the fix is easy.

So, combining the first fix with this new one we get:
PATH=/Library/PostgreSQL/8.3/bin:$PATH sudo env ARCHFLAGS='-arch i386' gem install pg

Running this we get:

Building native extensions. This could take a while...
Successfully installed pg-0.8.0
1 gem installed

Sweet!!! No more postgres-pr. We are now on the very fast pg gem for Postgres on Ruby.

Update on Saturday, March 27, 2010 at 11:44AM by Brian Chamberlain

Somehow I forgot all the trouble I went through getting this to work. After switching to use Ruby EE 1.8.7 I needed to re-install the Ruby pg gem and ran into this error:

Can't find the 'libpq-fe.h header

After some searching on blogs and not finding anything that worked. One comment on this blog pointed me in the right direction.

I needed to do this:

sudo env ARCHFLAGS="-arch i386" gem install pg -- --with-pg-include=/Library/PostgreSQL/8.3/include/ --with-pg-lib=/Library/PostgreSQL/8.3/lib/

Note: The double dash after ‘gem install pg’ this passes the stuff after to ‘pg’ and not to ‘gem’. Also note the location of PostgreSQL. I used the installer package it seems. If I had installed it by hand (which I might do when I upgrade to PostgreSQL 8.4) it would live under /usr/local/pgsql (or something similar).