Wednesday
May182011

Introduction to Node.js packages: Part 1- How to create a npm package and setup a test suite

Environment used for this post:
OS => OS X 10.6.7
Node.js => v0.4.7
npm => 1.0.6
Terminal => bash

In this post I’ll explain how to create a node package complete with tests and documentation.

Step 0 - Install the pre-req’s

Note the env stats above. I used Homebrew to install Nodejs v0.4.7 and then followed the manual git clone and ‘sudo make install’ procedure for installing npm. I ran into some permission issues using the one liner on the nmp website. I’m not pro enough to tackle internal node install errors yet. The manual git checkout, sudo make install, worked just fine.

Step 1 - Some background

I suggest you take a look at this: http://howtonode.org/introduction-to-npm This is a good introduction to the node package management system. Pay close attention to the Development section. Note what the author says about creating an node package from the get-go, not as an after thought. I could not agree with this more. Let’s have some fun!

Step 2 - Create your package.json file

It’s a good idea to write most of your Node.js application as a node package. The node package system will help you manage dependencies and other project related tasks for you. You can reuse and share these packages through the npm registry. Also, a package is much easier to test than a full blown application. These points, plus the ease of the ‘npm link’ which links your live development changes into your local node env, gives you no excuse not to go the app-as-a-package route right from the start.

Every package has to have a few things; a name, a version, a description, an author, a repo home, etc… All this info has to be in a special format inside your package.json file. Thankfully there is a simple tool to set this up for you (interactively) called ‘npm init’.

First we create a folder for our project:

mkdir filecore
cd filecore

Then we run the ‘npm init’ command and answer some questions.

npm init

After you finish, it will create the following structure for you. Yours will be different if you answer the questions differently.

{
  "author": "Brian Chamberlain ",
  "name": "filecore",
  "description": "Manages the files and directories of a Naccelle project",
  "version": "0.0.1",
  "repository": {
    "type": "git",
    "url": "git://github.com/breakpointer/filecore.git"
  },
  "main": "filecore.js",
  "engines": {
    "node": "~0.4.7" #<<== NOTE: There is no 'v' in this version number
  },
  "dependencies": {},
  "devDependencies": {}
}

Note: I had to tweak the output of the ‘npm init’ script to remove the letter ‘v’ it was putting in the the “node” attribute. Prior to my edit, the output for that section was “~v0.4.7” which is not correct. “~0.4.7” is the proper format.

This is just a start for this file… there are a number of other options you can configure. See this page for more info: https://github.com/isaacs/npm/blob/master/doc/json.md

Step 3 - Basic package structure

If you note from the package.json file above we’ve stated that our “main” is something called “filecore”. It doesn’t have to match the name of your project but it does have to be where you actually want to mount your app.

You should create a file with the name you specified as your “main”. In my example I’ll create one called ‘filecore.js’. In this file you’ll have to “export” the methods that you want available to the consumers of your package. In other words, when you do something like:

var fc = require('filecore');

The variable ‘fc’ will have the methods attached to it which you export. See the code below.


// Requires go here
// var x = require('x');
// var y = require('y');
// ...etc..

exports.method_a = function(){
 // do stuff here
}

exports.method_b = function(){
  // do stuff here
}

Step 4 - Test driven development

I’m a big fan of test driven development. While I don’t always practice test-first development (because frankly, that kinda takes the fun out of hacking on some new code) but when I have solid idea of what I want a piece of code to do I’ll lay down some tests. A good test base can help propel your development and catch those “I swear I didn’t change anything and now it’s broke!” kinda errors that can pop up from time to time.

I’ve used Jasmine on other JS based projects and found it to have a nice syntax and having built in mocking is nice too. Here’s a link to the project: http://pivotal.github.com/jasmine/

For a Node.js version of Jasmine, see this page: https://github.com/mhevery/jasmine-node. It installs just like any other Node.js package. I installed it globally because I intend to use it on other projects (hence the ‘-g’ option)

npm install jasmine-node -g

In the git repo for the jasmine-node package there are some example spec files to help you get going. Here’s my current project tree:

..README
..package.json
..filecore.js
..spec/filecore_spec.js

And as far as the code is concerned here are the filecore.js and the filecore_spec.js files:

//filecore.js

exports.method_a = function(){
 // do stuff here
 return('a');
}

exports.method_b = function(){
  // do stuff here
  return('b');
}
//filecore_spec.js

var fc = require('../filecore.js');
describe('jasmine-node', function(){

  it('should pass', function(){
    expect(fc.method_a()).toEqual('a');
  });

  it('should pass', function(){
    expect(fc.method_b()).toEqual('b');
  });

});

These tests and code are very trivial but they make sure we’ve setup our Node.js module properly and that our test suite works.

When you run the test suite you should see the following:

brian$ jasmine-node spec
Started
..

Finished in 0.002 seconds
1 test, 2 assertions, 0 failures

* That’s all for this part. In Part 2 I will go through fleshing out the filecore module and how to package and publish your Node.js package.*

Sunday
Nov142010

Testing your Rails 2.3.9 Gem plugin with rspec

Environment Notes:
OS: OS X 10.6.4
Ruby: ree-1.8.7-2010.01 [ x86_64 ]
Terminal: bash

In this post I’m going to cover how to make a Rails specific gem using jeweler, how to embed a test rails app in the gem, and how to test your gem code with rspec.

I have a gem that I’ve helped build. It’s called Surveyor. We’ve used it on several of our apps here at NUBIC and it’s been working out pretty well.

I’ve never been happy with our test suite. It’s very temperamental. For the longest time it had to run it inside of an app that had been built just for testing and development purposes which is annoying. Also, it is another barrier to having people in the budding Surveyor user group help out, add features, etc, etc… Not having a proper test suite is somewhat irresponsible, time to correct that problem so development can move forward at full speed.

Surveyor now is working fine as a standalone gem and the specs run inside the gem repo and use the Rails app context.

Step 1 - Jeweler

We started using Jeweler for Surveyor some time ago. It seems to work well. We’re going to use it here.

Get Jeweler from Rubygems. Here’s the github page with instructions on how to install the gem. The version I use in this example is version 1.4.0. Follow the instructions in the Readme on the github page or the downloaded gem. Also remember to run jeweler --help to see all the params you can pass in. The gist of the process is:

jeweler [options] <name_of_gem>

For this example I’ll pretend I’m creating a gem to help us have customized Rails log reporting. I’m going to call my gem ‘bclog’ (bc is for bio-computing, though coincidentally ‘bc’ are also my initials). This gem will be tested with cucumber and rspec so I’ve included those options when running the jeweler command. See below:

brian$ jeweler --rspec --cucumber bclog
create  .gitignore
create  Rakefile
create  LICENSE
create  README.rdoc
create  .document
create  lib
create  lib/bclog.rb
create  spec
create  spec/spec_helper.rb
create  spec/bclog_spec.rb
create  spec/spec.opts
create  features
create  features/bclog.feature
create  features/support
create  features/support/env.rb
create  features/step_definitions
create  features/step_definitions/bclog_steps.rb
Jeweler has prepared your gem in bclog

Great! If you cd into your new gem dir you’ll be able to run $ rake -T and see all the neat things Jeweler can do for you. We won’t be covering any of those tasks here but it’s good to know that Jeweler has a lot of helpful features for your gem development.

Step 2 - Our dev gemset with RVM

Before we get going into modifying our gem we’ll need to set up our development gemset. To do this you’ll need RVM installed. I highly recommend using RVM for development. Even if you don’t have need for multiple ruby versions the ability to easily setup gemsets on a per-directory level is VERY helpful.

I’m assuming you’ve installed RVM, follow these instructions to install RVM if you don’t have it. To create a gemset stand in our new gem directory and run the command below. For more info on RVM Gemsets see this page.

rvm gemset create bclog

We’ll want to have this gemset being used every time we work on this project. Create an .rvmrc file in the project so that everytime you change to this directory the gemset is auto-loaded. One less thing to worry about. Here’s some more information on the RVM site about the workflow.

echo 'rvm_gemset_create_on_use_flag=1; rvm gemset use bclog' > .rvmrc

Now when we cd into the directory we’ll automatically start using this gemset. Give it a try now to make sure it’s working. cd up one dir and back into the project dir. You’ll see a warning message the first time you do this (say ‘yes’ to the message).

Step 3 - Bundler and Gemfile

In the gemset for the gem, install Bundler (1.0.3 as of writing).

Running gem list should show two gems installed; bundler and rake. See below:

brian$ gem list

*** LOCAL GEMS ***

bundler (1.0.3)
rake (0.8.7)

Create a file called Gemfile in your project directory. Load it up with the Gems we need at this point.

source :rubygems

gem 'jeweler'
gem 'rspec', '~> 1.3'
gem 'rspec-rails', '1.3.3'
gem 'cucumber', '~> 0.8.0'
gem 'rails', '2.3.10'

Save this file and then run bundle install

This should install some gems into your local gem bundle.

Step 4 - First gem build and dev setup

Now that your gems are installed (specifically jeweler at this point). Run rake -T to see the Jeweler tasks available to you. What we need to do at this point is setup the gem we’re building so when we include it in our Gemfile as a local, path specific gem, bundler will recognize it as a gem. This means, having a version number and a .gemspec file.

I opened up my Jeweler generated file lib/bclog.rb and added a module definition… it’s for fun at this point. We’ll be adding more to this file later.

module Bclog
# Awesome code goes here...
end

Now run some commands!

brian$ rake version:write
Updated version: 0.0.0

brian$ rake gemspec
Generated: bclog.gemspec
bclog.gemspec is valid.

Pop over to your Gemfile and add the line below. Substituting ‘bclog’ with the name of your gem. This tells bundler to load up the gem your building into the gemset for the project.

plugin_root = File.dirname(__FILE__)
gem 'bclog', :path => plugin_root

Run the install command for bundler

bundle install

Look for your gem in what is printed to the command line. You should see something like this: Using bclog (0.0.0) from source at /Users/brian/bclog

Step 5 - Rake tasks for building test app

Now we are going to create some rake tasks to handle the setting up and tearing down of the test Rails app we’re going to embed in our spec folder. In the Rakefile for this new gem add the following lines from this gist

If you run Rake -T now you should see these new tasks:

rake testbed:build_app               # Install rails base app in spec dir
rake testbed:copy_files              # Copy in setup files to test app
rake testbed:install_surveyor        # Install surveyor in rails base app, runs migrations, preps for testing 
rake testbed:remove_app              # Remove rails base app in spec dir
rake testbed:setup                   # Setup for the test app (create)
rake testbed:teardown                # Teardown for the test app (remove)

There are a few files you’ll need to put in your gems spec directory. These are files that are specific to your development and bundler environment. There is a rake task called testbed:copy_files that will copy the files from the base of the spec folder to where they should belong inside your test rails app. Here are the files as gists: testboot.rb, testpreinitializer.rb, test_Gemfile

After you have created these files in your spec/ folder in the gem you can run the rake testbed:setup task to install the rails app in your spec folder.

Running this task should put a lot of output to the console, it’s running a Rails generator, installing a bundle, copying files, etc… When it’s done you should see a test_app folder in your spec directory.

Step 6 - Start your testing!!

At this point I’ve covered the basics and you should be ready to start writing and running tests on your gem as you would in a Rails app (ie rake spec, rake features).

Note: For a good overview of Rakefiles and their usage see this blog post by Jason Seifer.

Thursday
Nov042010

Ruby strings over multiple lines

This is the one thing in Ruby I always have to look up. This is the best post I’ve found on the topic. http://blog.jayfields.com/2006/12/ruby-multiline-strings-here-doc-or.html

Thursday
Nov042010

Bunder problem with a Rails gem that has generators

I ran into this problem recently so I thought I’d share.

I have a gem that I am working on. For development I use bundler and have a hard coded :path to where the gem source code is located. This allows me to develop in the context of a Rails app and have any change I make to the gem immediately available it the app. This works fine except for the generators in the gem.

With a Gemfile like this source :rubygems

gem 'rails', '2.3.10'
gem 'surveyor', :path => '/Users/brian/Rails/surveyor_for_dev'

I get “Couldn’t find ‘surveyor’ generator” when I run bundle exec script/generate surveyor from the cmd line.

If I change that line in the Gemfile and remove everything after the “gem ‘surveyor’” part, bundle install, and run the same command, everything works fine.

What I’ve had to do is use the gem for running the generators, then remove the gem from the system (or gemset) and rebundle. This way I can work on the gem despite the issue bundler is causing.

This problem has something to do with how Rails looks up generators, bundler is not providing the gem generators for local gems to Rails in the proper way.

I don’t have a good solution at this time for the problem. It’s just something to keep in mind when doing Rails gem development using bundler.

Monday
Oct252010

Compiling Wings3D from source

Wings3D is 3D modeling program which happens to be written in Erlang. Here’s how I installed in on my 10.04 Ubuntu system.

I always put my src in /usr/local/src so you need to use sudo to make changes here.

This was a fresh VM for me so I had to put on some build tools first:

sudo apt-get install build-essential g++ libssl-dev m4 xsltproc fop libwxgtk2.8-dev

You’ll need erlang installed.

sudo curl -O http://www.erlang.org/download/otp_src_R14B_erts-5.8.1.1.tar.gz
tar -zxvf otp_src_R14B_erts-5.8.1.1.tar.gz 
sudo tar -zxvf otp_src_R14B_erts-5.8.1.1.tar.gz 
cd otp_src_R14B/
sudo ./configure --enable-threads --disable-hipe
sudo make
sudo make install

Download the Wings3D Source: http://sourceforge.net/projects/wings/files/wings/1.3.1/wings-1.3.1.tar.bz2/download

Move the source to /usr/local/src and un-bz it. Reading through the instructions in the BUILD.unix you’ll need SDL and a few other things.

Installing SDL from source gave me a small problem. SDL development library. http://www.libsdl.org.

sudo curl -O http://www.libsdl.org/release/SDL-1.2.14.tar.gz
sudo tar -xvzf SDL-1.2.14.tar.gz 
cd SDL-1.2.14/
sudo ./configure --disable-video-aalib --disable-video-directfb 
sudo make
sudo make install

At this point I got:

libtool: compile:  gcc -g -O2 -I./include -D_GNU_SOURCE=1 -fvisibility=hidden -DXTHREADS -D_REENTRANT -DHAVE_LINUX_VERSION_H -c ./src/video/x11/SDL_x11dga.c  -fPIC -DPIC -o build/.libs/SDL_x11dga.o
In file included from ./src/video/x11/SDL_x11video.h:50,
             from ./src/video/x11/SDL_x11dga_c.h:24,
             from ./src/video/x11/SDL_x11dga.c:30: 
./src/video/x11/SDL_x11dyn.h:45: fatal error: X11/extensions/XShm.h: No such file or directory
compilation terminated.
make: *** [build/SDL_x11dga.lo] Error 1

After googling for the “X11/extensions/XShhm.h” error I saw some other posts on project pages that seemed to indicate that maybe I was missing some X11 dev headers. Instead of weeding through to find the exact package, I took the sledgehammer approach and installed the entire xorg-dev package

sudo apt-get install xorg-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following extra packages will be installed:
  libdmx-dev libdmx1 libexpat1-dev libfontconfig1-dev libfontenc-dev libfreetype6-dev libfs-dev libpciaccess-dev libpixman-1-dev libxaw7-dev libxcomposite-dev libxcursor-dev
  libxdamage-dev libxext-dev libxfixes-dev libxfont-dev libxft-dev libxi-dev libxinerama-dev libxkbfile-dev libxmu-dev libxmu-headers libxmuu-dev libxpm-dev libxrandr-dev
  libxrender-dev libxres-dev libxss-dev libxtst-dev libxv-dev libxvmc-dev libxxf86dga-dev libxxf86vm-dev x11proto-bigreqs-dev x11proto-composite-dev x11proto-damage-dev
  x11proto-dmx-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-fonts-dev x11proto-gl-dev x11proto-randr-dev x11proto-record-dev x11proto-resource-dev x11proto-scrnsaver-dev
  x11proto-video-dev x11proto-xcmisc-dev x11proto-xext-dev x11proto-xf86bigfont-dev x11proto-xf86dga-dev x11proto-xf86dri-dev x11proto-xf86vidmode-dev x11proto-xinerama-dev
  xserver-xorg-dev
The following NEW packages will be installed:
  libdmx-dev libdmx1 libexpat1-dev libfontconfig1-dev libfontenc-dev libfreetype6-dev libfs-dev libpciaccess-dev libpixman-1-dev libxaw7-dev libxcomposite-dev libxcursor-dev
  libxdamage-dev libxext-dev libxfixes-dev libxfont-dev libxft-dev libxi-dev libxinerama-dev libxkbfile-dev libxmu-dev libxmu-headers libxmuu-dev libxpm-dev libxrandr-dev
  libxrender-dev libxres-dev libxss-dev libxtst-dev libxv-dev libxvmc-dev libxxf86dga-dev libxxf86vm-dev x11proto-bigreqs-dev x11proto-composite-dev x11proto-damage-dev
  x11proto-dmx-dev x11proto-dri2-dev x11proto-fixes-dev x11proto-fonts-dev x11proto-gl-dev x11proto-randr-dev x11proto-record-dev x11proto-resource-dev x11proto-scrnsaver-dev
  x11proto-video-dev x11proto-xcmisc-dev x11proto-xext-dev x11proto-xf86bigfont-dev x11proto-xf86dga-dev x11proto-xf86dri-dev x11proto-xf86vidmode-dev x11proto-xinerama-dev xorg-dev
  xserver-xorg-dev
0 upgraded, 55 newly installed, 0 to remove and 0 not upgraded.

Overkill perhaps?

Running: (Note, the instructions for Wing3D say to use the config options below… see BUILD.unix in the project root for more information).

sudo make clean
sudo ./configure --disable-video-aalib --disable-video-directfb 
sudo make
sudo make install

… worked just fine! So problem solved.

Next step is to install ESDL. An erlang binding package to SDL. For this I grabbed the source from the github.com repo.

sudo git clone git://github.com/dgud/esdl.git
sudo git checkout esdl-1.0.1

I read the Readme file here and it said you’ll need glu-1.2 which is an OpenGL utility, and the dev headers for that too. I found freeglut3-dev to be the closest and recommended match so I installed that.

sudo apt-get install freeglut3-dev

Installing ESDL worked fine (no ./configure required).

sudo make
sudo make install

Remember to checkout out the BUILD.unix file for more information on how to compile Wings3D.

Set the environment variable ESDL_PATH to point to top of the compiled ESDL installation. Put this in your ~/.bashrc file. Like this:

export ESDL_PATH=/usr/local/lib/erlang/addons/esdl-1.0.1

Now back to compiling Wings3D! Change back to the src folder and run

sudo make

Error. Well, I’m a stuck now. I’ve tried this on two computers now and I keep getting this error:

brian@cifra:~/3Ddev$ cd wings-1.3.1/
brian@cifra:~/3Ddev/wings-1.3.1$ sudo make
[sudo] password for brian: 
WINGS_VSN = 1.3.1
(cd intl_tools; make)
make[1]: Entering directory `/home/brian/3Ddev/wings-1.3.1/intl_tools'
make TYPE=opt common
make[2]: Entering directory `/home/brian/3Ddev/wings-1.3.1/intl_tools'
erlc  -W +debug_info +debug_info -o. tools.erl
make[2]: Leaving directory `/home/brian/3Ddev/wings-1.3.1/intl_tools'
make[1]: Leaving directory `/home/brian/3Ddev/wings-1.3.1/intl_tools'
(cd src; make)
make[1]: Entering directory `/home/brian/3Ddev/wings-1.3.1/src'
make TYPE=opt common
make[2]: Entering directory `/home/brian/3Ddev/wings-1.3.1/src'
erlc -pa /ebin  -I /include -I ../e3d -W +debug_info '-Dwings_version="1.3.1"' -pa             ../intl_tools  -o../ebin wings_lang.erl
erlc -pa /ebin  -I /include -I ../e3d -W +debug_info '-Dwings_version="1.3.1"' -pa ../intl_tools  -o../ebin array.erl
erlc -pa /ebin  -I /include -I ../e3d -W +debug_info '-Dwings_version="1.3.1"' -pa ../intl_tools  -o../ebin user_default.erl
erlc -pa /ebin  -I /include -I ../e3d -W +debug_info '-Dwings_version="1.3.1"' -pa ../intl_tools  -o../ebin wings.erl
./wings.erl:316: undefined macro 'KMOD_CTRL'
./wings.erl:341: undefined macro 'KMOD_CTRL'
./wings.erl:1442: undefined macro 'SDL_RELEASED'
./wings.erl:1461: undefined macro 'SDL_RELEASED'
./wings.erl:1876: undefined macro 'KMOD_CTRL'
./wings.hrl:17: can't find include lib "esdl/include/sdl.hrl"
./wings.hrl:18: can't find include lib "esdl/include/sdl_events.hrl"
./wings.hrl:19: can't find include lib "esdl/include/sdl_video.hrl"
./wings.hrl:20: can't find include lib "esdl/include/sdl_keyboard.hrl"
./wings.hrl:21: can't find include lib "esdl/include/sdl_mouse.hrl"
./wings.hrl:22: can't find include lib "esdl/src/sdl_util.hrl"
./wings.hrl:31: can't find include lib "esdl/include/gl.hrl"
.... lots of 'undefined'/'cannot find' type messages removed ...
./wings.erl:1846: Warning: function hotkey_select_setup/2 is unused
make[2]: *** [../ebin/wings.beam] Error 1
make[2]: Leaving directory `/home/brian/3Ddev/wings-1.3.1/src'
make[1]: *** [opt] Error 2
make[1]: Leaving directory `/home/brian/3Ddev/wings-1.3.1/src'
make: *** [all] Error 2

This looks like a problem with the $ESDL_PATH is not being picked up by the ‘make’. I’ve set this path several different ways and did not make progress.

brian@cifra:~/3Ddev/wings-1.3.1$ echo $ESDL_PATH
 /usr/local/lib/erlang/addons/esdl-1.0.1

So, I’m not sure what to do at this point. I found on this wiki an example of how to set the path for ESDLPATH http://en.labs.wikimedia.org/wiki/Wings3D:UserManual/Installation Seems like this is how I’ve set it. Perhaps this is ESDL_PATH thing is a red herring.