Heads up!
The blog has moved!
If you are interested in reading new posts, the new URL to bookmark is http://blog.valeriogheri.com/
If you need to build an API, you want it done quickly and simply and you love JavaScript, then this article is for you!
I’m going to show a simple, quick and effective way to build and unit test your API and your routes, using Node.js, Express and a few amazing test libraries: Mocha, SuperTest and should.js that integrate very well together in a natural fashion.
The API style is Rest, that is it leverages URLs and HTTP verbs, but I won’t go much into details in this tutorial.
If you ever used technologies such as WCF, you will be amazed at how much quicker it is to build your API with Node, there’s virtually no machinery to put in place, no service references to update, no WSDL… it’s pretty much just logic to implement!
Of course when you want to design your API around a Rest concept, it is very important to think about the access paths to your resources, they NEED to make sense and to be well structured, otherwise you just end up walking down the good old RPC path.
For a nice intro to this concept, see http://www.slideshare.net/domenicdenicola/creating-truly-res-tful-apis
By the way, in this tutorial I won’t cover advanced topics such as securing the API through HMAC or OAuth solutions.
From words to code:
A nice and free dev environment where to start coding is cloud9 (https://c9.io), as it offers everything you need and maybe more: a text editor with syntax highlighting, a Node.js environment, a debugger (!), a shell, cloud storage and it connects to your GitHub repository… because you have a GitHub repository, don’t you? We don’t need more than that to write our first API, so let’s start!
Ok we want to write a HTTP API, so we obviously want our API to be accessible via some sort of webserver, but as you may already know, Node.js doesn’t come with its own webserver.
Instead it comes with some facilities built-in; on top of these facilities lies Express.js, the web server of choice for this tutorial. Express.js (http://expressjs.com/) describes itself as a web application framework for node, and I think you should use Express too, as it is a solid and well tested piece of software, easy to setup and test.
After reading some tutorials around, I came up with a structure of 4 short files that setup and run the API:
-
config.js exports an object that contains all configuration keys and values, pretty much like web.config in an Asp.Net web app. Following this line we can think about having a config-debug.js and a config-release.js. We’ll see later why.
-
routes.js exports a setup function that takes care of declaring the routes and the handlers that will manage each request.
-
server.js where we configure Express and we export a start function that takes care of setting up the routes and starting the web server
-
index.js is the entry point of our API. It uses pretty much everything we have defined so far to start the logger (my logger of choice is winston), to connect to a database, if present, and to finally start the Express server.
config.js
module.exports = { | |
"db": { | |
"mongodb": "mongodb://username:password@dsXXXXX.mongolab.com:45077/databasename" | |
}, | |
"logger": { | |
"api": "logs/api.log", | |
"exception": "logs/exceptions.log" | |
} | |
}; |
routes.js
function setup(app, handlers) { | |
app.post('/api/profiles', handlers.account.createAccount); | |
app.get('/api/profiles/:username', handlers.account.getAccount); | |
app.put('/api/profiles/:username', handlers.account.updateAccount); | |
app.del('/api/profiles/:username', handlers.account.deleteAccount); | |
app.post('/api/lists', handlers.list.createShoppingList); | |
app.post('/api/lists/:id', handlers.list.createShoppingList); | |
app.put('/api/lists/:id', handlers.list.updateShoppingList); | |
app.get('/api/lists/:userId', handlers.list.getShoppingLists); | |
app.del('/api/lists/:id', handlers.list.deleteShoppingList); | |
} | |
exports.setup = setup; |
As you can see, setup expects two parameters: app is the Express app and handlers is an object that maps to a set of functions that handles user requests. Each of these functions accepts as parameters the request and response objects: e.g.
function handleCreateAccountRequest(req, res) { … }
server.js
// ******************************************************* | |
// expressjs template | |
// | |
// assumes: npm install express | |
// defaults to jade engine, install others as needed | |
// | |
// assumes these subfolders: | |
// public/ | |
// public/javascripts/ | |
// public/stylesheets/ | |
// views/ | |
// | |
var express = require('express'); | |
var app = express(); | |
var AccountHandler = require('./handlers/AccountHandler'); | |
var ShoppingListHandler = require('./handlers/ShoppingListHandler'); | |
var routes = require('./Routes'); | |
var fs = require('fs'); | |
var expressLogFile = fs.createWriteStream('./logs/express.log', {flags: 'a'}); | |
// Configuration | |
app.configure(function(){ | |
app.use(express.logger({stream: expressLogFile})); | |
app.use(express.bodyParser()); | |
app.use(express.methodOverride()); | |
app.use(app.router); | |
app.use(express.static(__dirname + '/public')); | |
}); | |
app.configure('development', function(){ | |
app.use(express.errorHandler({ dumpExceptions: true, showStack: true })); | |
}); | |
app.configure('production', function(){ | |
app.use(express.errorHandler()); | |
}); | |
var handlers = { | |
account: new AccountHandler(), | |
list: new ShoppingListHandler() | |
}; | |
function start() { | |
routes.setup(app, handlers); | |
var port = process.env.PORT || 3000; | |
app.listen(port); | |
console.log("Express server listening on port %d in %s mode", port, app.settings.env); | |
} | |
// ******************************************************* | |
exports.start = start; | |
exports.app = app; |
The first part is just some standard Express setup + Express logging.
Now we can see clearly what the parameters for routes.setup are: app is just the Express instance and handlers contains two objects that point to the different handlers that will be able to handle API requests.
Finally, after declaring and exporting the start function, this module exports the Express instance, which will be used in the tests.
index.js
var config = require('./Config-debug'); | |
var winston = require('winston'); | |
var mongoose = require('mongoose'); | |
var server = require('./Server'); | |
// We will log normal api operations into api.log | |
console.log("starting logger…"); | |
winston.add(winston.transports.File, { | |
filename: config.logger.api | |
}); | |
// We will log all uncaught exceptions into exceptions.log | |
winston.handleExceptions(new winston.transports.File({ | |
filename: config.logger.exception | |
})); | |
console.log("logger started. Connecting to MongoDB…"); | |
mongoose.connect(config.db.mongodb); | |
console.log("Successfully connected to MongoDB. Starting web server…"); | |
server.start(); | |
console.log("Successfully started web server. Waiting for incoming connections…"); |
As already said, index.js is the entry point, that means our API will be executed by running the command
$ node index.js
We can easily configure the API logger using winston and the configuration file, we connect in this case to MongoDB using mongoose, a fantastic tool, and then we start the server using the freshly exported function.
That’s it, you don’t need anything else to setup your API along with a solid logger and a database connection.
Ok I have an API but… how do I test it?
The test framework of choice for this tutorial is Mocha (http://visionmedia.github.io/mocha/) along with Supertest for HTTP assertions (https://github.com/visionmedia/supertest) and Should.js for BDD style tests (https://github.com/visionmedia/should.js). Mocha is a great choice because it makes async test natural and fluent.
The purpose here is to unit test our routes (integration tests) and to build the tests so that we will be able to read our test and the results as if they were plain English sentences!
Let’s create a folder named “test” where we will place our test files and our mocha.opts configuration file where we can specify the output style for our tests.
Finally we can create our unit tests for the routes in a file called “test/routes.js”.
The structure of a test with Mocha is simple and verbose:
var should = require('should'); | |
var assert = require('assert'); | |
var request = require('supertest'); | |
var mongoose = require('mongoose'); | |
var winston = require('winston'); | |
var config = require('./config-debug'); | |
describe('Routing', function() { | |
var url = 'http://someurl.com'; | |
// within before() you can run all the operations that are needed to setup your tests. In this case | |
// I want to create a connection with the database, and when I'm done, I call done(). | |
before(function(done) { | |
// In our tests we use the test db | |
mongoose.connect(config.db.mongodb); | |
done(); | |
}); | |
// use describe to give a title to your test suite, in this case the tile is "Account" | |
// and then specify a function in which we are going to declare all the tests | |
// we want to run. Each test starts with the function it() and as a first argument | |
// we have to provide a meaningful title for it, whereas as the second argument we | |
// specify a function that takes a single parameter, "done", that we will use | |
// to specify when our test is completed, and that's what makes easy | |
// to perform async test! | |
describe('Account', function() { | |
it('should return error trying to save duplicate username', function(done) { | |
var profile = { | |
username: 'vgheri', | |
password: 'test', | |
firstName: 'Valerio', | |
lastName: 'Gheri' | |
}; | |
// once we have specified the info we want to send to the server via POST verb, | |
// we need to actually perform the action on the resource, in this case we want to | |
// POST on /api/profiles and we want to send some info | |
// We do this using the request object, requiring supertest! | |
request(url) | |
.post('/api/profiles') | |
.send(profile) | |
// end handles the response | |
.end(function(err, res) { | |
if (err) { | |
throw err; | |
} | |
// this is should.js syntax, very clear | |
res.should.have.status(400); | |
done(); | |
}); | |
}); | |
it('should correctly update an existing account', function(done){ | |
var body = { | |
firstName: 'JP', | |
lastName: 'Berd' | |
}; | |
request(url) | |
.put('/api/profiles/vgheri') | |
.send(body) | |
.expect('Content-Type', /json/) | |
.expect(200) //Status code | |
.end(function(err,res) { | |
if (err) { | |
throw err; | |
} | |
// Should.js fluent syntax applied | |
res.body.should.have.property('_id'); | |
res.body.firstName.should.equal('JP'); | |
res.body.lastName.should.equal('Berd'); | |
res.body.creationDate.should.not.equal(null); | |
done(); | |
}); | |
}); | |
}); | |
}); |
So if you try and read the test, it would come out something like:
Describe Routing, describe Account, it should return error trying to save duplicate username, it should correctly update an existing account, and so on.
To add more test, simply add more describe or it functions, and that’s it!
To execute the tests, start your api with
$ node index.js
and then in another shell run
$ mocha
as by default mocha will run everything in /test off of your main project.
If you want to run your test by typing
$ npm test
then all you have to do is to create a makefile and can even be as follows:
test: | |
@./node_modules/.bin/mocha | |
.PHONY: test |
then add the following lines to your package.json file:
“scripts”: {
“test”: “make test”
}
and that’s the result (note that in the screenshot I have more tests as it’s taken from an actual project of mine)
That’s it, pretty easy, isn’t it?
Wow, this post is nice, my sister is analyzing such things, so I am going to
tell her.
Thanks for your post. I’m just looking how to test a REST API build with node.js. It would be great if the test started up whenever I save a file.
Hi,
well it’s certainly possible and easy to do: just run your test typing mocha -w or modify your Makefile like the following
test:
@./node_modules/.bin/mocha
test-w:
@NODE_ENV=test ./node_modules/.bin/mocha \
--watch
.PHONY: test test-w
and then you can run your tests with the following command: make test-w or if you prefer using npm test then modify the package.json
"scripts": {
"test": "make test",
"test-w: "make test-w"
}
Let me know if this works for you (it should).
Valerio
Hi,
Again, thank for you answer. I will try it.
Perhaps, there is a way to simplify the test. Instead of “request(url)”, you could use “request(server)” as described in the supertest exemple on github. So you won’t have to start the api with “node index.js” before testing.
Rémi
There is another issue: if you use mocha watch with mongoose, you will bump into this error: “OverwriteModelError: Cannot overwrite `***` model once compiled.”
The issue is documented but not solved on: https://groups.google.com/forum/?fromgroups=#!topic/mongoose-orm/PXTjqqpaDFk
Uhm, it seems like a few possible solutions are in this page https://github.com/LearnBoost/mongoose/issues/1251
Otherwise like you said, nodemon is probably the way to go.
Do you have an article which explains how to secure the API using a simple API Key?
Hello, unfortunately not yet.
The project I’m currently working on will use Facebook OAuth to secure each API endpoint, but if you want something different, you can follow this:
http://www.thebuzzmedia.com/designing-a-secure-rest-api-without-oauth-authentication/
Thanks – big help as I’ve just started a node.js/express RESTful API project.
One question – what’s…
var server = require(‘../Server.js’);
…doing in your test file? (And why the uppercase ‘S’).
Hello,
thanks for your kind comment!
You’re right it’s a mistake of mine, I forgot to remove it from the test file.
I think it was some experiment to start the API directly from the test file (and yes).
Thanks man! Excellent post!
Thanks 🙂
HI,
Can u send me the mocha unit test sample for nodejs REST web service .I am using GET method and that service returns json output.
Internally that service connects the oracle DB
Hello,
You can find an entire mocha unit test sample in this repository of mine: https://github.com/vgheri/ShopWithMe
and here https://github.com/vgheri/ShopWithMe/tree/master/test
Let me know if you have more questions
How would you suggest testing not so linear API which has more error handling and possible error occurrences.
Hello, thanks for your comment!
Well about integration tests, I’d say that no matter how complex your API is, you have a finite set of HTTP Status codes, so you can write (at least) one test per return status code.
If you are dealing with unit tests, try to keep each unit as simple as possible, so that it’s easier to test.
Does this answer to your question?
Hi, thanks for this excellent article.
I’m having a bit of difficulty understanding the way you instantiate the different handlers. Could you give an example of how you’d code one of them ?
Or better yet, do you have the full code of this article ?
Thanks
Hello Luc, I’m sorry I didn’t make my self clear enough.
Please check my repository located at for the full code https://github.com/vgheri/ShopWithMe
An handler is just a plain javascript object, defined for example as:
var AccountHandler = function() {
this.createAccount = handleCreateAccountRequest;
this.getAccount = handleGetAccountRequest;
this.updateAccount = handleUpdateAccountRequest;
this.deleteAccount = handleDeleteAccountRequest;
};
Then the module exports this function as:
module.exports = AccountHandler;
Then in your module which defines the start function of your server, you can do something like:
var handlers = {
account: new AccountHandler()
// Here you can instantiate all your handlers
};
function start() {
routes.setup(app, handlers);
var port = process.env.PORT || 3000;
app.listen(port);
console.log("Express server listening on port %d in %s mode", port, app.settings.env);
}
Where routes is the module that defines your API routes. The handlers are injected into it as follows:
function setup(app, handlers) {
app.post('/api/profiles', handlers.account.createAccount);
...
}
exports.setup = setup;
I hope this helps,
Valerio
Thanks Valerio,
That’s what I ended up doing, but I wanted to make sure there wasn’t something else hiding underneath 🙂
My best regards
Hi, thanks for the post. I have a pretty similar setup for testing.
The issue I have now is that my API calls a third party API (google’s), and I would like to mock this third party call (so that I can run my test offline and use fake access tokens for the purpose of testing).
I’ve looked into nock, but I feel like it only mocks the http calls for the test file, not for the API.
How would you do that?
Thanks!
Hello Martin, thanks for you comment!
Mmm, maybe you can create a module that on API startup sets up nock to intercept the HTTP calls you want to mock, and then when from your test file you will request your API method, then that method will use the nock result.
Because you don’t want this code to be there when you package your code for production, you may want to use GRUNT to automate it.
Did I understand your question? What do you think about this solution?
Thanks a lot for your response! I’ve just seen it.
Yes you perfectly understood my question. I was thinking of a similar solution in the meantime but I hoped there would be a better way. But hey, if you are thinking the same, it is probably the best way 🙂
I’m already using Grunt on the front-end (angularJS app) but yeah, using it on the backend too for that purpose is a good suggestion.
Thanks again
Martin
Can you explain the purpose of establishing a db connection in before() in the test? It seems like everything in the test happens via the API. Why does the test need a connection?
You save my life. Thanks!!
awesome!!!
nice. do u have example of SignIn and signOut Process in Android to Nodejs
Nice…I’ve been using express 4 for restful…but since the urge of using the Express 3, i found this is cool
how to implement logout functionally both in mobile (android) and server (nodejs) ?
Great article, thanks. Though I think res.should.have.status(400); is outdated. res.should.have.property(‘status’, 400); worked for me
Thanks for this post! It was very helpful and really straight forward.
Thansk for sharing this. While implementing tests aligned to your post, I recognized require(‘should-http’); to be missing. Maybe you want to add this.
Do future people a favor mate and leave a note near your config.js that the port number you’re using isn’t default for a local mongo instance.