Tag Archives: nodejs

io.js 1.0.1 (unstable) is out. ES6!

A team of developers, including some core node.js developers, forked node.js a while ago into a project called io.js. They’ve just released version 1.0.1 (Unstable) and it brings ES6 natively to server-side JavaScript applications.

To test drive, you can download io.js. Be warned, it will completely replace your node and npm system links. Some people have complained about this or reported unexpected behavior after compiling io.js from source.

“Why io.js?” you ask? Joyent was a little stingy with node.js progress (it’s the basis for a huge money-maker for them). The io.js team’s goal is to make io.js more of a community-driven project. It seems to have worked, judging by the progress they’ve made.

Version 1.0.0 of io.js ships with V8 3.31.71.4, which includes ES6 features well beyond version 3.26.33 that will be shipped with joyent/node@0.12.x.

No more –harmony flag

With io.js, you can now start using some ES6 features natively (e.g. no transpiling using Traceur or 6to5). Here’s the fibonacci example using an ES6 iterator taken from 6to5’s tour:

// fib.js
var fibonacci = { };
fibonacci[Symbol.iterator] = function*() {
  var pre = 0, cur = 1;
  for (;;) {
    var temp = pre;
    pre = cur;
    cur += temp;
    yield cur;
  }
}

for (var n of fibonacci) {
  // truncate the sequence at 1000
  if (n > 1000) {
    break;
  }
  process.stdout.write(n + '\n');
}

You’ll notice that I’ve had to change how an iterator is constructed and how the output is written. Run this with node fib.js (remember, io.js will overwrite the system node).

While this is excellent news, you’ll need to be aware of what ES6 features are currently supported. For example, io.js does not yet support classes. This won’t work in io.js:

class Animal{
  constructor(legs) {
    switch (legs) {
      case 2:
        this.locomotion = 'bipedal';
        break;
      case 4:
        this.locomotion = 'quadrupedal';
      default:
    }
  }
  move() {
    process.stdout.writeln("I'm a %j animal!", this.locomotion);
  }
  static getTypeName() {
    return "Animal";
  }
}

class Bird extends Animal {
  constructor(){
    super(2);
  }
}

class Lion extends Animal {
  constructor(){
    super(4);
  }
}

var bird = new Bird();
bird.move();

var lion = new Lion();
lion.move();

This will, however, work via transpilers:

These examples are on github.

Flattr this!

Your First App: Node.js is complete!

I’m stoked to announce that I’ve finished writing my first self-published book, Your First App: node.js.

The book is a full-stack application development tutorial using what is commonly known as the ‘MEAN’ stack, but with a heavier focus on the backend technologies: node.js, express.js, and MongoDB. The book ends with a starter AngularJS project of an HTML5 instant messaging chat application.

While following along in the book, it may be useful to run test queries in a REST client. Postman is available for Google Chrome as a packaged application. Import `yfa-nodejs.postman_dump.json` (located in the root of the supplemental code’s repository) into Postman to evaluate the application’s API.

Check out the code on GitHub

Flattr this!

Your first app: Node.js (update)

I am currently writing a book called Your first app: Node.js. I am self-publishing this book through leanpub.com. You can check it out here.

One cool thing about self-publishing on leanpub is that people can purchase the book early and receive all future updates as I write. Feedback from this process may also help me to revisit areas I’ve unintentionally left a little light or not explained very well.

The book is currently about 66% completed. I recently completed the chapters through the node.js API. To complement that API, I will be writing a chapter on creating an AngularJS front-end. Then I’ll explain how to deploy your application to Digital Ocean (although the steps will be similar for many other hosts). Finally, I will provide some suggestions for further learning.

Below is the book description as shown on leanpub.

ABOUT THE BOOK

This book is intended for professionals or students who have a comfortable working knowledge of JavaScript, HTML, CSS, and the stateless nature of HTTP. If you’re not comfortable with JavaScript, I highly recommend reading both JavaScript: The Definitive Guide and JavaScript Garden. Although I’ll quickly go over some necessary fundamentals for beginners, it will be difficult to become a ‘node.js developer’ without a solid foundation in JavaScript. HTML, CSS, and client-side JavaScript will be necessary for the presentation of your application, while understanding the stateless nature of HTTP will make some of the techniques discussed in this book easier to learn.

Why this book?

Many books on a given technology focus on really digging into a certain framework or a few simple concepts. Node.js is an interesting technology for many reasons, including:

  • it popularized server-side JavaScript
  • encourages modularization
  • has a fairly large community
  • provides amazing throughput for data

I have read many technology books and yet I’ve always felt like there was a gap between books covering a given technology and how you’d actually undertake creating a full application on your own. When I realized that node.js is attracting a lot of front-end developers and hobbyists to the community, I understood what that missing “gap” was: the application life cycle. For developers without a degree in Computer Science or Information Systems, it may be difficult to figure out how to actually start creating a full application. I hope to address this in the text by providing some simple guidelines that help to lay out a clear path between the starting point and the finished product. The intended audience for this book is anyone familiar with HTML, CSS and JavaScript. You don’t need to be a professional engineer or even have years of experience. In early chapters, I discuss the aspects of JavaScript that I feel are crucial to understanding how to write an application in node.js.  Where I feel newer developers may benefit from further reading, I suggest online or print resources that I’ve found helpful.

What makes this book different?

When working with Junior developers, I try to avoid the sink or swim mentality that many Seniors seem to have.  Instead, I like to explain why I’ve chosen one thing over another. In this book, I include an entire chapter to discuss development considerations which are usually omitted from mainstream texts.  For instance, when was the last time you read a book that suggested a build automation tool? I can’t remember any book that clearly explained why a sample application was written to use MongoDB instead of Redis or MySQL. These are things I attempt to cover in this book before diving in to the meat of the text you’d find in similar books. I even have a short prototyping chapter for readers who are unfamiliar with the technologies used in the book to try some hands-on code before applying this knowledge to the sample application.

These are all things people with professional full-stack (e.g. client-server-database) experience do on a regular basis. With such a fresh, young community being attracted to node.js application development (many without server-side or database experience) I think this book’s early focus on process and maintainability are important aspects for any beginner to pick up.

Technologies

Writing an application requires a developer to pick up skills in multiple technologies for frameworks. I plan to at least cover the basics of all the tools necessary to create a full node.js application. Some of the technologies covered in this book are:

  • express.js (node.js)
  • grunt.js (node.js)
  • mocha (node.js)
  • Twitter’s Bootstrap (CSS)
  • angular.js (client JavaScript)
  • MongoDB (database)
  • git (source control)

The intention is not to fully teach a developer everything about every one of these technologies, but I will go over the basics for everything that is not considered node.js and go over the node.js technologies in greater detail.

Planned Chapters

Future chapters will focus on:

  • ✓ git workflow
  • ✓ project structure and setup
  • ✓ setting up a database
  • ✓ modeling a database “schema”
  • ✓ creating a minimal API
  • creating a front-end application
  • asynchronous communication between client and server
  • ✓ authentication strategies
  • … more

As this book is written, I welcome your feedback and comments at nodejs@ipreferjim.com

Status

This book is currently about 66% complete.

The final product is estimated to be between 200 and 250 pages in length (not including appendices).

Sample Code

Sample code for this book is located on github at jimschubert/yfa-nodejs-code.

Flattr this!

What is node.js?

I’ve recently decided to write a book about node.js. I don’t feel like a lot of books out there (on any technology) cover real-world scenarios. Most of them are introductions to languages or deep-dives into a single aspect of a language or technology. There are very few books that cover real-world application development scenarios to include things like process, build, sample application, and deployment. A good example of the type of book I’d like to write is Agile Web Development with Rails.

Below is an excerpt from one of the first chapters of the book. This is *very* rough draft, but I thought I would share because it may be useful for anyone beginning in node.js development.


What is node.js?

Node.js is a server-side JavaScript environment which provides evented asynchronous I/O and a minimalistic set of core features. In the beginning, node.js combined v8 with libev. Unfortunately, libev is an evented I/O library for Unix-based systems only. Eventually, libev was replaced with a higher-level abstraction library called libuv which allows node.js to run on Unix-based systems and Windows. In 2011, Microsoft began contributing to node.js to bring it to the Windows platform. More recently, libuv removed libev as a dependency.

The key concept for node.js is evented I/O (mostly network-based) operations. This is handled via an event loop. Although understanding the implementation details of event loops can be pretty complicated, the idea is fairly simple. Let’s analogize.

Try to think about an application as a fast food restaurant. A cashier is patiently waiting at the register. A customer walks up to the register and places an order. The cashier hands the order off to the kitchen and maybe fulfills the customer’s drink order before returning to the register. Another customer arrives and places an order, initiating the same scenario as before. A third customer is walking to the register, but just before the customer reaches the register a cook dings a bell and the cashier steps away to grab the first order and hand it to the customer. The cashier then begins taking the third customer’s order. Midway through the order, the second customer’s order is ready. Because the cashier is not rude (maybe this is in Canada?), the third order must be completed before the second customer’s order can be handed back to the customer. But, the cashier can easily hand over the second customer’s order before filling the third customer’s drink order. Eventually, the third customer’s order is fulfilled and there are no more customers. The cashier waits patiently.

It may sound strange, but the above scenario is a simplification of an event loop. We see things like this every day. New node.js developers seem to have a hard time either understanding the event loop or understanding where it comes from. Take the following example, for instance:

function main(){
  console.log("Hello, world!");
}

main();

There’s a little more going on in this example than the standard obligatory printing of “Hello, world!” I’ve wrapped the output line in a function called main and subsequently called that function. If you run this file, you’ll see that it prints out “Hello, world!” and exits. There is no event loop because we haven’t told the runtime to listen to or emit any events.

There are a couple ways we can initiate an event loop. Possibly the easiest way to visualize this is to use the standard JavaScript setInterval function to cause our application to write out “Hello, world!” every 250ms. To kill this application, you’d need to press Control-c.

function main(){
  console.log("Hello, world!");
}

setInterval(main, 250);

The above example is technically an event loop, although there aren’t really any events going on other than the setInterval function. Another way to initiate an event loop is to bind to or set up any event which interacts in some way with a file handle. In Unix, nearly everything is considered a file, so this would mean binding to a socket, file descriptor, or even screen input:

// respond to the SIGINT event
process.on('SIGINT', function(){
    process.stdout.write('\nHandling SIGINT event!\n');
    // cleanly exit with non-error status
    process.exit(0);
});

// listen to events on standard input
process.stdin.setEncoding('utf8');
process.stdin.on('data', function(chunk) {
  process.stdout.write('Hello, world!\n');
});

// Start reading from stdin so we don't exit.
process.stdin.resume();
console.log('Type anything and press ENTER...');

In the above example, process.stdin.resume() causes node.js to continually monitor the readable standard input stream for data. Binding a listener to watch for ‘SIGINT’ does not cause node to create an event loop on its own. Play around with this last example to get an idea of how the event loop works. If you’re interested in how libuv creates and maintains an event loop, there is a free uvbook available which dives into the internals of libuv.

Flattr this!

Karma error: RangeError: Maximum call stack size exceeded

I’ve been working on a bootstrap application for express.js, angular.js, and bootstrap. The idea being that I can provide a boilerplate for quick prototyping of applications.

I was having some issues setting up karma, the test running used by angular.js. I was receiving an error:

...
	
INFO [Chrome 28.0 (Linux)]: Connected on socket id 2uhtx5qf6HDM8_A1qssQ
INFO [karma]: Delaying execution, these browsers are not ready: Chrome 28.0 (Linux)
Chrome 28.0 (Linux) controllers should .... FAILED
	RangeError: Maximum call stack size exceeded
	    at String.match (native)
	    at Object.b.event.remove (/home/jim/projects/express-angular-bootstrap-boiler/src/public/javascripts/vendor/jquery-1.9.1.min.js:3:25827)

...

I messed around with the file includes for a while, then Googled a bit for any possible issues.

It turns out that I was using bootstrap v 2.3.0, which appears to have issues with jQuery 1.9.1 as reported in this issue.

As noted in the issue, upgrading to a newer version of Twitter’s Bootstrap (2.3.2) resolved the issue.

Flattr this!

git prepare-commit-msg with node.js

I think people often overlook using node.js as a scripting tool like ruby or python. If you search the web for examples of customized git hooks, you’ll probably have visited git-scm.com, where most of the examples are in ruby.

I wanted to show a pretty simple prepare-commit-msg hook written in node.js. This hook will append the name and description of whatever working branch you commit from. I use this technique to create a working branch for a JIRA ticket number, so all my commits to this ticket automatically include that ticket number. Then, JIRA can be configured to display github commits directly in the ticket details when that commit references the ticket in the commit message.

The script:

#!/usr/bin/env node
var exec = require('child_process').exec,
    util = require('util'),
    fs = require('fs'),
    contents = null,
    branch, desc;

console.log(process.argv);

// expect .git/COMMIT_EDITMSG
if(/COMMIT_EDITMSG/g.test(process.argv[2])){
    // look for current branch name
    branch = exec("git branch | grep '*'",
      function (err, stdout, stderr) {
        if(err){
			// git branch will fail if initial commit has not been done,
			// so we can safely skip this hook
			process.exit(0); 
		}

		// opens .git/COMMIT_EDITMSG
        contents = fs.readFileSync(process.argv[2]);
		
		// trims extra characters from start/end of line
        var name = stdout.replace('* ','').replace('\n','');
		
		// If the branch has a description, pull that
        desc = exec('git config branch.'+ name +'.description',
            function(err, stdout, stderr){
				// don't handle errors (because we write out branch anyway)
                
				// we found a description, add to 'name'
				if(stdout){ name = util.format('%s (%s)', name, stdout.replace(/\n/g,'')); }

				// '(no branch)' indicates we are in a rebase or other non-HEAD scenario
                if(name !== '(no branch)'){
				
					// Append branch name to original contents.
                    contents = util.format('%s\n\n:%s\n', contents, name);
					
					// write contents back out to .git/COMMIT_EDITMSG
                    fs.writeFileSync(process.argv[2], contents);
                    process.exit(0);
                } else {
                    process.exit(0);
                }
        });
    });
}

The above script is pretty well-commented. If you decide to use it, you may want to comment out the first console.log which dumps all arguments sent to the script.

Here’s an example of its usage.

Create and initialize a new local git repository.

$ mkdir example
$ cd example
$ git init

Then, copy the above example script into the git hooks directory under .git/hooks/prepare-commit-msg

Now, in the example directory, create and commit a file into the repository:

$ touch first
$ git add .
$ git commit -m 'Initial commit'

You’ll see the array of arguments passed through the custom git hook, but if you issue a git log, you’ll only see the commit message ‘Initial commit’. That’s because this custom hook expects a git branch, but there is no git branch until your initial commit is complete. To verify the script is working:

$ touch second
$ git add .
$ git commit -m 'Second commit'

A git log will now show you a tweaked message:

    Second commit
    
    :master

Now, you can create a branch named ‘ABC-123’ and do another commit:

$ git checkout -b ABC-123
$ touch third
$ git add .
$ git commit -m 'Third commit'

The commit message:

    Third commit
    
    :ABC-123

This hook also allows you to print out the branch description. To test, invoke the branch description editor:

$ git branch --edit-description ABC-123

I’ve added the text ‘Supporting branch for ABC-234 and ABC-567’. Save this buffer according to your editor (:wq if you’re using vim).

Add a fourth file as we have before, then verify with git log:

    Fourth commit
    
    :ABC-123 (Supporting branch for ABC-234 and ABC-567)

Because this script strips newlines from the branch description, you’d need to keep it short. Usually, that’s not a problem.

This shows how to append text to your commit message, but it can easily be modified to replace commonly misspelled words (teh -> the).

This prepare-commit-msg hook is available in my blogs repo on github.

Flattr this!

Why I don’t recommend the Step module [node.js, npm]

I prefer asyc to step. The async module has a cleaner interface, more options, and utilities. It’s a little beefier in size, but those things are beside the point.

Here’s why I don’t really like step. Tell me what you’d expect from this function:

var Step = require('step');

Step(
	function first(){
		console.log('first');
		
		this();
	},
	
	function second(){
		var x;
		console.log('second');
		if(!x){
			// do something that should set x
		}
		return x;
	},
	
	function third(){
		console.log('third');
	}
)

Did you guess:

$ node step-example.js 
first
second

If you’re not familiar with Step, you’ll probably look at that first function and wonder what this(); actually does. Well, it is actually shorthand for next(); (I’m simplifying a little here).

Assuming you’re at least somewhat familiar with asynchronous control flow, you could assume this(); calls the next function. But, what about the second function? It’s returning a value. Does that return from second, or from Step, or from some other function? It returns from second(), but… it passes the value to third. Or, it should. Unless x is undefined, in which case it will be considered an unchained function. Now, I don’t think your variables should regularly be ‘undefined’ at the point of return, but what if you use or someone on your team uses the popular convention of calling a callback as return callback(x);. If the Step module’s convention of tweaking this semantics is ignored, another developer may look at it as a typo, “You can’t call the context like that…” Right? Also, what if someone doesn’t understand the return value can’t be undefined and comments out the middle of that function? You may have cleanup logic that isn’t getting checked in third().
We’ve all seen that happen before.

In the above example, if x was not undefined, it would be passed as a parameter to third().

It’s this inconsistency which makes me feel like Step is an accident waiting to happen in a regular development team. The source code is pretty well written and concise. I think the author has done a great job, but the usage is unintuitive and I wouldn’t recommend using the module.

On the other hand, async is beautifully intuitive.

Consider this example:

var async = require('async');

async.series([
	function first(done){
		console.log('first');
		done(null, 'first');
	},
	function second(done){
		var x;
		console.log('second');
		if(!x){
			// do something that should set x
		}
		done(null, x);
	},
	function third(done){
		console.log('third');
		
		done(null, 'third');
	}
], function(err, results){
	console.log(results);
})

async.series allows us to run a series of tasks (functions defined within the array), which pass a value to the callback (that last function you see).

If you forget to pass a value, the results will contain undefined at the index of the array. Your third function won’t be skipped unless you forget to invoke the callback. To me, this is a pretty obvious error and one that is easy to spot and correct. Here’s is the output of the above call:

first
second
first
third
[ 'first', undefined, 'third' ]

To be fair, the example of Step acts more like a waterfall function. Here’s what that would look like in async.

async.waterfall([
	function first(done){
		console.log('first');
		done(null, 'first');
	},
	function second(val, done){
		var x;
		console.log('second has val: %j', val);
		if(!x){
			// do something that should set x
		}
		done(null, x);
	},
	function(val, done){
		console.log('third has val: %j', val);
		
		done(null, 'third');
	}
], function(err, val){
	console.log('callback has val: %j', val)
});

The above code is available on github: https://github.com/jimschubert/blogs

Flattr this!

CommonJS Modules, node’s require() and private members

CommonJS Modules, node’s require() and private members

Interestingly, node.js module documentation doesn’t even mention CommonJS or the specification proposal it implements. I won’t go over CommonJS Modules 1.0 in-depth in this post, but I suggest you read both of these links if you plan to explore node.js development.

An important take-away from this blog post (so important that I state it first) is that node.js caches modules. A module’s exports are nothing more than an object. When you set a function or property to be exported, you’re setting it on an object.

Conceptually, this:

var someFunction = function(){ /* implement */ };
module.exports.someFunction = exports.someFunction = someFunction;

…is the same as doing this outside of CommonJS modules…

var someFunction = function(){ /* implement */ };
var exports = {};
exports.someFunction = someFuntion;

I know, that seems like it should be common sense. When node.js caches the exports object at the application level, I think understanding this can become hairy — especially if you come from a compiled language background.

For example, consider the following module.

var count = 0;

var doSomethingShared = function(){
	count += 1;
	console.log("Shared call count: %d", count);
};

var doSomethingNotShared = function(){
	console.log("Other modules don't know about me!");
};

module.exports.shared = exports.shared =  doSomethingShared;
module.exports.unshared = exports.unshared =  doSomethingNotShared;

We can require this module easily and play around with it in node REPL:

$ node
> var example = require('./example.js');
undefined
> example.shared()
Shared call count: 1
undefined
> example.shared()
Shared call count: 2
undefined
> example.shared()
Shared call count: 3
undefined
> example.shared()
Shared call count: 4
undefined
> example.unshared()
Other modules don't know about me!
undefined

In the same REPL instance, you can re-require the module and see that ‘count’ is indeed shared:

> var example2 = require('./example.js');
undefined
> example2.shared()
Shared call count: 5
undefined
> example2.shared()
Shared call count: 6
undefined

You can even require the shared function directly:

> var shared = require('./example.js').shared;
undefined
> shared()
Shared call count: 7
undefined
> shared()
Shared call count: 8
undefined

I think you get the point; private variables which aren’t exported are shared. In this example, ‘count’ is a private static property of the module.

Why this matters

This is very important, because when working in the asynchronous environment of node.js you have to be aware about *what* is being cached privately at the module level. In addition to this, if you plan to cache functions for whatever reason, you have to be incredibly sure you know what you’re doing. I recommend to NEVER CACHE CALLBACK FUNCTIONS.

As an example of this statement, I’ve created a sample file processor script in the github repo as fs-example.js. That script reads this blog’s directory in the repo, then loops over the files and executes ‘processor’, which is a custom broken module displaying the above problem. fs-example.js does an inline require, which as you’ve seen above doesn’t make a difference.

The problem I’ll display here exists because async calls are likely to take different lengths of time to execute. My example is using the linux commands ‘head -1’ and ‘tail -1’ on each file to get the first and last lines, respectively. These commands are nearly instantaneous, so I’m using a random setTimeout to display the problem with caching functions. You could easily modify these scripts to ‘head’ and ‘tail’ different web sites. But, whatever. Because head.js and tail.js are simple spawn wrappers, I’ll focus on processor.js, which is where the potential bugginess exists.

// begin processor.js
var head = require('./head'),
	tail = require('./tail');
	
// don't do this IRL
var callback = null;

var processor = function(file, cb){
	if('function' === typeof cb){
		callback = cb;
	}
	
	// these fail
	head(file, function(d){
		callback(d)
	});
	tail(file, function(d){
		callback(d)
	});
	
	// these succeed
	// head(file, callback);
	// tail(file, callback);
};

module.exports = exports = processor;

This is a simple file for having such a subtle bug. The file includes the customized head and tail scripts. Then, it creates a function that tags a file, caches the callback function, and calls head and tail to execute the callback. You’ll notice the commented-out lines at the end that claim ‘these succeed’. Here’s why…

When a callback is passed directly to a function as is done with head(file, callback), the function itself is passed as a reference. Meaning, when the head module executes that callback, it will always execute the referenced callback. Now, what if another developer wants to add logging or other processing of the data? You’ll likely get something like the example, where a function is created and callback is called directly (rather than the cb passed into the function originally). Because ‘callback’ is cached at the module level and shared across calls to processor, the scope created by the processor function is shared with the anonymous function callback and points to whatever is the most recent callback function rather than what one might expect to be the intended function.

We could avoid the problem in the above non-commented code by referencing the ‘cb’ parameter rather than the cached ‘callback’ variable. This is because processor creates a closure over ‘callback’, but ‘cb’ would always be within the current execution context.

You can play around with the code from this post, which is available on github.

Flattr this!

Generating sprites with HTML5 canvas (node-canvas)

About a week ago, I posted about drawing simple shapes in HTML5. HTML5’s Canvas isn’t only useful on the client side.

About a year ago, I rewrote my Quaketracker mashup which pulls rss content from USGS and displays markers on a Google Map. For part of this rewrite, I wanted to use custom markers to indicate the magnitude of the earthquake, both numerically and colorfully. That task was more involved than I realized it would be. There doesn’t seem to be any easy way to generate such a sprite in The Gimp or other graphics software, so I set out to generate a sprite of magnitudes 0.0 through 9.9 with a color range from yellow to orange using node-canvas. This actually made the task very simple.

base.png
marker

Before writing the script, I had to decide on an image. I created a pretty simple quote-bubble-like marker with a white background and a black outline. The image is 35×35 square with some transparency padding the actual marker image. This allows me to calculate the layout pretty easily. The white background on black outline gives me a very specific color to replace with the desired color of intensity.

node-canvas is a custom implementation of Canvas for node.js. When I originally wrote this script, I had a few issues compiling node-canvas. When I tweaked the script today, I had no issues. However, it seems like some things are not well-documented or incomplete (PixelArray).

Scripting it in node.js

A lot of times, if your script writes out to files, it’s best to check for dependencies and cause your script to exit if they’re not met. For example, I know this script is completely useless without node-canvas, so I can probe for the availability of that module and provide directions for installation. I’d do this before attempting to load any other functions from the canvas module. You could also check for versions at this point. If someone runs this script with node-canvas 0.6.0 it will still work, but this may not be the case with other modules. Here’s how I start off the script.

var p = require('util').puts;

try {
    var probe = require('canvas');
} catch(e) {
    p("Cannot find node-canvas module.");
    p("Did you install dependencies:\n");
    p("\tnpm install -d");
    process.exit();
}

var fs = require('fs');
var Canvas = require('canvas');
var Image = Canvas.Image;

var canvas, ctx, baseImage, outImage, img;

I don’t usually use ‘global’ variables like those in the last line of the above snippet for applications or client-side scripting. For a script like this, you could have everything as a global variable as long as the names don’t clash with your imported modules.

To initialize shared variables, I like to define an init() function and call that function at the end of the script. This is pretty common in other languages like Ruby and Python, so why node server-side JavaScript?

var init = function() {
	// our working image is 35x35, we want 10x10 sheet of sprites
	canvas = new Canvas(35*10,35*10);
	ctx = canvas.getContext('2d');
	baseImage = __dirname + '/base.png';
	outImage = __dirname + '/markers.png';

	// pre-load the image
	img = new Image;
	img.onload = function() {
		processImage(fileProcessingComplete);
	};
	
	img.src = baseImage;
};

The cool thing about node-canvas is that the usage is very similar to the Canvas API implemented in browsers supporting the HTML5 Canvas. In fact, aside from the __dirname special property of node.js, you should be able to drop this init function into a browser with no problems. For this script, I want 100 different magnitudes to be generated from 0.0 through 9.9 (00-99 is 100) so the canvas is initialized to 10 rows and 10 columns of 35×35 squares. Just like a client-side Image object, we want to bind any post-processing of the image via the onload() function.

I’m going to Memento you a bit, and for that I’m sorry but the end is the easier concept. When all the processing is complete, we want to pull image data from the Canvas element and write it out to a file.

var fileProcessingComplete = function() {
	var out = fs.createWriteStream(outImage),
            stream = canvas.createPNGStream();

	stream.on('data', function(chunk){
	  out.write(chunk);
	});

	// when PNG stream is done, drain WriteStream
	stream.on('end', function(){
	  p('saved ' + outImage);
	  out.destroySoon();
	});
};

Accessing streams in node.js is usually done asynchronously. We can take chunks of data from the ReadStream of createPNGStream() and pump that to the WriteStream of the output file. When the data from the PNG is done streaming, we tell the output stream to finish writing its buffered data and destroy itself. It sounds intense, but it’s pretty straightforward.

Processing the colors and marker images is the fun part. To match the function call in the init function, we just create a function with a callback in the standard way.

var processImage = function(cb) {
        // other code removed
	if(typeof cb === "function") {
		cb.call(this);
	}
}

If there’s a callback function passed as a parameter, when processImage is finished, it will call that function passing the current scope as the execution scope and no parameters.

To draw out the custom marker, we’ll want to define how we want to display the text and at what color we want to have the 0.0 marker start.

ctx.font = 'normal 12px Impact';
ctx.textAlign = 'center';
var color = [255, 255, 0, 235];

The color array represents red, green, blue, and alpha channel (opacity). This may look odd if you’re used to specifying alpha channels in css as “rgba(255,255,0,0.92)”, but that value of 235 will give roughly 92% opacity.

We can make the script run pretty quickly by using a second canvas 2d context for the recolor phases (which modifies every white pixel).

// use a temp Canvas and Context of 35x35 size.
var tempCanvas = new Canvas(35,35);
var tempCtx = tempCanvas.getContext('2d');

This is an optimization I made today when tweaking the original script. In the original version of this script, I would write the base image to the output canvas then iterate over ‘NxN’ pixels on the output canvas to change the color of one pixel. Using a temporary canvas context, we only have to iterate over 35 pixels wide by 35 pixels high for every new marker. This not only speeds up the process, but it could be a difference of having a script that won’t run on slower machines.

For each of the desired 100 magnitudes, we’ll want to find the x and y values (top-left pixel) to which we’ll write the updated marker. For every 1/2 magnitude, the Green color value decrements by 13 points.

for(var magnitude = 0; magnitude <= 100; magnitude++) {
	var y = 35 * Math.floor(magnitude/10),
		x = ( 35*(magnitude % 10) );

	// This increments the color slightly
	if(magnitude % 5 == 0){
		color[1] = color[1] - 13;
	}

        // some code removed

	ctx.fillText("" + parseFloat(magnitude / 10, 1), x + (35/2), y + (35/2), 35);
}

The last line of the above snippet fills the magnitude text on the output canvas. This could be moved to the next part of the script for a further optimization: instead of just modifying each white pixel’s color, we could recolor and apply the text.

The part of the script which changes the color according to the magnitude looks like this:

tempCtx.drawImage(img, 0, 0, 35,35);
var imgData = tempCtx.getImageData(0, 0, 35, 35);
var data = imgData && imgData.data;
if(data) {
	try {
		for(var pixel=0;pixel<data.length;pixel=pixel+4) {
			var red = data[pixel]
			var green = data[pixel+1];
			var blue = data[pixel+2];
			var alpha = data[pixel+3];

			if(red == 255 && green == 255 && blue == 255) {
				data[pixel] = color[0];
				data[pixel+1] = color[1];
				data[pixel+2] = color[2];
				data[pixel+3] = color[3];
			}
		}
	} catch (err) { console.error(err.message); }

	// Write our temp image data to the final canvas context
	/* imageData, dx, dy, sx, sy, sw, sh */
	ctx.putImageData(imgData,x,y,0,0, 35, 35);
}

Writing to the temporary canvas’s 2d context gives us access to a data buffer which represents a 35×35 image where each pixel is represented by 4 bytes (rgba). Iterating this buffer by 4’s, we can check that the given pixel (excluding alpha because that doesn’t matter in this case) is white or rgba(255,255,255). If the pixel is white, we replace those 4 indexes with the values in our color array. When we’re done iterating the buffer representing the 35×35 image, we can call putImageData on the output canvas. With that, the sprite is complete and our callback handles writing the file.

A note about putImageData

putImageData can be a strange beast. Notice the comment imageData, dx, dy, sx, sy, sw, sh… these are the parameters accepted by the function. Even the HTML5 Specs might make you go, “Uhm… what?” The idea is pretty simple once you understand it. You want to write a buffer (imgData) where the top-left pixel is at dx,dy. sx and sy (or dirtyX and dirtyY) represent a dirty rectangle of size sw x sh (or dirtyWidth by dirtyHeight).

If the imageData passed to the function is the same height and width as the target canvas context, you wouldn’t want to overwrite every single pixel would you? You’d only want to overwrite pixels that have changed. Suppose in this script, I had an imageData object of 350×350 to complement the output canvas. When putting the modified imageData, I would have to calculate the current 35×35 box offset and write out the imageData to 0,0. As an example, if I wanted to overwrite the 35×35 box for the 9.9 magnitude marker with an imageData buffer of 350×350, I might write:

ctx.putImageData(imgData, 0, 0, 315, 315, 35, 35);

This would tell the context that although my buffer is 350×350, I’ve only changed a 35×35 rectangle starting at 315×15. Canvas is smart enough to only update those pixels rather than every pixel in the canvas.

On the other hand, in the script I’ve written, I only have an imageData object which represents a 35×35 buffer. So, I can tell putImageData to start at the point defined by (x,y) and write the buffer into a 35×35 rectangle. I like this method a little more, but it may not always be the case that you’re writing a smaller buffer to the context.

The code

As always, the code for this blog post is available on github at jimschubert/blogs.

The sprite

Here’s what the generated sprite looks like.

Markers generated for Quaketracker

Flattr this!

[node.js] Simple Logging in tweeter.js

tweeter.js is a simple OAuth implementation I started working on for a node.js application I had envisioned. The library works, as in it runs through the whole OAuth dance and allows the client to make API calls. There are some more in-depth things I never got around to (like caching the tokens and clearing sessions and whatnot) before I realized I don’t like Twitter. Oh well.

I was just looking through my github repositories and I thought it would be cool to post about the simple logging mechanism I created for tweeter.js. Although not the coolest pattern I used in the project (which probably goes to mixins), I like how simple the solution is. Check out the code for other cool features..

Logging

The cool thing about this logging framework is that it allows you to write out logs to different streams. For instance, suppose you want to write all logging information other than errors to log.txt, but you want to keep errors in errors.txt.

The way I implemented this was to create an object of streams on the Log object. I used getters/setters to provide validation that an object being assigned to one of the properties is actually a stream. Although I didn’t bother with it, this object could possibly be sealed or frozen.

// An object with safe getters/setters for each stream.  
// Streams are:  
// debug, info, error, warn
Log.prototype.streams = {
    get none() { return { write: function() { }}; },
    get debug() { return this._debug; },
    set debug(val) {
        requiresStream(val);
        this._debug = val;
    },
    get info() { return this._info; },
    set info(val) {
        requiresStream(val);
        this._info = val;
    },
    get warn() { return this._warn; },
    set warn(val) { 
        requiresStream(val);
        this._warn = val;
    },
    get error() { return this._error; },
    set error(val) {
        requiresStream(val);
        this._error = val;
    }
}

Because this is written for node.js, requiresStream() verifies the value is an instance of EventEmitter with a write function.

Defaults for these streams are set in the constructor with a default logLevel of -1 (disabled).

var Log = module.exports = exports = function Log(level) {
    // default error to stderr and the rest to stdout
    this.streams.debug = process.stdout;
    this.streams.info = process.stdout;
    this.streams.warn = process.stdout;
    this.streams.error = process.stderr;

    // default the logLevel to disabled
    this.logLevel = level || -1;
}

An interesting thing about this logger is how I make the calling convention flexible. I use .NET regularly and I would call a logging function in .NET with an enum value to specify the logLevel. I wanted a similar functionality in this logger, so I created a lookup object to map a key (i.e. DEBUG) to a logLevel (0) and a stream (debug).

// [DISABLED] else [DEBUG > INFO > WARN > ERROR]
Log.prototype.logLevels = {
    DISABLED:   { value: -1, stream: 'none' },
    DEBUG:      { value: 0, stream: 'debug' },
    INFO:       { value: 1, stream: 'info' },
    WARN:       { value: 2, stream: 'warn' },
    ERROR:      { value: 3, stream: 'error' }
}

I considered accessing these on the streams object directly, but did it this way for readability and flexibility. A major deciding factor in choosing this route was the ability to reorder logging levels in a single place in the code.

// ## log(level msg);
// logs the output with a nice colorful indicator of the stream.  
// `msg` may be a string or a formattable string and options.
Log.prototype.log = function(level, msg) {
    var original = level;
    try {
        if(typeof level === 'string'){
            level = this.logLevels[level.toUpperCase()];
        }
        // get logLevel value
        if( (this.logLevel > -1) && (level.value >= this.logLevel) ) {
            var stream = this.streams[level.stream];
            stream.write( util.format.call(this, '[\033[%sm%s\033[39m]\t', colors[level.value], level.stream) );
            stream.write( util.format.apply(this, slice.call(arguments,1) ) + '\n');
        }
    } catch (err) { 
        console.error("logging is improperly configured!\nIs %j a supported logLevel?\nError:\n%j\n",original, err);
    }
}

An example

I’ve uploaded an example to github.

In this example, I write out logs for each of the 4 supported levels. The default is to write these out to stdout. After writing to the defaults, I set

var Log = require('./log.js'),
    fs = require('fs');

var level = process.argv[2] || 1;
var logger = new Log(level),
    level = logger.logLevels;

logger.log(level.DEBUG, "This is a debug message");
logger.log(level.INFO, "This is an info message");
logger.log(level.WARN, "This is a warning message");
logger.log(level.ERROR, "This is an error message");

var debugStream = fs.createWriteStream('./debug.txt');
var infoStream = fs.createWriteStream('./info.txt');
var warnStream = fs.createWriteStream('./warn.txt');
var errorStream = fs.createWriteStream('./error.txt');

logger.streams.debug = debugStream;
logger.streams.info = infoStream;
logger.streams.warn = warnStream;
logger.streams.error = errorStream;

logger.log(level.DEBUG, "This is a debug message");
logger.log(level.INFO, "This is an info message");
logger.log(level.WARN, "This is a warning message");
logger.log(level.ERROR, "This is an error message");

debugStream.end();
infoStream.end();
warnStream.end();
errorStream.end();

This provides the following input when passing a log level in via the command line:

Flattr this!