http://blog.ghost.org/ghost-0-5/

If you might have used a Ghost blog site dev workflow such as I explained in my Ghost Workflow post, it's nice to know that nothing in the new version of Ghost breaks the approach I used.

Note that I have only done the upgrade process as decribed below for upgrading from Ghost 0.4 installed on Ubuntu as I describe in my Ghost Workflow post.

It's always a little stressful updating your production sites - especially when you have put in some of your own process around the install. I was very happy that this install went so smooth. For reference, here are the exact as steps I followed. This process is based on Ghost's helpful documentation.

Important Note: Be sure to backup your Ghost data (blog posts and settings) before starting any upgrade process using the Ghost export tool as explained on the Ghost Backup Instructions page. You need to also copy any custom theme directories you might want to backup, although these should be in your Git repo if you're using the same workflow approach as I am.

Local Mac dev install upgrade

First, download the latest version of Ghost from the Ghost download page.

Now extract the files:

cd ~/downloads
unzip ghost-0.5.0.zip -d ghost-0.5.0

Copy the following files to your Ghost install:

cd ghost-0.5.0
cp *.js *.json *.md LICENSE ~/yourblog

Remove the core directory from your blog, copy in the new core directory, and copy in the new casper theme:

rm -rf ~/yourblog/core
cp -R core ~/yourblog
cp -R content/themes/casper ~/yourblog/content/themes

Now you can do the new Ghost install:

cd ~/yourblog
npm install --production
npm start

Ubuntu production upgrade

Download the latest version of Ghost:

cd /home/git/tmp
wget http://ghost.org/zip/ghost-0.5.0.zip

If you set up Ghost to run as an Upstart service, stop your ghost service using stop yourblogservicename. Remember to be logged in as root here, or a user that has permissions to start/stop this service.

Now, as git user, remove the core directory from your blog intsall and copy in the new Ghost files:

su git
cd /home/git/yourblog
rm -rf core
cd /home/git/tmp
unzip -uo ghost-0.5.0.zip -d /home/git/yourblog

Now you're ready to do the install:

cd /home/git/yourblog
npm install --production

You can not test it using npm start, or just restart your blog service (as root) if you have one:

start yourblogservicename

Now check out all the new features Ghost 0.5 has to offer!

User state

What is "user state" anyway? In the case of a typical e-commerce site, this would include the user's shopping cart, and any partially completed forms in the order process. You might also have a customer login status and customer preferences. For Pampered Poultry, I have a cart and an order form.

Since older browsers don't support Local Storage, relying on it for a public web site is not acceptable. So I opted to store the user's cart in browser Cookies, since the cart data is so small and simple. I decided to store any partialy completed forms in Local Storage for users with modern browsers, but fallback to Cookies for users with older browsers.

Testing if Local Storage is supported

It's very easy to test if the user's browser supports Local Storage. For example:

// Determine id local storage is available
if (typeof(Storage) !== 'undefined') {
    localStorageEnabled = true;
} else {
    localStorageEnabled = false;
}

Using Local Storage and Cookies

Local Storage and Cookies both use a key/value approach, so sharing the same functionality across these two technologies can be easy. Cookies are more limited in size and performance- for a starting guide, you shouldn't exceed 50 cookies per domain, or more than 4093 bytes per domain. Local Storage will grant you 5M more of data space!

The key/value approach becomes very powerful when you realize you can use a key such as 'cart' and a value that is a JSON string of all the user's shopping cart data. That means we only need one simple line of code to write or read the whole cart object to/from Local Storage or Cookies!

Note that when you store data in Cookies, that data will be sent back and forth with every server request. This is another reason why Local Storage is such an improvement, as it lives on the browser and is not sent in the HTTP requests.

To read and write Local Storage:

The following code sets the Local Storage key 'USER_CART' to a JSON string representing the current cart object data:

localStorage.setItem('USER_CART', JSON.stringify(cart));

Now you can read this data out of Local Storage and process it back to JavaScript data as follows:

USER_CARTstr = localStorage.getItem('USER_CART');

if (USER_CARTstr) {
    cart = JSON.parse(USER_CARTstr);
} 

To read and write Cookies:

I use the following two JavaScript functions to emulate the Local Storage functions above:

function setCookie(name, value) {
    var cookieStr = name + '=' + value + ';path=/;domain=' + thisSiteDomain;
    document.cookie = cookieStr;
}

function getCookie(name) {
    var nameEQ = name + "=";
    var ca = document.cookie.split(';');
    for(var i=0;i < ca.length;i++) {
        var c = ca[i];
        while (c.charAt(0)==' ') c = c.substring(1,c.length);
        if (c.indexOf(nameEQ) == 0) return c.substring(nameEQ.length,c.length);
    }
    return null;
}

Note: The setCookie function expects a global thisSiteDomain variable to contain your site's domain, since cookies are stored at the domain level. For site-wide cookies, use '.mydomain.com'. Or you can kee your cookies per-subdomain such as 'orders.mydomain.com'. For localhost, such as when testing on your local laptop, use an empty string ('') for the domain.

JSON concerns

A couple notes about using JSON functions in the browser:

Browser support- Older versions of IE do not support the JSON JavaScript functions. To work around this issue, just download and reference the JSON 3 library. Use the following code in the <head> section of your HTML:

<!--[if lte IE 8]>
<link href='http://seanvbaker.com/css/ie.css'  rel="stylesheet" />
<script type="text/javascript" src='http://seanvbaker.com/js/json3.min.js' ></script>
<![endif]-->

JSON.parse- Unless you are 100% certain the string in question is valid JSON, be sure to run the parse function in a try/catch:

try {
    data = JSON.parse(dataStr);
} catch(err) {
    // Handle error here
}

If JSON.parse tries to process invalid JSON, it will crash your application unless you trap the error. I tend not to worry about it for front end variables that have only been created using a JSON.stringify() command. But I do use it on anything sent to/from the server via Ajax, etc.

Next post I will talk about Content-forward design.

For a full introduction and index to this blog: Node.js: One New Approach

Cheers!

The HTML5 Web App

An HTML5 web app employs some or all of the following emerging technology standards:

• HTML5 tags- new browsers support more advanced layout and user interface controls
• Style Sheets- CSS3 allows powerful control and easier programatic access to layout, element positioning, and design styles
• Ajax- Asynchronous JavaScript enables real-time dynamic access to server resources from a static web page
• Local Storage- A simple key value pair data store in the browser makes it possible to maintain user state on the client instead of the server

Before HTML5

The PHP, ASP, .NET "dynamic web pages" architecture might seem limiting compared to today's highly interactive and responsive web apps. However, there are some major benefits for the programmer:

• Design follows a natural story board. Somewhat static pages take in user input to be processed when the user is ready.
• Story board design approach naturally organizes code. Each "page" (file) becomes analogous to a function.
• User state is managed on the server, which is fundamentally more secure and controllable than the client browser

An HTML5 architecture

I took an HTML5 approach for the Pampered Poultry e-commerce site. The web site runs as a client app, maintaining user state via cookies and/or local storage depending on browser support. All dynamic content (products and news) is loaded via Ajax calls to the Node server. The Node server serves the static files (such as HTML, JavaScript, CSS, and images), implements the Ajax handlers, and interfaces with MySQL for data storage.

Now, let's look at managing state in the browser using cookies and local storage.

For a full introduction and index to this blog: Node.js: One New Approach

Cheers!

I feel that Node.js beginners are told far too often that the only way to accomplish asynchronous JavaScript is to use an async library. In my experience, solving relatively simple async problems is the best way to learn Node, and will give you a deeper more intuitive understanding of the environment. (It will also give you the tools you will need to implement more complicated async solutions.)

Callback hell

"Callback hell" refers to the excessive code nesting created as you attempt to code a series of logical steps that each require a call to an asynchronous (non-blocking) function. For example, the following function processes three asynchronous steps one at a time:

function nestedExample() {
    setTimeout(function() {
        console.log('Do step one');
        setTimeout(function() {
            console.log('Do step two');
                setTimeout(function() {
                    console.log('Do step three');
                    setTimeout(function() {
                        console.log('Finalize process');
                        return;
                    }, 200);
                }, 200);
        }, 200)
    }, 200);
}

Note: Using setTimeout() it is a great way to simulate an asynchronous function when you are learning. Much like a database or http call, it takes a callback function to execute when it completes. Here the delay is set to 200ms.

Notice the way to code marches to the right, with each successive callback nested inside the previous step. This code can be very hard to read and manage.

A recursive approach

Before jumping to a full fledge asynchronous solution such as async, or q, consider whether a recursive pattern might work. The above example would look something like this:

function recursiveExample() {

    var step = 'one';
    processData();

    function processData() {

        switch (step) {
            case 'one':
                setTimeout(stepOne(), 200);
                break;

            case 'two':
                setTimeout(stepTwo(), 200);
                break;

            case 'three':
                setTimeout(stepThree(), 200);
                break;

            case 'finalize':
                stepFinalize();
                break;
        }
    }

    function stepOne() {
        console.log('Do step one');
        step = 'two';
        processData();
    }

    function stepTwo() {
        console.log('Do step two');
        step = 'three';
        processData();
    }

    function stepThree() {
        console.log('Do step three');
        step = 'finalize';
        processData();
    }

    function stepFinalize() {
        console.log('Finalize process');
        return;
    }

}

This approach creates more code, but it is very simple, easy to read, and permits a large degree of flexibility.

Note that the step variable is global to all the step functions because of the closure.

I have used steps named "one", "two", and "three" above, but in practice you can name these to fit your process - such as "validateuser", "processorder", and "send_email" for example.

You can easily expand this approach to handle some parallel tasks, and branching conditionals in the async process steps. (You can also start with an array of records to process and .pop() each next value to process until the array is empty.)

This is certainly not the best approach for many cases - but it is a great way to become more intimate with asynchronous programing before jumping into an async library. And you might be surprised how manageable your async challenge becomes once you start to become comfortable with some new patterns.

I was able to implement all of the MySQL transaction functionality for my eCommerce sites in a clean readable manner without using an async library.

I think the jump to a formal async library comes when the complexity level increases to the next level, or when you are implementing a larger scope project that requires integrating asynchronous functionality across multiple modules.

Read more on how to implement promises here.

Next post I will talk about Moving user state to the browser.

For a full introduction and index to this blog: Node.js: One New Approach

Cheers!

Procedural programming

If you have done any front-end browser programming with the browser DOM and AJAX, you are already familiar with an event-driven environment and some asynchronous programming. Since the front end tends to respond to user events, the event-driven approach is easier to understand in this context.

But on the back end, the instinct to build services in a procedural manner is hard to change. The procedural approach closely mimics our logical approach to solving problems. We see the solution as in a flow diagram, with conditional branches, process loops, and an eventual output. Everything happens one step after another. This is great for the programmer, but creates a challenge for the server. Each request to the server essentially fires up a dedicated program to process the request. The process may do many things - query a database, request data from another server, write a file, etc. The server must keep this ball in the air while still serving other requests. To keep things moving, the server manages many threads at the same time - even when they are just sitting there waiting for a response from the database, file system, or external server.

Asynchronous programming

Node.js does not directly support the procedural approach. Instead, it leverages the asynchronous capability of JavaScript and puts the burden (and the control) in the hands of the programmer.

In Node, you still implement a procedural process, but you must design an asynchronous coding solution for it. As a byproduct of this design process, you will craft a fundamentally more efficient program.

In Node, whenever you reach a step that might take some time (like querying a database, writing a file, or making an http request to another server), you call an asynchronous function. The asynchronous function is written in such a way that it returns control back to your process flow right away, but continues to execute on the server. This asynchronous function most always takes a "callback" as an argument, which is a function you pass to it to be executed when it completes. Because of the "closure", the callback function has access to the caller's variables and keeps their state on the server until the asynchronous function completes.

If you don't already understand closure in JavaScript - here is a nice quick explanation.

On a busy Node server, there may be many simultaneous requests active on the server at any one time, which seems to go against the notion you get when you hear that Node is single-threaded. The answer to this apparent contradiction comes from the way JavaScript manages function scope using closure. Even though there are multiple independent client requests active at one time, each function called to handle a request maintains its own set of variables until it completes.

Here is a helpful overview of JavaScript functions and scope: Functions and function scope

Great for I/O intensive work

I like to think of Node as a "conductor", orchestrating requests and the various I/O tasks required to fulfill the request. An I/O task might be querying a database, reading a file, or making an http request to an API. Also, Node.js supports delegating a CPU-intensive task to a "child process" which gets its own thread.

Callback hell

This is a disparaging term that refers to the syntactical challenge that can crop up when you try to implement a long branching procedural process flow that requires many nested asynchronous function calls. There are a couple ways to mitigate this scenario:

• There are special libraries and approaches to support complicated logical flows in an asynchronous environment. Popular solutions include promises, async, and other approaches
• Define functions outside your logic flow code block instead of putting them in-line. With some clever design and function naming, this can yield surprisingly impressive results
• Using recursion can solve many asynchronous list-looping patterns with a small amount of nice readable code
• Just accept some level of function nesting - it is not as bad as it seems once you accept it as part of the language. I find I can nest up to three async calls without confusion, and that I can code a majority of my solutions without needing to go further.

Next post I give an example of using recursion to tame callback hell.

For a full introduction and index to this blog: Node.js: One New Approach

Cheers!

Moving from IIS and Apache

As someone who has implemented many server-based web solutions, from e-commerce to integrated corporate financial systems, I hope to extol the virtues of Node.js from a familiar vantage point.

What Node.js is

Node.js is an application that implements and manages a JavaScript run-time environment using Google's V8 engine, along with a library of built-in core functionality that implements much of the standard network and OS plumbing required to build web-centric applications and servers.

Node also includes the npm package manager that facilitates the use of community created functionality (in the form of modules).

What Node.js does

Node.js can do many things, but this blog focuses on using Node where we might have used IIS and Apache in the past. And as you will see, the flexibility of creating your own web servers will open the door to many intriguing new approaches.

The Node.js way

One of the most powerful aspects of Node.js is its minimalism. Even though there are thousands of modules available, the community generally fosters a "use only what you need" approach.

If I had to sum up any development wisdom I might have acquired over the years, I would say: "Less is more" - the fewer layers, functions, and lines of code, the more manageable, scalable, and malleable the final solution will be.

Of course you need layers, functions, and other organizing constructs to succeed. The trick is to not introduce any superfluous complexity, and always understand the purpose and use of each unit of abstraction (module, function, or framework.)

For those just learning Node.js, I strongly recommend starting with as few modules as possible. I often see folks start with a framework and several modules that claim to simplify things for you. While these modules often do have great value for their intended use, beginners will end up learning the framework rather than the fundamental approaches critical to succeeding with Node.js and asynchronous server programming.

JavaScript

One of Node's big selling points is JavaScript. Benefits of using JavaScript on the server side include:

• Uses same language as the browser client
• Supports asynchronous and object paradigms
• Plays well with JSON
• Many developers already know and enjoy JavaScript

A few use cases for Node.js

Here are just a few things you might use Node.js for that would be more challenging and/or less scalable with a traditional IIS/Apache server approach:

• A scalable RESTful API server that does not need to manage client sessions or serve static file content (e.g., a back end service to an HTML5 app which manages its own state already)
• A transaction server that brokers transactions between numerous clients (e.g., a massive multiplayer HTML5 or native app game server)
• A centralized state management server that allows numerous clients to update and access real-time information (e.g., a low-latency in-memory data server)
• A server that fulfills requests by consuming and aggregating other web services (e.g., a mashup service)

Felix's Convincing the boss article as some great use case examples.

Next post I answer the question, Why async?

For a full introduction and index to this blog: Node.js: One New Approach

Cheers!

We used Microsoft's Visual Source Safe, and I quickly found it to be very powerful and deceptively simple to use. (I have spent far too many hours in my career trying to explain how VSS works to developers new to it.)

For personal projects and small teams I continued to use VSS for version control, but more importantly, as a project organization and production deployment tool. Even in situations that required less formal testing, versions were carefully "labeled" and deployed to production. This helped the team share development code, kept to source code reliable, and most importantly, helped me keep my sanity ;)

Funny how times change, but many of the same magic steps seem to persist, only with different approaches and lingo. Where we had the ceremonious "pulling of the label" to production with VSS, today I "push the master branch" to production using Git.

Using Git for deployment

The article explains one way to use Git to deploy Node.js sites to an Ubuntu Linux server. I do not get into the version control and development management aspects of Git. I also assume you are developing on a Mac, but most of the development side should work with Git on Windows as well.

This approach keeps a remote Git repository on the production server that you will push your updates to. A Git "hook" on the server will automatically execute whenever you push a code update to the server. The hook will stop the Node app service, deploy the updated files from the Git repository to the the actual Node app directory, and then restart the Node app again.

Prerequisites:

• You need to have Git installed on your development machine: git-scm.com or Homebrew, etc.
• Configure your Ubuntu server as I describe in my Setting Up a Node.js Website post. This will:
  • Create a git user on the server who owns the app files
  • Create an Upstart service for your Node app

Setup Git for development

Go to the directory of your project on your development machine cd ~/mysite and run git init to create a local Git repository for the site.

You may have some files in your project that you do not want to track with Git or deploy to production. To have Git ignore these files, use a .gitignore file to filter out all files except those you want to manage and deploy to production. (Such files might include log files, environment settings files, passwords, keys, etc.)

Note: Git only tracks files, not directories. So if you want to include an empty directory as part of your project deployment, be sure to create a readme.md file or something so the directory has at least one file. (For example, I have a /log directory that needs to be created as part of the app.)

To create a .gitignore file for your project, go to your ~/mysite project root directory on your development machine, and create a new file called .gitignore and use something like this:

env.json
*.log
public/images/uploads/*.*
!public/images/uploads/readme.md
ssl/
.DS_Store

This will instruct Git to:

• Ignore any env.json files
• Ignore any files with a .log extension
• Ignore all files in public/images/uploads/
• But DO include public/images/uploads/readme.md
  • This will pull in the uploads directory
• Ignore any files in ssl/ (where I keep my keys)
• Ignore those pesky Mac .DS_Store files

You can test your .gitignore file by running this in your ~/mysite directory:

git add .
git commit -m "init commit"
git ls-tree --full-tree -r HEAD

This will add all the trackable files and the last command show you the files that were added and are now tracked by Git. You should only see the files you expect. If not, edit your .gitignore file until it works as required for your needs. To "un-track" a file in Git that was added by accident, use git rm --cached <file> for each you need to remove.

Install Git on your server

ssh into your VM, then install Git:

apt-get update
apt-get install git

Setup your Node site for Git

I assume your Node site is set up as I explain here. The git user owns to app files, which are located at /home/git/myite.

Log in to your server as the git user so we create the git repository with permissions that the git user will be able to use.

su git

First, add a bare git repository on the server for your site:

cd /home/git
mkdir mysite.git
cd mysite.git
git --bare init

Next, create the post-receive Git hook file that will deploy updates to the /home/git/mysite directory:

cd /home/git/mysite.git/hooks
cat > post-receive

Then paste in this file content modified for your node app path if needed:

#!/bin/sh
GIT_WORK_TREE=/home/git/mysite git checkout -f

(ctrl-d to exit the cat process)

Change permissions on the file to allow it to be executed:

chmod +x post-receive

Now go back to your local development mac and add the remote git repository you just created:

cd ~/mysite
git remote add mysite_label git@yourproductionserver.com:mysite.git

Git is going to use mysite_label as a name for the repository. You will be typing mysite_label a lot, so use something short that still describes your site ;)

Test the deployment process on your mac. (Add your site files to Git and make an initial commit if you did not do so when you first set up the local git repo for this project: git add . and git commit -m "init commit").

cd ~/mysite
git push mysite_label master

The git push command will prompt you for the password for the git user you created on the production server (unless you're using an ssh key to connect.)

This should push your committed code changes to the server, and the git hook on the server should automatically deploy the updated files to the node app directory. ssh to the production server and verify that your code updates have now indeed been deployed to /home/git/mysite.

Troubleshooting: Be sure you can ssh to your server as git: ssh git@yourproductionserver.com. Once you log in as git user, you should be in the /home/git directory and your mysite.git repository should be right in that directory.

There is a limitation to our deployment process at this point. If we make updates to the node app or its code dependancies, the node app needs to be restarted in order for the updates to take effect. We will address this next when we enable git user to start and stop the node service, and update the Git post-receive hook to start and stop the service.

Enable Node app restart on deployment

For this step, you need to have your node app set up as an Upstart service as explained here.

We can leverage the node service commands to enhance our Git deployment process to restart the Node app.

First we need to change the sudoers file to allow the git user to sudo as root to start and stop your blog service. (Only root-access users can stop and start Upstart services.) We also need to prevent the sudo process from prompting for a password when using sudo for this task, otherwise the automated git hook script will fail due to the prompting.

Add one line to the end of the sudoers file. I like to use vi, but you can use the editor of your choice. To use vi: export EDITOR="vi".

Here is an overview and command reference for using vi to edit files: Basic vi Commands.

Now to edit the sudoers file:

visudo

Visudo is a special vi editor that parses the file before allowing you to save changes, since mistakes in the sudoers file can make your server impossible to access.

It is important to add the following line to the end of the file:

git ALL = (root) NOPASSWD: /sbin/stop mysite, /sbin/start mysite

This added line will allow git user to sudo as root with no password prompt, but only when running the start and stop commands for your service. Note: mysite refers the service name you created for your Node app- the same name of your Upstart .conf file. Also note that you need to use the full path for the start/stop commands. To double check your path, use which start to see the full path.

Note: You need to logout and log into the server again for the sudoers file change to take effect in your ssh session.

To test, ssh as git@yourserver.com.

You should be able to stop/start the service as git user as follows:

sudo /sbin/stop mysite
sudo /sbin/start mysite

Now update the git post-receive hook to stop and start the service:

cd /home/git/mysite.git/hooks
cat > post-receive

Then paste this file contents in, modified for your site if need be:

#!/bin/sh
sudo /sbin/stop mysite
GIT_WORK_TREE=/home/git/mysite git checkout -f
sudo /sbin/start mysite

(ctrl-d to finish)

Now when you deploy updates via Git, your mysite Node service will restart as well. Make a small change to a file in your Node site on your Mac, commit the change, and deploy:

cd ~/mysite
git add .
git commit -m "test deployment process"
git push mysite_label master

You should see the new process id # returned to confirm the restart of the service - something like this:

git push mysite_label master

git@yoursite.com password: xxxxx

Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 499 bytes | 0 bytes/s, done.
Total 6 (delta 2), reused 0 (delta 0)
remote: mysite_label stop/waiting
remote: mysite_label start/running, process 3469
To git@yoursite.com:mysite.git
   60f6037..ece0ca8  master -> master

Seeing the process # gives us nice comfort that the service has indeed been restarted.

Conclusion

Now you can develop your Node.js app, stage and commit the changes via Git, and then deploy the updates to production with just one command :)

For a full introduction and index to this blog: Node.js: One New Approach

Next post I talk about Understanding Node.js.

Cheers!

There are a few common challenges to overcome first such as:

• Installing Node.js
• Using Node as a static file web server
• Running a Node app as a service so it starts on system boot and restarts after any potential crashes
• Hosting more than one Node app site on the same server

This article assumes you are using the Ubuntu Linux platform such as I discuss in Starting Linux for Windows developers.

I. Install Node.js

ssh into your server as a user that has root sudo privileges. So using our example from before:

ssh myadmin@162.243.238.83 -p 50231

Now run these commands one at a time to install the latest version of Node.js:

sudo apt-get update
sudo apt-get install python-software-properties python g++ make
sudo apt-get install software-properties-common
sudo add-apt-repository ppa:chris-lea/node.js
sudo apt-get update
sudo apt-get install nodejs

Notice that the first command is apt-get update. It is always a good idea to make sure your Ubuntu package manager has the latest package settings before starting any install.

To verify Node has been installed:

which node
node -v

This will tell you where Node is running from, and which version you are running.

II. Set up a Node.js static file server

In preparation for setting up a Git deployment process later, let's create a new Linux user called git who will own the web site files:

sudo adduser git

Follow the prompts and set the user's name to something like "Git user" - you can leave the rest of the settings blank. Make sure you make a note somewhere of the password you use for this new user!

Now change to this new git user using: su git. Use the password you just created above.

Create a new directory where the node server app and web site files will be located:

cd /home/git
mkdir mysite
cd mysite
mkdir public

Note that is is fairly standard to use public as the web root directory in node web sites - much as you're used to seeing wwwroot in IIS.

We're going to use connect to create our web server. Connect is a Node.js module that acts as a router for incoming requests to the server and offers a number of very popular "middleware" modules to handle many standard functions.

The most popular Node.js server framework module is probably Express, which is built on connect. I opt to use connect without Express because I don't leverage the templating and server state management functionality of Express. If you use Express, there are just some slight syntax changes to what you will see here when using stand-alone connect.

To install connect for this web site, make sure you are in the /home/git/mysite directory and use the Node.js NPM package manager to install connect as follows:

npm install connect

This should load the connect files and finish showing you the structure something like this:

connect@2.12.0 node_modules/connect
├── uid2@0.0.3
├── methods@0.1.0
├── cookie-signature@1.0.1
├── pause@0.0.1
├── fresh@0.2.0
├── qs@0.6.6
├── debug@0.7.4
├── bytes@0.2.1
├── buffer-crc32@0.2.1
├── raw-body@1.1.2
├── batch@0.5.0
├── cookie@0.1.0
├── negotiator@0.3.0
├── send@0.1.4 (range-parser@0.0.4, mime@1.2.11)
└── multiparty@2.2.0 (stream-counter@0.2.0, readable-stream@1.1.9)

Now you have a node_modules directory under your mysite directory. This is the standard directory other node modules will go in as well as you install them with NPM. You can also put your custom modules there.

Now create the actual node server app. Make sure you are still in /home/git/mysite and run the command cat > server.js and paste the following code in:

var http = require("http");
var connect = require('connect');

console.log('\n\n--- Node Version: ' + process.version + ' ---');

// Set up Connect routing
var app = connect()
    .use(connect.static(__dirname + '/public'))
    .use(function(req, res) {
        console.log('Could not find handler for: ' + req.url);
        res.end('Could not find handler for: ' + req.url);
    })
    .use(function(err, req, res, next) {
        console.log('Error trapped by Connect: ' + err.message + ' : ' + err.stack);
        res.end('Error trapped by Connect: ' + err.message);
    });

// Start node server listening on specified port -----
http.createServer(app).listen(80);

console.log('HTTP server listening on port 80');

Remember to use ctrl-d to finish the cat file creation process.

Now create the good old fashioned "Hello World" HTML file in the public folder: cd public and cat > index.html, then paste in the html:

<html>
    <head>
        <title>Mysite</title>
    </head>
    <body>
       Hello World!
    </body>
</html>

Now to test your site. At this point, you can only run the node app when you have root privileges, because it is using port 80, which is a privileged port. We will fix this later - but for now, change to root using: su -. (Or you can use your administrative account by putting sudo in front of the node command.)

Now cd /home/git/myite and use this command to start the node server app:

node server.js

If all went well, the server will be running and outputting any standard output (when you use console.log() in the app) to the terminal. Now use your web browser to see if it worked. Go to http://your.ip.address and you should see the "Hello World!" text. If you get any errors, you will see them in the server terminal window.

Notice that connect's static middleware defaults to use index.html as a default file of no resource was specified. You may also see Could not find handler for: /favicon.ico logged to the screen - that's ok for now - your browser was just looking for the favicon file which you do not have yet.

To stop the server, use ctrl-c in the terminal.

III. Run your Node app as a service

Ubuntu has a nice process for creating system services called Upstart. You create a simple script to define your service which will allow you to:

• Start the Node app whenever the server restarts
• Restart the app if it should fail for some reason (a crash from a bug, memory leak, etc.)
• Conveniently start/stop the app via system commands

To create the script you must be root user. Change to root and create the Upstart script as follows. The name of your .conf file will become the name of your new service:

su -
cd /etc/init
cat > mysite.conf

Now paste in the script:

description "Mysite Node Service"
author      "Your info ifya want"
start on started mountall
stop on shutdown

respawn
respawn limit 99 5

script
    sudo node /home/git/mysite/server.js >> /var/log/mysite.log  2>&1
end script

post-start script

end script

This service script tells the server to execute your node app, and log the stdout and stderr from the app to the log file you specified. Once started, it will keep running now. Note: if you have a major error in the app that causes a crash on startup, this script will start it over and over again 99 times (or keep trying to start it for 5 seconds, whichever comes first) before giving up.

(Note the use of sudo to execute the node server. This will help later when we grant the git user sudo permissions to start/stop the app.)

Now you can start your node website service using:

start mysite

It should start and tell you the process id, something like:

mysite start/running, process 7412

Now test the site with your browser - it should be working. Now at this point, if you reboot the server, the mysite service will start automatically! To stop or restart your service, use stop mysite and restart mysite. Nice!

IV. Using Nginx to host multiple node sites

At this point you can put any html, css, JavaScript, and image files in your public directory and host them using your node server. But what if you want to run several node web sites on the same machine? Assuming you have domain names for each site and can point their DNS records to your server's IP address, you can use the following solution.

One common approach is to use Nginx as a reverse proxy. In this scenario, Nginx receives all public requests for your websites and routes the requests to your Node apps based on the requested host and domain name, and then sends your Node app's responses back to the client.

Nginx is very fast, and easy to configure - making it a great solution for this purpose.

Install Nginx- First make sure your Node app is not running to avoid any conflicts with port 80: sudo stop mysite. Remember that you have to be root or in the sudo group to stop your service at this point.

Now install Nginx:

sudo apt-get update
sudo apt-get install nginx
sudo service nginx start

Test Nginx is running by pointing your browser to your server: http://your.ip.address. You should see the Nginx splash screen at this point.

Update your Node app port- For this approach, each of your Node apps will run on a different port, and Nginx will use that port to find them. So update your server.js file to change the port by editing the last few lines to look like this to move the app to port 8000:

// Start node server listening on specified port -----
http.createServer(app).listen(8000);

console.log('HTTP server listening on port 8000');

Configure Nginx- For each of the Node sites you want to host, create an Nginx .conf file. You can use any name for the .conf file, but matching your site name will help avoid confusion. Use root user for this:

su -
cd /etc/nginx/conf.d
cat > mysite.conf

Now paste in this file content, modified for your setup:

server {
    listen 80;

    server_name mysite.com www.mysite.com;

    location / {
        proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

The important settings are the server_name and proxy_pass lines.

This file tells Nginx to listen for any requests on port 80 for mysite.com or www.mysite.com and route them to http://localhost:8000 which is the port our Node app is listening on. (You will assign different unique port numbers for each subsequent Node app you want to host.)

Note: You can also combine all your sites into one .conf file - just put one server {} block after another. Also, you can use *.yourdomain.com to grab any host names and route them. I did have one issue where I needed to specifically add mydomain.com *.mydomain.com for it to route requests to mydomain.com with no host prefix.

To test the Nginx reverse proxy process:

• Restart Nginx to enable the configuration: service nginx restart (or use nginx -s reload)
• node /home/git/mysite/server.js to start the Node app
• Point your web browser to yoursite.com and verify you see your site now

If you map more than a few domain names, the proxy may fail and you will get something like the following error in the Nginx error log:

could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32

To increase the server_names_hash_bucket_size, edit the nginx.conf file using vi:

vi /etc/nginx/nginx.conf

The following line may be commented out with a # at the front of the line. Delete the # to re-activate the line:

server_names_hash_bucket_size 64;

Restart Nginx. If 64 does not fix the problem at first, increase it by a powers of 2, until it works (64, 128, 256, etc.)

Troubleshooting: To better see what is going on, you can monitor the Nginx error or access logs while you try to reach the server:

tail /var/log/nginx/access.log -f

tail /var/log/nginx/error.log -f

tail will show you the last few lines of the log, and the -f will let you monitor any new log entries as they come in. Use ctrl-c to exit.

In the next article we'll set up an efficient Git deployment process so you can easily develop on your laptop and push full site deployments to your hosted server with one command.

For a full introduction and index to this blog: Node.js: One New Approach

Next post I talk about Using Git for deployment.

Cheers!

As you start to work more with Node.js and many other emerging web technologies, you will begin to realize how much better and easier they work together when implemented in a Linux environment. Also, a majority of the helpful documentation and tutorials you find will assume you're doing things "the Linux way."

You don't need to be a Linux administrator to securely and reliably develop and host your own Linux solutions. Let's go through the process of provisioning a new Linux Virtual Machine (VM), accessing the server, setting up basic security and users, and get up to speed copying files to/from the server.

Development- For this example, I assume you develop on a Mac. You can work from Windows using a tool like PuTTY, but moving to Apple is nice because 1) the Unix-based OS helps your Linux skills, 2) most Linux tools and libraries have Mac versions, 3) the Mac terminal is nicer to use, and 4) Sublime Text 3 supports retina displays ;)

(Plus, it's the perfect excuse to get a shiny new MacBook Pro!)

Creating a Linux Virtual Server

Sign up for an account at DigitalOcean and create a new Linux "droplet." I'm using Ubuntu 13.10 x64 for this example. (If you plan to plan to network multiple VM's together via a private connection, use NYC2 for a location and enable Private Networking.)

Why DigitOcean? I agree with Jeedo's sentiments, as I also express here.

• Note: Once your droplet is created, DigitalOcean will email you your new server connection info.

Accessing and updating the server

Setting up your server to authenticate using SSH keys is considered more secure, but I find it adds confusion when you're trying to learn and creating/destroying a bunch of VM's. So just ssh into the server as a user. You will, however, want to lock down the server to improve security and improve manageability and reliability.

A good reason do these lock down steps when you first get started is because it slightly changes the process of accessing your server and configuring your deployment process.

Terminal and basic commands- The Mac terminal has a lot of pretty themes. Find terminal in your Applications/Utilities folder and drag it to your Dock for easy access. Some quick tips:

• Open new "tabs" (Shell -> New Tab) to run multiple sessions at the same time. Standardize on using certain themes for certain servers or functions... for example, I always use Homebrew theme for sessions on my local Mac, Novel for remote VM sessions, and Grass for MySQL sessions. This can go a long way to help keep you sane.
• The up arrow will bring back your last command. Press again and again to get prior commands. Nice.
• When referencing a file name in a command, type the first few letters and then tab to autofill the rest. If there are multiple matched, (it fills as much as it can) type another letter or so, and tab again to fill the rest.
• Your "home" directory is represented by the ~, so cd ~/ will take you to your home folder. So it is quite helpful to keeop your main project directories right off of your home folder.

The bread and butter:

• cd will change directory, as in cd ~/myproject. For Windows users it takes a little time to used to back slashes instead of forward slashes. Up one level is cd ../
• pwd will Present your current Working Directory - very handy.
• mkdir newdirectoryname will create a new directory at your current location
• ls will list the directory files. ls -a will show all including hidden files that start with a .. ls -l shows files in a list - along with permissions and file size.
• Directories can have one or more . in them. If a file or directory starts with a . it will be hidden. Use ls -a to see them.
• cat filename will write out the contents of filename.
• cat > filename will sit there and wait for you to type or paste text in what you want to write into filename. Use cat >> filename to append the text. The single > will overwrite the file with your new content. Use ctrl-d when you are done typing or pasting the text to finish. Using copy and paste, this becomes a very fast way to move small files contents around.
• rm filename deletes a file
• rmdir directoryname deletes an empty directory
• rm -rf directoryname deletes a directory and all files and subdirectories in it
• tail filename will write out the last few lines of that file - great got log files you want to peek at. Use -n 500 to see the last 500 lines. Use -f and it will sit there and write anything out that hits the file in real time. Example: tail /var/log/node.log -n 500 -f Use ctrl-c to exit.
• cp filename anotherfilename copies filename to a new anotherfilename
• mv filename newfilename will rename/move the file from filename to newfilename

ssh lets you connect to your virtual server and run a session there. Use the info DigitalOcean emailed you to connect, such as:

ssh root@your.ip.address.here

Then type in the password they sent you when prompted. If you have a domain name pointed to that IP address, you can just ssh root@yourdomain.com as well.

You may be asked if you want to continue to connect, just enter yes as follows, and then enter your password from the email:

MacBook-Pro:~ sean$ ssh root@162.243.238.83

The authenticity of host '162.243.238.83 (162.243.238.83)' can't be established.
RSA key fingerprint is c2:5d:e5:d0:f5:43:d8:52:73:22:fd:b6:7e:e3:ca:34.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '162.243.238.83' (RSA) to the list of known hosts.
root@162.243.238.83's password: 

root@nodeden:~#

The root@nodeden:~# promt above shows us we're logged into the nodeden machine as root. (nodeden is what I set up as the host name when I created the droplet at DigitalOcean.) The ~ means we're in root's home directory. A quick pwd shows is we are indeed in /root.

Package Managers- One of the first important Linux lessons I learned when moving from Windows is how critically important package management systems are. I was initially impressed with Node's NPM package manager, but when I started to use Ubuntu's package management and Mac's Homebrew, I realized that I could not work in this new world without leveraging package managers. (A full week battle trying to manually install and build GraphicsMagick on my Mac and Linux server was replaced with one command!)

Updating Ubuntu- Once you have ssh'd into your new server as root, run these commands one at a time to upgrade Ubuntu's packages and OS using the package manager:

apt-get update
apt-get upgrade
apt-get dist-upgrade

"update" will update Ubuntu's package definitions, "upgrade" will upgrade the packages, and "dist-upgrade" will do any potentially more major updates that could require a server restart.

Basic Linux user management

root is the "super user" account. You will need to be root to do certain things, but it's important to set up a new account that will generally administer the server. This will also allow you to disable root from being able to ssh into the server - a great security improvement.

Change the root user password- Even once you disable root and create a new administrative user, you will need the root password to get root access. Log in as root and run the passwd command to reset the root password to something you can remember and type.

Create a new user to administer the server- You can call this user anything you want:

adduser myadmin

Enter in a password for this user, and the user's name. You can enter blanks for the other questions. Now add the new user to the sudo group:

usermod -aG sudo myadmin

sudo- The sudo group has the ability to run most root level commands. Once your administrative user is in the sudo group, you can log in as this user and use sudo before commands run that command with root level access. You can also change your user to "become" root using su -, which means "substitute user root." You can become any user using su if you know the password. So when logged in as root, run su myadmin to become myadmin. Note: exit will return you back to the user you where when you ran su.

To test your new user, logout of the server using exit and then ssh as the new user: ssh myadmin@162.243.238.83. Important- make sure you can su - to get to root from this user, since we are about to disable the root ssh login. Also remember to use the new root user password you set up above!

Disable root ssh login

ssh into your server as root. To disable the ability for anyone to ssh as root, edit the sshd_config file using the vi editor:

vi /etc/ssh/sshd_config

Here is a nice overview on using vi to edit files: Basic vi Commands

Basically, use the down arrow to get to the line:

PermitRootLogin yes

Arrow to the right to put the cursor on the "y". Now press the i key to enter insert mode. Then you can type "no" and use the right/left arrow keys and the delete key to delete the "yes". The line should now look like this:

PermitRootLogin no

Now use the esc key to get out of insert mode. Type the : key to get to the command line at the bottom of the vi editor and type wq! and then return which means, "write the file, quit, and ! don't complain on exit." (Yes, vi takes a bit to get used to.)

Now before we can test the change, we need to restart the ssh service:

restart ssh

Now log off using exit and try to ssh to your server as root. It should fail. To log in, just ssh as the administrator user you created, then you can su - to become root whenever you need.

Note: There are some commands and tasks you can not accomplish using only sudo, sometimes you must su - to become root. I find it best to try everything using sudo first, then fall back to su - as root when required. This helps your learn what tasks you must use root for.

Change the ssh port

ssh runs on port 22 by default. There are many debates about whether it is more or less secure to move it to a different port to gain security through obscurity. I choose to change it to limit the endless attacks you will see in your auth.log. With ssh moved to a different port, it's much easier to review the auth.log to review potential attacks. It's not essential, but if you want to change your ssh port:

Change to root su - and edit the sshd_config file:

vi /etc/ssh/sshd_config

Just change the line right near the top where it sets the port number:

# What ports, IPs and protocols we listen for
Port 22

Use any open port over 1024 and below 65537. Save your update and restart ssh using /etc/init.d/ssh restart.

Now exit and try to ssh again. It should fail. Now here is how you ssh using a different port, such as 50231 in this case:

ssh myadmin@162.243.238.83 -p 50231

Key Ubuntu directories and logs

It helps to understand where things are and what patterns other developers follow. Different technology platforms have their own somewhat standard/expected approaches. Newer technologies can mix and match these standards. For basic use, I have found the following few items to be helpful:

• Note that each system user gets a home directory. When you ssh, you start out in the user's home directory - the ~ points to this home directory when you are logged in as a given user. So when I ssh as myadmin, I start off in /home/myadmin, or /~.
• The root user home directory is /root
• If you plan to use a Git deployment process, create a user called git who will have permissions to push app updates to the server and deploy them to production. Leverage the git user's home directory and keep the git repository directory there to make the git url simple.
• Key logs are in /var/log/
  • tail auth.log -n 500 -f Review auth.log - security access
  • Nginx will have a directory for logs here
  • You can log your node apps here
  • /var refers to "variable" because these files can grow a lot and need to be managed/archived.

Copy files to/from your server

Use scp to Secure CoPy files. scp allows you to specify your ssh logon credentials along with the server path. For example, let's copy a database backup file from the DigitalOcean VM to my local Mac development machine:

scp myadmin@162.243.238.83:/home/myadmin/backup.sql /Users/sean/myproject/backup.sql

If you have moved your ssh port, you need to jump through the extra hoop of applying the -P argument to use the correct port:

scp -P 50231 myadmin@162.243.238.83:/home/myadmin/backup.sql /Users/sean/myproject/backup.sql

You can also copy files from your laptop to the server as well:

scp -P 50231 /Users/sean/myproject/backup.sql myadmin@162.243.238.83:/home/myproject/backup.sql

For a full introduction and index to this blog: Node.js: One New Approach

Next post I will talk about Setting up a Node.js website.

Cheers!

But committing to a new technology has its risks. To really learn a new approach takes time. To gain the benefit that comes from experience, you know you will run into road blocks, and inevitably, you will make mistakes. But in the end, most of us know that in the world of technology, not looking forward carries the most risk of all.

We all wonder- where are things going? But more importantly, we should wonder- what is the direction things ought to go?

Why Node.js?

I first heard about Node.js as many probably have... from secondhand tall tales about some new amazingly fast server platform that uses JavaScript of all things. Tantalizing. And folks who use it seem somehow contented, in a pleasantly bemused sort of way.

I started with Node.js because I was looking for:

• A new non-IDE server-side solution
• A server with no state management overhead built in
• A like-minded developer community
• A new technology that was gaining acceptance

I adopted Node.js because I found:

• An amazingly powerful server programming environment
• Exemplary package management implementation
• A huge library of very helpful well-written packages
• A talented helpful community
• A sparse default configuration - add modules as needed
• Very low-latency (fast!) response times
• The magic of the Asynchronous paradigm

Linux vs. Windows

I spent the first 6 months using Node.js in a Windows environment. It works, but you are constantly swimming upstream. Don't fight it. Go Linux. The whole idea of Node.js is a Linux approach. Node is a great excuse to start out with Linux if you've been a good Windows soldier up to now.

When it comes to flavors of Linux, like many others, I quickly realized that Ubuntu is ideal for application hosting. It is also the mostly widely supported and documented OS for this purpose. (I find it less IT and network administrator focused and more developer/app hosting focused than others.)

Node.js Hosting

I skipped more app-centric services such as Heroku. The appliance app hosting approach (such as is also offered by Azure) may be a great approach for scalable mobile app back-end servers. But for more robust solutions, you will want to take advantage of the inherent power Node.js offers through the Linux OS and other software you install on the server. (I also find value in learning on a dedicated VM first, even if you plan to leverage Node specific appliance hosting in the future.)

Azure- I started with Microsoft Azure. At first, this seemed surprisingly nice. The portal seemed simple and elegant, and they seemed to be supporting Node.js well via iisnode. Azure also allows you to set up Linux "Virtual Machines" and install Node yourself. But after a year, I moved away from Azure because:

• Azure is becoming more complex and Network/IT focused
• It is more expensive for open source hosting
• Azure's subscription/payment process is poor
  • It's easy to lose production servers for silly reasons
  • Their customer service is unreliable

Also make sure you understand how Azure throttles your cheaper hosted VM's and cloud databases. I did find the performance of their dedicated VM's to be very consistant though, with reliable network connectivity and up time.

Joyent- I used Joyent for a while:

• Amazing technology - I plan to use for performance-critical solutions
• Great team - very supportive and intelligent - wow
• Note that their SmartOS is custom Linux:
  • Some subtle changes to what you're used to
  • Not easy to re-host elsewhere
  • Worth it if you need speed / advanced performance monitoring
  • Plays VERY well with Node.js
  • A tad more expensive, but great value
  • Billing process is not as mature as it should be

DigitalOcean- I currently use DigitalOcean and have been very impressed:

• DigitOcean is so simple and elegant - I love it
• Very cost effective for open source hosting
• All SSD (Solid-State Drive)
• Great auto VM backup process
• Ability to network your virtual servers

Database and DB Access

Reliable relational database interfacing is critical for my endeavors. I have also noticed that relational database use can tend to be under appreciated by newer developers. SQL is more than just storing and retrieving data- it can also be an incredibly powerful element of your process design and functionality.

Microsoft SQL- is cost effective on Azure. In this context, however, you lose some of the benefits of the enterprise product that I have become accustomed to.

MySQL- is cost effective everywhere of course. It was also a fairly easy process to come up to speed with MySQL from MSSQL, and the performance has been great. Enterprise tasks such as backing up and performance monitoring can be a challenge though.

node-sqlserver- node-sqlserver is a Node.js driver for Microsoft SQL server. I spent a fair amount of time trying to get this driver working. They were (and still are...) in the early stages of development, so building the driver is a challenge. I was able to succeed only in an Azure Windows Server VM. It seems like very few (if any) folks have been able to get it working in the Azure "Web Site" appliance environment. And there are no plans yet to create a Linux version. (The Windows Server vision did work quite well for me though.)

node-mysql- node-mysql created by Felix Geisendörfer is a great Node.js module. It is a "pure node.js JavaScript Client implementing the MySql protocol." That it is, and it works wonderfully too. The escape feature is very handy for handling SQL injection issues and automatic date conversion between JavaScript and MySQL (very helpful!). It also supports connection pooling and database transactions.

NoSQL- I have not implemented a major NoSQL project yet - so I can't speak to that. (I know many Node.js platforms tightly integrate NoSQL database support.)

Version Control and Deployment

Coming from the word of Microsoft Visual Source Safe, learning Git was rather intimidating at first. I ended up focusing on the deployment and code management benefits over the version control and team development use. This proved to be a great way to get started with Git - and I could not imagine life without it now.

Email

You will likely need to send emails out from your Node server. I spent longer than I care to recount setting up Postfix SMTP on my VM. In the end, it was not too bad to install and use once I figured it out - but I learned that the IP address you get assigned by your hosting provider is often already blacklisted for email sending - so your emails will end up in the recipient's spam folder. You can clean the IP address, but it takes time and effort.

I took a cue from Ghost and tried Mailgun, a cloud email service created for developers. Mailgun is an amazing service for developers. It not only worked great, but is also very cost effective (free up to 10,000 emails per month.) Their API looks great, they have a simple to use inbound email forwarding filter, great email DNS configuration/troubleshooting tools, and awesome web-based logging.

I used the emailjs Node module to send to Mailgun from my node server with great success.

Running multiple Node.js apps on a single server

At first I stuck with one Node app per server. But as soon as I found myself needing to host several smaller dedicated domain node web sites on one VM, I needed to find a solution. I was used to using IIS with a multi-homed Windows Server to server multiple web sites on one server.

Reverse Proxy approach- Nginx is often used in front of Node.js, not only to host multiple Node sites, but at times to also serve static files and/or implement SSL. I was surprised how easy it was to install and configure Nginx. I still use Node.js to implement SSL and static file serving for the most part, but Nginx has certainly become a key layer in my Node stack now.

Server-side image processing

Getting GraphicsMagick installed and working on the server has allowed me to implement fancy UI functions I never considered before. The Node.js GraphicsMagick wrapper gm makes server-side image processing as easy as possible - although the actual exposed GraphicsMagick commands are rather tricky to get up to speed with, they are incredibly powerful.

Blog integration

If you want to integrate a nice Node.js-based blog into your site, look into Ghost. Ghost is a very new open source blog platform built with Node.js. (You're reading a Ghost blog now.) It uses Handlebars as a template engine and supports custom theme creation. It's quite simple to use their theme approach to embed the whole blog into your existing site design. You can read more about how I set it up here: A Ghost Workflow.

For a full introduction and index to this blog: Node.js: One New Approach

Next post I talk about Starting with Linux for Windows developers.

Cheers!

From out of the torrent of emerging web technologies, it was Node.js that finally inspired me to drop my otherwise adequate legacy platform in favor of "a new approach".

Before- When 14.4 modems still roamed the Earth, I created e-commerce sites on my Mac using Perl and uploaded them to Unix servers. User state was managed in the HTTP POST data, and persistent data was managed on the server by reading and writing static files.

And then- I met Microsoft SQL and Active Server Pages (known as ASP classic now). It blew my mind. We called it "Dynamic SQL" and "Dynamic HTML" back then. We used JavaScript in hidden frames to accomplish what Ajax does today. Google's original Gmail web app was considered a "Dynamic HTML" solution before "DOM manipulation" became the new phraseology.

Now- The next step. With HTML5, user state moves to the client. The server becomes RESTful, servicing autonomous requests. A more natural service-oriented architecture is emerging - not unwieldy XML Web Services, but lightweight JSON RESTful services.

This blog

There are many good introductions to Node.js, and even more awesome examples of how modules can be implemented. But there are too few examples showing complete implementations of Node.js being used as a new server platform to support the HTML5 application paradigm.

I created The Node Den to post my notes and share my experiences creating a Node.js e-commerce site that includes many of the fundamental features critical to most new website applications.

The Guinea Pig

Pampered Poultry is a real world e-commerce site I created a while back using Microsoft ASP Classic and MS SQL server. It ran on a Windows Server VM hosted on Azure. The site included a back-end operations/admin site to facilitate order processing and catalog management.

The goals of the new rebuilt site would be:

• Simplicity- Limit superfluous technology
• Low latency- A quick responding site
• Modern user experience- but good cross-browser support
• Manageability- Efficient deployment and malleable code base
• Low cost- This is a business site after all!

To accomplish these goals, I settled on the following architecture:

• HTML5 approach with user state on the client
• jQuery- it's like SQL for the DOM!
• MySQL database- transactional, fast, and free...
• Node.js RESTful services to handle data I/O
• Node.js serves static content for simplicity
• Ubuntu Linux- fast, cost effective, and plays well with others
• Git code management and deployment- a good workflow is key

Blog Index / Roadmap

Here is what I plan to cover. I will link each topic as they're published.

For reference, the complete site code base is available on Github:

https://github.com/svbaker/pyp2-site

I. Introduction and Orientation

• A Node.js web app platform recipe
  • Lessons learned
  • Recommendations
• Starting Linux for windows developers
  • What you need and nothing more
  • Program your OS!
• Setting up a Node.js website
  • Setting up Node and Nginx
  • Run your node app as a service
  • Use Nginx to host multiple Node sites
• Using Git for deployment
  • So much better than FTP hell

II. Starting with Node.js

• Understanding Node.js
  • Moving from IIS and Apache
  • Why async?
• Using recursion to tame callback hell

III. The HTML5 Approach

• Moving user state to the browser
  • Cookies, Local Storage, and JSON
  • Browser considerations
• App UI design in a content driven web
  • CSS, jQuery, and Ajax
  • Search Engine Optimization (SEO)
  • Favicon, Meta tags, Mobile support
  • Retina support

IV. The Node.js HTML5 App Server

• Setting up a Node.js server using Connect
  • Connect and its key middleware
  • Logging, redirecting, and error trapping
  • Serving static files
  • Managing environment configuration data
  • Creating Node.js modules
  • A Pattern for Ajax service handlers
• Sending email from Node.js
  • Using mailgun and emailjs
  • Hosting your own SMTP
• Setting up SSL in Node.js
  • Creating/managing SSL Certificates and keys
  • Implementing the SSL server
  • Redirecting SSL and non-SSL traffic

V. Node.js Ajax services with JSON and MySQL

• The power of JSON and JavaScript
  • Using JSON with jQuery and Node.js
  • Patterns for productive database interfacing
• Data processing in Node.js with node-mysql
  • Database connection management
  • Security considerations
  • Serving requests for data
  • Reliable database updates: transactions
  • Error trapping
• Encrypting sensitive data using Node.js

VI. Implementing an operations/admin site

• An HTML5 operations site architecture
  • Shell layout and DOM organization
  • Security management
  • Efficient data management patterns
• An Etsy-like image uploader using Node.js
  • Graphics-based UI
  • A Node.js multiple file upload process
  • Scale images on the server using GraphicsMagick
  • Save original and thumbnail images to disk
  • Save image upload info to the database

VII. Final Touches and Tweaks

• HTTP Compression and Caching in Node.js
• Using Nginx to host multiple sites
• Integrating a Ghost blog into a Node.js site
• Cross browser testing
• MySQL backup process

Host multiple Ghost blog sites on the same server. Develop and deploy your theme updates with ease. Most expeditious.

I wanted to start using Ghost for my blogs. For those who don't know, Ghost is a spiffy new Open Source platform solely created to manage and publish blogs. I was drawn to Ghost because it is a Node.js application, and because it uses Markdown for publishing.

This guide shows one way to set up a Ghost blog dev environment to

• host multiple blogs sites on a single server and
• use Git to deploy your theme updates.

I am using the following dev/hosting platforms:

• I develop on a Mac
• I host production on an Ubuntu 13.04 x64 VM at DigitalOcean

Overview of the Environment:

• Mac development has Node.js and Ghost app installations
• Production Linux VM uses Nginx to route requests to the respective Ghost blog Node apps
• Production node apps run as Upstart services to stay up
• Each blog app has its own Ghost installation
• Git tracks only custom theme files or code edits
• Git deploy hooks in production allow one command deployments for each blog site

Install Ghost on Development Mac

Be sure you have Node.js installed first.

Sign up for Ghost: https://ghost.org/signup

Follow their instructions to download and extract the zip file.

If you plan to manage multiple blogs, it will help if you stay consistent with where you put your blog projects. I keep all mine right off the root of my user folder. So for this example, move all the extracted ghost files to ~/yourblog.

Now install (build) the ghost app:

cd ~/yourblog
npm install --production

Then rename the config.example.js to config.js to avoid any confusion. (You can keep the config.example.js file for reference if you rather - Ghost will create the config.js file from it the first time you start Ghost.)

The config.js file is used to manage your specific environment settings. Ghost keeps separate settings for development and production environments here. When you start the Ghost app, you pass an argument to control which environment settings you want Ghost to use.

Edit the development section of the config.js file to assign an open port you want to use. Ghost defaults to 2368. I find it easiest to use the same port for development and production. So here I change the port to 8000 for this example:

development: {
    // The url to use when providing links to the site, E.g. in RSS and email.
    url: 'http://localhost:8000',

    database: {
        client: 'sqlite3',
        connection: {
            filename: path.join(__dirname, '/content/data/ghost-dev.db')
        },
        debug: false
    },
    server: {
        // Host to be passed to node's `net.Server#listen()`
        host: '127.0.0.1',
        // Port to be passed to node's `net.Server#listen()`, for iisnode set this to `process.env.PORT`
        port: '8000'
    }

So while your at it, let's change the production section as well. Also, set the url to match the domain of your blog:

production: {
    url: 'http://yourblogdomain.com',
    mail: {},
    database: {
        client: 'sqlite3',
        connection: {
            filename: path.join(__dirname, '/content/data/ghost.db')
        },
        debug: false
    },
    server: {
        // Host to be passed to node's `net.Server#listen()`
        host: 'localhost',
        // Port to be passed to node's `net.Server#listen()`, for iisnode set this to `process.env.PORT`
        port: '8000'
    }
},

Now test the install (npm start uses --development by default):

cd ~/yourblog
npm start

Open a web browser to http://localhost:8000 to verify Ghost is running.

To learn about setting up and switching Ghost themes, visit the Ghost documentation.

Troubleshooting: Watch the terminal's live logging to see what might be causing any potential issues.

Initiate Your Custom Theme(s)

If you're developing your own theme, copy the default Casper theme into your own theme folder:

cd ~/yourblog/content/themes
cp -r casper yourtheme

Setup Git Version Control

Make sure you have Git installed: git-scm.com or Homebrew, etc.

Now go to your project cd ~/yourblog and run git init to create a local repository for the blog site.

You will have different Ghost files installed in production (due to the binaries and such created during installation), so you want to be sure to only track code changes to your theme files, or any additional Ghost core files as you may need to edit. To accomplish this, use a .gitignore file to filter out all files except those you want to manage and deploy to production.

This can be tricky because .gitignore is optimized to ignore files, not include them.

In your ~/yourblog directory, add a new file named .gitignore and use something like this:

# Ignore everything first:
/*

# Then un-ignore the following:
!.gitignore

# --- Ghost core server index.js file ---
# --- Only needed if you edit this file ---
core/*
!/core/
core/server/*
!/core/server/
/core/server/*
!/core/server/index.js

# ---- Theme files -----
content/*
!/content/

content/themes/*
!/content/themes/

content/themes/yourtheme/*
!/content/themes/yourtheme/

!/content/themes/yourtheme/*

The trick is to first specifically ignore each directory that you then re-include files from.

You can test your .gitignore file by running this in your ~/yourblog directory:

git add .
git commit -m "init commit"
git ls-tree --full-tree -r HEAD

This will add all the trackable files and the last command show you the files that were added and are now tracked by Git. You should only see the files you expect. If not, edit your .gitignore file until it works as required for your needs. You don't want the core Ghost files (unless you asked for some specifically). You want to also make sure it is picking up the files you plan on developing and deploying. To "un-track" a file in Git that was added by accident, use git rm --cached <file>.

Create and Configure VM Server

I use DigitalOcean and create a Linux VM using the latest version of Ubuntu.

ssh into your VM, then update Ubuntu:

apt-get update
apt-get upgrade
apt-get dist-upgrade

Install Git:

apt-get install git

Create your git user who will deploy updates to production. I call the user git:

adduser git

Install Node.js:

apt-get install python-software-properties python g++ make
apt-get install software-properties-common
add-apt-repository ppa:chris-lea/node.js
apt-get update
apt-get install nodejs

Install Ghost:

You want to log in or change your user to git for the following steps. This will help ensure that the permissions settings allow git user to deploy files to production when we get to that step. Also create a directory to hold the production blog app.

Putting the git repository in the git user home directory (/home/git) makes for a nice easy path when we set up the remote git repo in development.

Note: Be sure to use the latest version of ghost where I have ghost-0.4.0.zip for this example.

su git

cd /home/git
mkdir yourblog
mkdir tmp
cd tmp
wget https://ghost.org/zip/ghost-0.4.0.zip
unzip -uo ghost-0.4.0.zip -d /home/git/yourblog
cd /home/git/yourblog
npm install --production
mv config.example.js config.js

Note: If unzip is not installed yet, log in as root su - and install it: apt-get install unzip

If you want to test your Ghost install at this point (before getting to Nginx), remember to edit the config.js file to add the correct host and port settings for your server. For a quick test, change the port to 80, run npm start --production from your /home/git/yourblog directory, then point a web browser to your server and verify that you see the default Ghost blog page.

Troubleshooting: As in development, hopefully the interactive logging will show you any issues.

Install Nginx

This will allow you to run multiple Ghost blog apps on the server. We will use Nginx as a reverse proxy to route user requests to the appropriate Node.js app.

Log in as root for this step:

su -

apt-get install nginx
service nginx start

Test Nginx is running now by visiting your server in a web browser. You should see the Nginx splash screen.

Configure Nginx

For each of the Ghost blog sites you want to host, create an Nginx .conf file. You can use any name for the .conf file, but matching your blog name will help avoid confusion:

cd /etc/nginx/conf.d
cat > yourblog.conf

Now paste in this file content, modified for your setup:

server {
    listen 80;

    server_name yourblogdomain.com www.yourblogdomain.com;

    location / {
        proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

The important settings to modify are the server_name and proxy_pass lines.

This file tells Nginx to listen for any requests on port 80 for yourblogdomain.com or www.yourblogdomain.com and route them to http://localhost:8000 which is the host and port we set up in our Ghost config.js file. (You will assign different unique port numbers for each subsequent Ghost app you want to host.)

Note: You can also combine all your blogs in one .conf file - just put one server {} block after another. Also, you can use *.yourdomain.com to grab any host names and route them. I did have one issue where I needed to specifically add mydomain.com *.mydomain.com for it to route requests to mydomain.com with no host prefix.

To test the Nginx reverse proxy process:

• Restart Nginx to enable the configuration: service nginx restart (or use nginx -s reload)
• cd ~/yourblog and npm start --production to start the Ghost Node app
• Point your web browser to yourblogdomain.com and verify you see your blog now

If you map more than a few domain names, the proxy may fail and you will get something like the following error in the Nginx error log:

could not build the server_names_hash, you should increase server_names_hash_bucket_size: 32

To increase the server_names_hash_bucket_size, edit the nginx.conf file using vi:

vi /etc/nginx/nginx.conf

The following line may be commented out with a # at the front of the line. Delete the # to re-activate the line:

server_names_hash_bucket_size 64;

Restart Nginx. If 64 does not fix the problem at first, increase it by a powers of 2, until it works (64, 128, 256, etc.)

Troubleshooting: To better see what is going on, you can monitor the Nginx error or access logs while you try to reach the server:

tail /var/log/nginx/access.log -f

tail /var/log/nginx/error.log -f

tail will show you the last few lines of the log, and the -f will let you monitor any new log entries as they come in. Use ctrl-c to exit.

Another key check: Your proxy_pass destination in the Nginx .conf file must match the production server: {host: 'XXX'} host setting in your Ghost config.js file. If you use localhost, it must be used in both files. Or if you use your server ip address, it must be used in both places as well!

Setup Git Deployment

This process will let you easily push out updates to your blog theme or any custom Ghost server edits you might implement. We'll use the post-receive Git hook to automatically deploy changes to the Ghost app and restart the Ghost service any time we push our an update.

Log in as git user so we create the git repository with permissions that the git user will be able to use.

su git

First, add a bare git repository on the server for your blog:

cd /home/git
mkdir yourblog.git
cd yourblog.git
git --bare init

Next, create the post-receive git hook file that will deploy updates to the /home/git/yourblog directory:

cd /home/git/yourblog.git/hooks
cat > post-receive

Then paste in this file content modified for your blog path:

#!/bin/sh
GIT_WORK_TREE=/home/git/yourblog git checkout -f

(ctrl-d to exit the cat process)

Change permissions on the file to allow it to be executed:

chmod +x post-receive

Now go back to your local development mac and add the remote git repository:

cd ~/yourblog
git remote add yourblogrepo_label git@yourproductionserver.com:yourblog.git

You will be typing yourblogrepo_label a lot, so use something short ;)

Test the deployment process on your mac. (Add the files to git and make an initial commit if you did not do so when you first set up the local git repo for this project: git add . and git commit -m "init commit").

cd ~/yourblog
git push yourblogrepo_label master

The git push command will prompt you for the password for the git user you created on the production server (unless you're using an ssh key to connect.)

This should push your tracked and committed code changes, and the git hook on the server should automatically deploy the updated files to the ghost app. Log into the production server and verify that your code change has now indeed been deployed to /home/git/yourblog.

Troubleshooting: Be sure you can log in to your server as git: ssh git@yourproductionserver.com. Once you log in as git user, you should start at /home/git and your yourblog.git repository should be right in that directory.

There is a limitation to our deployment process at this point. If we make any updates to the Ghost node server or its code dependancies, the node app needs to be restarted in order to use the updates. We will address this next when we set up our node app as an Upstart system service.

Running Ghost as a Service

In order to keep your Ghost blog apps running at all times, you can create an Ubuntu Upstart script and set up each node Ghost app as a system service which will do the following:

• Start the Ghost Node.js app whenever the server restarts
• Restart the app if it should fail for some reason (a crash from a bug, memory leak, etc.)
• Allow us to conveniently start/stop the app via system commands

To set up a service for the Ghost node app, you need to be root user:

su -
cd /etc/init

cat > yourblog.conf

Now copy in this file content modified for your blog specifics:

description "yourblog Ghost Node Service"
author      "Your info ifya want"
start on started mountall
stop on shutdown

respawn
respawn limit 99 5

script
    cd /home/git/yourblog
    npm start --production >> /var/log/yourblog.log  2>&1
end script

post-start script

end script

(ctrl-d to finish cat process)

This service script tells the server to cd to your ghost app root, start it, and log the stdout and stderr from the app to the log file you specified.

Test starting, stopping and restarting your service:

start yourblog
stop yourblog
restart yourblog

Once started, it will keep running now. Note: if you have a major error in the app that causes a crash on startup, this script will start it over and over again 99 times (or for 5 seconds, whichever comes first) before giving up.

Enable App Restart on Deployment

We can leverage the service commands to enhance our git deployment process to also restart the node app.

First we need to change the sudoers file to allow the git user to sudo as root to start and stop your blog service. We also need to stop the sudo process from prompting for a password when using sudo for this task, otherwise the automated git hook script will fail due to the prompting.

Add one line to the end of the sudoers file. I like to use vi, but you can use the editor of your choice. To use vi: export EDITOR="vi"

Now to edit the sudoers file:

visudo

It is important to add this line to the end of the file:

git ALL = (root) NOPASSWD: /sbin/stop yourblog, /sbin/start yourblog

This tells the system to allow git user to sudo as root with no password prompt, but only when running the start and stop commands. Note: yourblog refers the service name you created- the same name of your Upstart .conf file. Also note that you need to use the full path for the start/stop commands. To double check your path, use which start to see the full path.

Note: You need to logout and log into the server again for the sudoers file change to take effect in your ssh session.

To test, ssh as git@yourserver.com

You should be able to stop/start the service as git user as follows:

sudo /sbin/stop yourblog
sudo /sbin/start yourblog

Now update the git post-receive hook to stop and start the service:

cd /home/git/yourblog.git/hooks
cat > post-receive

Then paste this file contents in modified for your blog:

#!/bin/sh
sudo /sbin/stop yourblog
GIT_WORK_TREE=/home/git/yourblog git checkout -f
sudo /sbin/start yourblog

(ctrl-d to finish)

Now when you deploy updates via git, the ghost service will restart. Make a small change to a ghost site file on your mac, commit the change, and deploy:

cd ~/yourblog
git add .
git commit -m "test deployment process"
git push yourblogrepo_label master

You should see the new process id # to confirm the restart of the service - something like this:

git push yourblogrepo_label master

git@yourblogdomain.coms password: xxxxx

Counting objects: 11, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (6/6), 499 bytes | 0 bytes/s, done.
Total 6 (delta 2), reused 0 (delta 0)
remote: yourblogrepo_label stop/waiting
remote: yourblogrepo_label start/running, process 3469
To git@yourblogdomain.com:yourblog.git
   60f6037..ece0ca8  master -> master

Seeing the process # gives us nice comfort that the service has indeed been restarted.

Conclusion

Now you can develop your Ghost theme files in ~/yourblog/content/themes/yourtheme, stage and commit the changes via Git, and deploy the updates to production with one command :)

When you want to add an additional blog to your server:

• Create a new Ghost app in development
  • Create a new project directory
  • Install Ghost in the directory
  • Configure a unique port number in the config.js file
  • Create the local git repo for the project: git init
• Create a new Ghost app in production as above
  • The app directory goes under the /home/git directory
  • Create a bare git repo for production: git --bare init
• Configure Nginx
  • Create a .conf file with the new port number for this blog
  • Restart Nginx: service nginx start
• Create Upstart service and enable deployment process
  • Create the Upstart .conf script
  • Add a new line to the sudoers file so git user can start/stop the new service
  • Create the post-receive file and grant execute permissions to it
  • Add a remote git repository reference to your development project

Now you can get back to development knowing you have a lightning fast and reliable way to deploy your changes to production.

Cheers,

-Sv