Using the Callback Pattern and the Async Module

Here’s Part 2 in a four-part series on asynchronous Node.js development.

By Dan McGhan

September/October 2017

In the previous article in this series, “Asynchronous Processing in Node.js,” I covered some of the basics of asynchronous programming in Node.js. In this article, I’ll dive deeper into the topic and share some of the patterns used for this development, including the Node.js callback pattern and the Async module.

The examples in the first article used setTimeout to simulate asynchronous work. Because setTimeout is a simple timer API, there’s no chance of an error’s occurring. But this is not the case with real asynchronous work. Whether you’re writing to a file, interacting with a REST API, or executing a database query, errors can and will occur.

In this article, you’ll use Node.js to execute a SQL query in Oracle Database. This is a three-step process that must be done serially: first obtain a connection to the database, then use the connection to execute the query, and finally close the connection. Each step is an asynchronous operation that must include error handling logic.

To set up a test environment you can use to work through this article’s examples, check out “Creating a Sandbox for Learning Node.js and Oracle Database” on the Oracle and JavaScript blog.

Node.js Style Callbacks

The developers of Node.js had to take potential errors into account when designing the asynchronous APIs in the core modules. To promote consistency, they developed a couple of callback handling rules:

• The callback function is always the last parameter passed to an asynchronous API.
• The first parameter in the callback function is reserved for errors. If an error occurs, the error will be an instance of the Error class; otherwise, the value of the first parameter will be null.

The resulting pattern became known as the “Node.js style callback,” although some refer to it as the “error-first callback.” In either case, it’s a very simple pattern that requires developers to check the error parameter to see whether an error occurred and handle it accordingly.

Here’s an example that demonstrates the callback pattern using the file system (fs) module:

const fs = require('fs');

fs.readFile('/path/to/file.csv', 'utf8', function (err, data) {
 if (err) {
  console.log('There was an error', err);
  return; // Returning here is important.
 }

 // If you get to this point, assume 'data' is valid.
 console.log(data);
});

As you can see, the callback function, function (err, data){…}, is passed in as the last parameter to fs.readFile and its first parameter, err, is for a potential error. The first part of the callback function checks to see if an error occurred. If the path is wrong, the file doesn’t exist, or any other error occurs, the expression in the if block will evaluate to true, enabling the error to be handled.

In this example, the error is simply logged to the console and the function exits via the return statement. The return statement is important, because it prevents the success logic from executing. Newcomers to Node.js often forget to add the return statement, meaning that success logic executes after errors are handled.

Executing a Query with Node.js-Style Callbacks

Because the Node.js-style callback pattern is used so heavily in the core Node.js modules, many third-party module developers have adopted the pattern as well. This includes the Oracle team members who developed the Node.js driver for Oracle Database—they used this pattern as the default for all the asynchronous APIs.

The driver, known as node-oracledb—first released to the public in 2015—is open source and hosted on GitHub. It’s essentially a layer on top of the Oracle Call Interface libraries included with the Oracle Instant Client. This means that, despite the driver’s relatively young age, it’s quite feature-rich and performant.

The node-oracledb driver exposes several classes for doing various operations, from executing SQL and PL/SQL to streaming large result sets and large objects (LOBs). You’ll use two of those classes—Oracledb and Connection—to execute a simple query. Oracledb, the base class, will be used to obtain a connection to the database, which will return an instance of the Connection class. The connection will be used to execute the query.

Here’s an example of using node-oracledb to execute a query on the employees table in the HR schema. The dbConfig code should work for the App Dev VM (described in “Creating a Sandbox for Learning Node.js and Oracle Database”), but it will need to be modified for other environments.

const oracledb = require('oracledb');
const dbConfig = {
 user: 'hr',
 password: 'oracle',
 connectString: 'localhost:1521/orcl'
};

oracledb.getConnection(dbConfig, function(err, conn) {
 if (err) {
  console.log('Error getting connection', err);
  return;
 }

 console.log('Connected to database');

 conn.execute(
  'select *
  from employees',
  [], // no binds
  {
   outFormat: oracledb.OBJECT
  },
  function(err, result) {
   if (err) {
    console.log('Error executing query', err);

    conn.close(function(err) {
     if (err) {
      console.log('Error closing connection', err);
     } else {
      console.log('Connection closed');
     }
    });

    return;
   }

   console.log('Query executed');
   console.log(result.rows);

   conn.close(function(err) {
    if (err) {
     console.log('Error closing connection', err);
    } else {
     console.log('Connection closed');
    }
   });
  }
 );
});

As you can see, three different asynchronous APIs are being used in this code:

• oracledb.getConnection
• conn.execute
• conn.close

I’ll point out a few other things in the code. First, because each operation must be done serially, the call to the next operation is embedded in the callback function passed to the preceding operation. This nesting of anonymous callback functions—three levels deep in this case—is not quite “callback hell,” but you could end up there if any additional steps were added to the sequence. Also, the call to conn.close is repeated in two places, one to handle an error and the other for successful completion of the code.

Now copy the code to a new file and name it anon-functions.js. To run the script with Node.js, open a terminal, change directories to where the file was created, and run node anon-functions.js. You should see the console log output, which includes the employees from the employees table.

Using the named function technique from the previous magazine article, you can refactor the code to eliminate some of the nesting and duplication while making the code a little easier to read and understand (although this requires a few additional lines of code).

const oracledb = require('oracledb');
const dbConfig = {
 user: 'hr',
 password: 'oracle',
 connectString: 'localhost:1521/orcl'
};

function getConnection() {
 oracledb.getConnection(dbConfig, function(err, conn) {
  if (err) {
   console.log('Error getting connection', err);
   return;
  }

  console.log('Connected to database');

  executeQuery(conn);
 });
}

function executeQuery(conn) {
 conn.execute(
  'select *
  from employees',
  [], // no binds
  {
   outFormat: oracledb.OBJECT
  },
  function(err, result) {
   if (err) {
    console.log('Error executing query', err);
    closeConnection(conn);
    return;
   }

   console.log('Query executed');
   console.log(result.rows);

   closeConnection(conn);
  }
 );
}

function closeConnection(conn) {
 conn.close(function(err) {
  if (err) {
   console.log('Error closing connection', err);
  } else {
   console.log('Connection closed');
  }
 });
}

getConnection();

Copy this code to a new file, name it named-functions.js, and run the script as before. You should get the same output as with the previous version of the code (anon-functions.js).

In the named-functions.js code, I’ve declared named functions for each operation and then kicked off the sequence of functions at the bottom of the code by invoking the first of the functions to be executed: getConnection. With this technique, it’s relatively easy to organize sequential asynchronous operations.

More-complex operations, such as iterating an array asynchronously or doing multiple asynchronous tasks in parallel, would require something more. You could write your own library, but why reinvent the wheel when you can just use Async, one of the most popular Node.js libraries ever?

Leveraging the Async Module

You can think of Async as a module that extends the Node.js-style callback pattern. Async is not included with Node.js, so it must be installed with a command such as npm install async. With Async installed, you can require the library and take advantage of its 70-plus methods for various asynchronous processing situations.

In Async’s documentation, its methods fall into three main categories: collections, control flow, and utils. Let’s look at an example from the collections category.

const async = require('async');

const fakeAsyncApi = function(thing, callback) {
 setTimeout(function() {
  const error = Math.random() > .8 ? true : false;

  if (error) {
   callback(new Error('Failed to process ' + thing));
  } else {
   console.log(thing + ' processed');
   callback(null);
  }
 }, 2000);
};

const thingsToProcess = [
 'thing 1',
 'thing 2',
 'thing 3'
];

async.eachSeries(
 thingsToProcess,
 fakeAsyncApi,
 function(err) {
  if (err) {
   console.log('An error occurred!');
   console.log(err);
   return;
  }

  console.log('All done!');
 }
);

Here’s an overview of this Async collections code:

• Line 1: The Async library is required (after having been installed).
• Lines 3–14: A fake API (fakeAsyncApi) that implements the Node.js callback pattern is defined. The API will occasionally simulate an error’s occurring so that the resulting behavior can be observed.
• Lines 16–20: An array of “things” to process is defined. Typically, the elements would be obtained from reading a file or querying a database. Each element needs to be processed with the fake API.
• Lines 22–34: Async’s eachSeries method processes the array. The first parameter is the array to be processed, the second parameter is a function (fakeAsyncApi) that will process a single element, and the third parameter is a function that should be invoked when all work is done or immediately after an error occurs. Async will pass fakeAsyncApi one element from the array, along with a callback that, when invoked, will let Async know that it can move to the next element in the array.

Copy this Async collections example code to a new file, name it async-loop.js, and run it with Node.js (and don’t forget to install Async). You should see that each element is processed serially. If an error occurs, processing will stop and the final callback will be invoked immediately.

If you change the eachSeries method to each, you’ll see that all elements are processed in parallel. Pretty cool, huh?

Let’s see how one of Async’s control flow methods, waterfall, can be used to rewrite the database logic from before. I’ll adapt the named function version of the code (named-functions.js) from above to make it a little clearer how Async works.

const oracledb = require('oracledb');
const async = require('async');
const dbConfig = {
 user: 'hr',
 password: 'oracle',
 connectString: 'localhost:1521/orcl'
};

function getConnection(callback) {
 oracledb.getConnection(dbConfig, function(err, conn) {
  if (err) {
   console.log('Error getting connection', err);
  } else {
   console.log('Connected to database');
  }

  callback(err, conn);
 });
}

function executeQuery(conn, callback) {
 conn.execute(
  'select *
  from employees',
  [], // no binds
  {
   outFormat: oracledb.OBJECT
  },
  function(err, result) {
   if (err) {
    console.log('Error executing query', err);
   } else {
    console.log('Query executed');
    console.log(result.rows);
   }

   callback(err, conn);
  }
 );
}

function closeConnection(conn) {
 if (conn) { // If error getting conn, no need to close.
  conn.close(function(err) {
   if (err) {
    console.log('Error closing connection', err);
   } else {
    console.log('Connection closed');
   }
  });
 }
}

async.waterfall(
 [
  getConnection,
  executeQuery
 ],
 function(err, conn) {
  closeConnection(conn);
 }
);

Copy the code above to a new file, name it async.js, and run the script with Node.js. The output should match that of the previous versions of the script.

The first difference in the code you may notice is that Async is required at the top. Also, the signature of the three named functions was changed to work with Async’s conventions. Notice at the bottom that Async’s waterfall method accepts two parameters: an array of functions to execute and a final function to execute when all the functions in the first parameter are done.

The functions in the first parameter will be executed serially. When invoked, each function is passed a callback function (hence the change in signature) to invoke to execute the next function in the array. When the callback function is invoked, the first parameter passed in should be either null or an instance of the Error class. This is how to signal to Async that an error has occurred. In the case of waterfall, additional parameters can be passed to subsequent functions. Many of Async’s methods follow similar conventions, so once you get comfortable with one, it’s often easier to use others.

I hope you now have a better understanding of Node.js-style callbacks and how Async can be used to supercharge that callback pattern. In the next article of this series, I’ll look at a native way of doing asynchronous work in JavaScript, known as promises.

Next Steps

LEARN more about JavaScript and Oracle.

Photograph by iStock.com/Pixtum