Open in app

Sign up

Sign in

Write

Sign up

Sign in

Programming

A Primer on Node.js

An article that makes you sufficient enough to create & play with the node-based web server, do CRUD operations, and more.

Balakrishnakumar V
Towards AI
Published in
34 min readJun 12, 2021

--

Source — https://en.wikipedia.org/wiki/Node.js

Table of Contents:

1. Getting Started

2. Node Module System

3. Node Package Manager

4. Building RESTful APIs using Express

5. Express Advanced Topics

6. Asynchronous JavaScript

7. CRUD Operations with Mongoose

8. Mongo Data Validation

9. Authentication & Authorization

10. Handling and Logging Errors

References & Resources

Contains all the codes and dependencies related to this article can be found here: https://github.com/bala-codes/A-Primer-on-Node.js

Most of the sections contain the necessary code blocks to follow up, if you want to execute the codes as well while you go through, you can clone the repo.

The packages list are present inside the package.json file in the corresponding directories and install using npm install

I. Getting Started

1. What is Node

Node is a runtime environment for executing JavaScript code outside a browser. It is often used to build back-end services like APIs which powers Web apps, Mobile apps, etc., It is also superfast and highly scalable during the production environment. Along with perks like it can be built fast, requires fewer lines of codes compared to other backend services, lesser files, can handle more requests, and provide faster response time.

2. Node Architecture

Every javascript code requires a javascript engine to convert it into a machine code for machine understanding and code execution. Browsers provide a run-time environment.

Node uses a javascript engine built by Google named V8 engine + other modules to process the node.js program.

Again node is not a framework, it is only a runtime environment for executing javascript code.

3. How Node Works

Node is highly scalable, data-intensive, and can create real-time apps. It is because of the Non-blocking (Asynchronous) form of operation.

This means a single thread serves/handles multiple requests asynchronously. Request can be anything from fetching a piece of data from the database or performing any of the HTTP Verbs, since they aren’t waiting for any request to complete, it is efficient.

You might ask, what happens to the response from the requests which has been initiated earlier by the thread, those responses are fed into something called Event Queue, the node constantly monitors the Event Queue, when it finds any messages in Event Queue, it process it.
Node is ideal for I/O intensive apps, and not for CPU-intensive operations which require more time for performing calculations.

4. Installing Node

Download the latest node version from here.

Now run the installer and open up the CMD to check the version by running node --version.

5. First Node Program

Create a new file app.js. Obviously a Hello World

function greet(){
     console.log('Hello World');
}greet()

Run the program in CMD, node app.js

II. Node Module System

1. Global Object

There are some objects and functions which can be accessed in node anywhere. Some of them are

console.log() (or) global.console.log()setTimeout()
clearTimeout()setInterval()
clearInterval()

They are called global objects.

2. Modules

There is a clash, where two functions with the same function names are declared in different node.js programs, they create a global conflict.

Thus to resolve it and for cleaner code, we use modules, where we can define the same variable or function in these modules and call them accordingly.

Every file in a node application is considered a module. The variables or functions defined inside the file are scopes. By default they are not visible outside the file, we need to explicitly export them.

3. Creating a modules

Let’s create a simple module to send an HTTP request.

```
first_module.js 
```var url = 'https://google.com'function get(message){
     // Send a HTTP request
     { ... }
     console.log(message)
}// Make get function available outside the programmodule.exports = get;

4. Loading a modules

Now let’s load the module we exported in the above step. Make sure the above file and the current file are in the same folder.

```
app.js
```const get_method = require('./first_module')console.log(get_method('Some message'))

5. Module Wrapper Functions

Generally, the node does not execute our code directly, rather it wraps it inside a function like the one shown below, and later it is being exported.

(function (exports, require, module, __filename, __dirname){
   '''
      your codes goes here
   '''
})

This is how the node wraps the functions we write in any node js program and makes it available elsewhere.

6. Path Module

There are some modules that are very primitive and are built into the node already and we can make use of them directly. So let us see some of them.

You can find them all built-in modules here.

Now let’s work around some of the functions of the path module. You can find all functions that incorporate inside the path module, which can be found here.

'''
   app.js
'''// Look modules inside core of Node
const path = require('path')var pathObj = path.parse(__filename);
console.log('pathObj',pathObj)// The output will be the properties of the current js file.

Also, we look at some of the other modules in the node.

7. OS Module

OS module sheds some light on the properties related to the operating system that we are running this node app.

'''
   app.js
'''// Look modules inside core of Node
const os = require('os')var totalMemory = os.totalmem();
var freeMemory = os.freemem();console.log('Total Memory', totalMemory);
console.log(`Free Memory: ${freeMemory}`); // ECMAScript 6

More functions available inside the OS can be found here.

8. Events Module

An event is a signal that indicates that something has happened.

There are a lot of functions available, so we look into some of them. Here we simply create/register a listener and once it is called, we emit the message.

'''
   app.js
'''// Look modules inside core of Node
const EventEmitter = require('events'); // CLASS --> EventEmitter// Since EventEmitter is a Class, let's create a instance of the class to access the methods inside it. const emitter = new EventEmitter(); // OBJECT --> emitter// Register a listener
emitter.on('messageReceived', function() {
   console.log('Received');
});// Raise an event 
emitter.emit('messageReceived');

More functions available inside the Events can be found here.

9. Events Arguments

An event is a signal that indicates that something has happened. Now with some event arguments combined with arrow functions

'''
   app.js
'''// Look modules inside core of Node
const EventEmitter = require('events'); // CLASS --> EventEmitter// Since EventEmitter is a Class, let's create a instance of the class to access the methods inside it.const emitter = new EventEmitter(); // OBJECT --> emitter// Register a listener
emitter.on('messageReceived', (arg) => {
   console.log('Received', arg);
});// Raise an event with arguments
emitter.emit('messageReceived', {id:1 , token:'1234'});

10. File System Module

How to work with the files with node.js programs.

With the fs module, every operation defined under it comes with both forms of synchronous and asynchronous execution.

'''
   app.js
'''// Look modules inside core of Node
const fs = require('fs');const files = fs.readdirSync('./') // Synchronous
console.log('Files present under the directory', files)// Asynchronous Function - provides a callback
fs.readdir('./', function(err, files){
   if (err) console.log('Error', err)
   else console.log('Result', files)
});

More functions available inside the File System can be found here.

11. HTTP Module

An HTTP module is the backbone of web apps., where we can create a web server and host it with an endpoint. Here we create a mini webserver to run on localhost.

'''
   app.js
'''// Look modules inside core of Node
const http = require('http');const server = http.createServer((req, res) => {
   if (req.url == '/'){
      res.write('Hellow World')
      res.end();
   }
});// Event Listener
server.on('connection', (socket) => {
   console.log('New connection')
});server.listen(3000); // PORT Numberconsole.log('Listening on port 3000...');

More functions available inside the HTTP can be found here.

III. Node Package Manager

A registry for installing a third-party package. Also if you have any idea to publish your own module, then this is the place for you.

Check your version of npm npm -v

More on that here.

1. package.json

It is basically a JSON file that contains the basic information about the app, such as name, version, description, dependencies, and other metadata.

To start a new fresh app/project and create package.json automatically by entering the metadata, run the command npm init.

It is recommended to run the above command whenever we create a new project.

Later if you want to install the packages within your project directory or if any one shared the project with you and you want to install the dependencies, just run the command npm install

2. Install a Node Package

You can find all the packages here.

To install a node package, go to the corresponding project folder and run the command in the CMD, npm i <package name>.

Now, this new package is automatically added to the package.json as we install the package and the package is saved/available inside the node_modules directory.

3. Using a Package

Typically to import a library/module.

'''
   index.js
'''// Looks for the modules inside Node Modules Folder
var _ = require('underscore')var result = _.contains([1,2,3], 3);
console.log('Found match', result)

4. Package Dependencies

When we install a particular package, that particular package can be found at the node_modules directory, and most of the time, you might also see another package being installed.

So they are the additional packages that your actual package can depend on. These are called dependencies.

5. NPM Packages and Source Control

When we actually built a web app in real-time, then it might require a lot of packages to run seamlessly.

But upon inclusion of more packages, they are added into the node_modules folder and it weights more in MBs.

So upon transferring to another developer or in source control apps like Github etc., we cannot include this file.

But luckily our package.json file contains the dependencies, which can be built anytime anywhere.

6. Semantic Versioning

In semantic versioning, a node package has 3 components, like ^1.10.4.

These are Major.Minor.Patch

Patch — It’s usually used to fix any bugs.

Minor — It’s usually used to add some functionalities without breaking the existing APIs.

Major — It’s usually used to add some functionalities which might break the existing APIs.

Finally, the ^ character in the package is an indicator for a node to look for the same Major version and possible modifications in the Minor and Patch versions in the future, when the actual packages have some updates and it shouldn't affect our codes.

7. Listing all Installed Packages

Run the code to see all the package and their dependencies, node list.

Run the code to see only your packages and ignore their dependencies npm list --depth=0.

8. Viewing Registry Information for a Package

Run the code, npm view <package_name>, to find metadata for a particular package.

Run the code to only see the dependencies of a particular package, npm view <package_name> dependencies.

Run the code to see the versions of a particular package, npm view <package_name> versions.

9. Install a Specific Version of a Package

Run the code npm i <package_name>@<Major>.<Minor>.<Patch>

10. Updating Local Packages

To compare our actual local package with the current version of that package, run the code, npm outdated.

To update the package, run the command, npm updateit only updates the Minor and Patch versions for any package.

Precautions have to be taken before updating the Major version, as it can break our code. Run npm-check-updates, npm-check-updates -u, to upgrade the Major version and later install it npm i to reflect it in our package.

11. DevDependencies

During the development stage, we might include unit test, integration test, etc., and we might use some package to test it and those packages should not move whilst the app in production.

So to install some development packages, run the command npm i <pkg_name> --save-dev. As they are saved under “devDependencies” in the package.json file and node will take care of the rest.

12. Uninstalling the Package

To uninstall a package, run the command npm un <pkg_name>

13. Working with Global Packages

npm is one of the examples for global packages, where it is universal for all the node apps to run.

To upgrade such global packages, run the command npm i -g npm, here it updates npm.

Find all updates of global packages here, npm -g outdated

14. Publishing a Package

Try the following sequence, to create a package from scratch till publishing it, for our considerations let’s assume our package name is icecream

  1. mkdir icecream
  2. cd icecream
  3. npm init --yes
  4. Add a new file index.js, and it is our entry point for our package. Here you can add your functions and others etc.,
  5. Create an account in npm by, npm adduser , otherwise, log in using npm login. Provide username, password, email.
  6. To publish the package, run npm publish
  7. To use the newly created package in another project, run npm i icecream
  8. Congrats you have published your very own first package.

15. Updating a Published Package

Now if we want to add more functionalities and create a new version of our package.

Run npm version <major/minor/patch> & npm publish.

IV. Building RESTful APIs using Express

HTTP package is useful for creating a web server, it is indeed one of the approaches, but in a complex application, we cannot rely on HTTP. But luckily we have an express package which is a fast and light framework built on top of an HTTP package, which can handle complex workflow with a little less hardcoding.

1. RESTful Services

REST — Representational State Transfer establishes HTTP Verbs (GET, POST, PUT, DELETE) in a webserver.

2. Introducing Express

Replacing our old HTTP package with our new express package. Also if you notice, we had a lot of if blocks in our HTTP methods to expose our endpoints.

Run the commands to start our new express project.

  1. mkdir express-demo
  2. cd express-demo
  3. npm init --yes
  4. npm i express

Find more about express here.

3. Building Your First Web Server

Create a new file index.js and here we build our web server.

Now for easier understanding let’s keep a record of the movie lists and manipulate them using the HTTP verbs.

const express = require('express')
const app = express() // Contains the HTTP Verbs// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send([1,2,3,4,5])
})// Listen PORT
app.listen(3000, () => console.log('Listening on port 3000'));

Now run the code in cmd, node index.js and hit the browser with the endpoint localhost:3000 and localhost:3000/api/movies , you must see a similar message like the below one.

locahost:3000 & localhost:3000/api/movies

More on that about express API here.

4. Nodemon

So for every change we make, we are supposed to stop the server and restart again. I understand it is tedious, so we shall use one package called nodemon (Node monitor) which restarts the server by itself if any changes in done to the codes.

Run the command to install nodemon, npm i nodemon.

Now launch the webserver by running nodemon index.js .

5. Environment Variables

An environment variable is basically that is part of a variable in which the process runs and it runs outside our application. Like our PORT number here.

First set the PORT value on windows by using set PORT=5000 or MAC using export PORT=5000

const express = require('express')
const app = express() // Contains the HTTP Verbs// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send([1,2,3,4,5])
})
// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));

6. Route Parameters

Now let us add new routes to our movie database and fetch them.

Now in the routes, we can index them by adding: id values and query them by adding ?queryparameters.

const express = require('express')
const app = express() // Contains the HTTP Verbs// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send([1,2,3,4,5])
})app.get('/api/movies/:id', (req, res) => {
  // res.send(req.params)
  res.send(req.query)
})// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));
Route parameters by id (left) and by the query (right)

7. Handling HTTP GET Requests

Now let’s start to receive GET requests and provide the movie based on user-provided id. So we write a small function to fetch the movie based on the id.

const express = require('express')
const app = express() // Contains the HTTP Verbsconst movies = [{ id:1, name: 'Tenet'},{ id:2, name: 'Quantum of Solace'},{ id:3, name: 'Inception'}]// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send(movies)
})app.get('/api/movies/:id', (req, res) => {
const movie = movies.find(m => m.id === parseInt(req.params.id));
if (!movie) res.status(404).send('No movie found for the given id') res.send(movie)
});// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));
http://localhost:3000/api/movies/1 & http://localhost:3000/api/movies/15

8. Handling HTTP POST Requests

Now we can use the POST method to create a new course in the movies database based on the request from the user.

const express = require('express')
const app = express() // Contains the HTTP Verbsapp.use(express.json());const movies = [{ id:1, name: 'Tenet'},{ id:2, name: 'Quantum of Solace'},{ id:3, name: 'Inception'}]// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send(movies)
})app.get('/api/movies/:id', (req, res) => {
const movie = movies.find(m => m.id === parseInt(req.params.id));
if (!movie) res.status(404).send('No movie found for the given id') res.send(movie)
});app.post('/api/movies/', (req, res) => {
  const movie = {id: movies.length + 1,name: req.body.name};
  movies.push(movie);
  res.send(movie)
});// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));

9. Calling Endpoints using Postman or Thunder Client

To check whether the above POST request is successful or not, we need to check via services provided by postman or thunder, based on your choice.

Apparently, browsers can only send GET requests on these endpoints as they are designed to do so.

Now we see, we get a success 200 response.

10. Input Validation

We must always verify the content of our input and perform some primitive checks to make sure they are ok. So here we assure the movie name that they are inserting is not empty and say it must be at least 3 characters minimum.

We can use simple logic like if (!req.body.name || req.body.name.length < 3), but as our project gets complex, we can use an input validation package called Joi, more on that here.

Joi — API & Documentation here.

const express = require('express')
const app = express() // Contains the HTTP Verbs
const Joi = require('joi');
app.use(express.json());const movies = [{ id:1, name: 'Tenet'},{ id:2, name: 'Quantum of Solace'},{ id:3, name: 'Inception'}]// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send(movies)
})app.get('/api/movies/:id', (req, res) => {
const movie = movies.find(m => m.id === parseInt(req.params.id));
if (!movie) res.status(404).send('No movie found for the given id') res.send(movie)
});app.post('/api/movies/', (req, res) => {
  const schema = Joi.object({name: Joi.string().min(3).required()
});
  const result = schema.validate(req.body)
  console.log('result', result)
  
  if (result.error){
    res.status(400).send(result.error.details[0].message)
    return; // Don't execute the rest of the program
}  const movie = {id: movies.length + 1,name: req.body.name};
  movies.push(movie);
  res.send(movie)
});// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));
Success (200) & Failure (404)

11. Handling HTTP PUT Requests

Now let’s update a movie name based on the id provided by the user.

const express = require('express')
const app = express() // Contains the HTTP Verbs
const Joi = require('joi');
app.use(express.json());const movies = [{ id:1, name: 'Tenet'},{ id:2, name: 'Quantum of Solace'},{ id:3, name: 'Inception'}]// GET METHOD
app.get('/', (req, res) => {
  res.send('Hello World');
});app.get('/api/movies', (req, res) => {
  res.send(movies)
})app.get('/api/movies/:id', (req, res) => {
const movie = movies.find(m => m.id === parseInt(req.params.id));
if (!movie) res.status(404).send('No movie found for the given id') res.send(movie)
});app.post('/api/movies/', (req, res) => {
  const schema = Joi.object({name: Joi.string().min(3).required()
});
  const result = schema.validate(req.body)
  console.log('result', result)
  
  if (result.error){
    res.status(400).send(result.error.details[0].message)
    return; // Don't execute the rest of the program
}const movie = {id: movies.length + 1,name: req.body.name};
  movies.push(movie);
  res.send(movie)
});
app.put('/api/movies/:id', (req, res) => {
  // Look up the movie
  // If not existing, return 404  const movie = movies.find(m => m.id === parseInt(req.params.id));
  if (!movie) res.status(404).send('No movie found for the given id') // 404
  
  const { error } = validateMovies(req.body); // Equivalent to result.error
  if (error){
    res.status(400).send(error.details[0].message)
    return; // Don't execute the rest of the program
  }  // Update movie
  // Return the updated movie
  
  movie.name  = req.body.name
  res.send(movie)
 })function validateMovies(movie){
  // Validate
  // If invalid, return 400 - Bad request
  
  const schema = Joi.object({
    name: Joi.string().min(3).required() });  return schema.validate(movie);
};// PORT
const port = process.env.PORT || 3000;// Listen PORT
app.listen(port, () => console.log(`Listening on port ${port}`));
Success (200) & Failure (404)

12. Handling HTTP Delete Requests

Now let’s delete a movie name based on the id provided by the user.

app.delete('/api/movies/:id', (req, res) => {
  // Look up the course
  // Not Existing, return 404  const movie = movies.find(m => m.id === parseInt(req.params.id));
  if (!movie) res.status(404).send('No movie found for the given 
      id') // 404  // Delete
  const index = movies.indexOf(movie);
  movies.splice(index, 1)
  
  // Return the same movie
  res.send(movie)})
Success (200) & Failure (404)

V. Express Advanced Topics

1. Middleware

A middleware is something that takes a request object and returns a response to the client.

A request processing may sometimes go through a pipeline of processes, each containing a middleware function. In the below process, the JSON and route operations are middleware.

app.use(express.json())

Like Request → json() → route() → Response

2. Creating Custom Middleware

So for this purpose, let’s create an authentication middleware for every request we get and then we pass the control to the next middleware in our project. On a lighter note, without passing it to the other middleware, it would be stuck within itself.

// Middleware - 1
app.use(express.json());// Middleware - 2
app.use(function(req, res, next){
  console.log('Logging request...');
  next(); // Pass control to the next middleware; otherwise stuck
})// Middleware - 2
app.use(authentication);function authentication(req, res, next){
  console.log('Authentication Starts...');
  next();
}

Now looking at the console, we can see for every request that is hit, it goes through our custom middleware every time.

Middleware gets logged

3. Built-in Middleware

There is a lot of built-in middleware, that we can make use of. Like app.use(express.json()); , which automatically parse the req.body for any JSON objects and provides in the JSON format for us to use upon.

app.use(express.urlencoded({extended: true}));

app.use(express.static(‘public’));

app.use(express.urlencoded({extended: true})); — To receive form data in the request.

app.use(express.static('public')); — To service static files in the web server, so if we create a public folder and use some static files, it can be accessed from the web.

4. Third-Party Middleware

So let’s use the helmet middleware for demonstration.

  1. helmet — link, Help secure Express apps with various HTTP headers.
  2. morgan — link, HTTP request logger middleware for node.js.
const express = require('express');
const helmet = require('helmet')
const morgan = require('morgan')
const app = express() // Contains the HTTP Verbsapp.use(express.json());
app.use(helmet());
app.use(morgan('tiny'));

Now if we try to hit our node.js program endpoints, it will be logged like the below.

Morgan Logger

More on that here.

5. Environment Variables

Sometimes, we need to store environment variables like port, production/development environment, API Keys, etc., so we can export or set them using the command set NODE_ENV=production and make use of them in code by accessing it. A simple use case as follows,

console.log(`Node Environment : ${process.env.NODE_ENV}`)if (app.get('env') == 'development'){
  app.use(morgan('tiny'));
  console.log('Morgan Enabled')
};

So here we only log our incoming requests only if we are on the development server.

6. Configuration

7. Debugging

Sometimes during the development, we might want to see the logs of most of our functions or routes and later we usually delete them or comment them out and in the future, if we are engaged in any future updates, then we probably insert debug codes or uncomment them. This is a tedious and time-consuming process.

So we shall look at a package, to do this process more efficiently.

To install the package debug, npm i debug,

const debug = require('debug')('app:debug') 
// export DEBUG=app:debug in CMDconsole.log(`Node Environment : ${process.env.NODE_ENV}`)if (app.get('env') == 'development'){
  app.use(morgan('tiny'));
  debug('Morgan Enabled')
};

So if we set the variable DEBUG as an app: debug in cmd and later execute the code, then you will see the statement ‘Morgan Enabled’, otherwise it doesn’t print everything.

8. Templating Engines

Sometimes we need to send the HTML markup for the client instead of raw JSON response on the HTTP verbs. So we shall make use of a package called pug, to install it run npm i pug.

Later set the engine type in code, and also create a new folder “views” and create a file name index.pug which contains the HTML Elements

'''
index.pug
'''html
   head
      title=title
   body
      h1=messgae
'''
index.js
'''app.set('views', './views');
app.set('view engine', 'pug');app.get('/', (req, res) => {   res.render('index', {'title':'My Express Movie App', message:      'Welcome to Movie Database'})});
PUG generated HTML page

9. Structuring the Code

You can check the implementation we have done so far and all of them are done in the same index.js and also check the embedded gist to ensure that we are on the same page.

Also, I concur, this is not an efficient way, so we need to neatly organize the whole code.

The organized code is separated into multiple files and folders and added the GitHub repo link in the bottom section of this article.

VI. Asynchronous JavaScript

1. Asynchronous Vs Synchronous Code

As we already clarified that Node Js performs asynchronous or non-blocking execution, which means it will not wait for a function to complete, as it executes line by line.

So let’s take a simple example.

console.log('Before')setTimeout(() => {
console.log('Simulating a database operation')
}, 2000);console.log('After')OUTPUTS : 
Before
After
Simulating a database operation

Now from the outputs we see that it didn’t wait for 2 secs in executing the timeout function we inserted, it rather launched the function and moved to the next piece of code. Later in the event queue, the thread is informed that the output is ready and it collects and displays it.

So we see the function output last instead of second.

2. Patterns for Dealing with Asynchronous Code

So say we want to return a variable from an async function for further processing. So if we run the below piece of code,

console.log('Before')const movie = getMovies(1)console.log(movie)console.log('After')function getMovies(id){
  setTimeout(() => {
    console.log('Simulating a movie fetching operation')
    return {id:id, Name:'Tenet'};
  }, 2000);
}OUTPUT:
Before
undefined
After
Simulating a movie fetching operation

We see that the variable movie is undefined, that is because, at that instance of time, that variable is not available yet.

So there are 3 patterns to look out for whilst writing an asynchronous program.

  1. Callbacks
  2. Promises
  3. Async/await

Will look into each of them and make our asynchronous code more clear.

3. Callbacks

Previously the variable movie is undefined, now we shall use

callbacks options to populate the variable.

Callback is a function we call, when the result of async operation is ready.

As from the below code snippet, we create a function getMovies with two arguments id &callback, which takes its own time to provide the output. So once the output is ready, the callback will collect it.

And on function call, we use an arrow functions / anonymous function to collect the callback received and display it on the console.log. Here I have nested two functions with callbacks for experimentation.

4. Promises

A promise holds the eventual result of an asynchronous operation.

So the async function once completed, either resolve to return a response or an error.

So a promise has 3 states,

  1. Pending — Ongoing async operation
  2. Resolve — Fulfilled and response received
  3. Rejected — Error

So let’s see how to use promises with simple async operations.

const p = new Promise((resolve, reject) => {
  // Do some async work
  setTimeout(() => {
    resolve(1); // Return Output - 1
    // reject(new Error('Some error')) // Return Error
}, 2000);
});p
  .then(result => console.log(`Result : ${result}`))
  .catch(err => console.log(`Error : ${err.message}`))

So the promise takes two arguments to resolve and reject, the resolve carries the output, and the reject carries the error messages if any.

As we store the promise in the variable p, we can use inbuilt functions like .then to get the result or .catch to display the output.

5. Replacing Callbacks with Promise

As we did create a movie fetch and its director fetch function in callbacks example (section — 3), let us replace them with promises as we described in section — 4.

And also see how to consume those resolve and rejects that come out of the promises.

6. Handling Promises in Parallel

There are times, where we create two promises for two different functions or API calls and later perform some operations once these two promises are performed.

So to handle such scenarios, we have a static method in Promise that comes in handy.

  1. promise.all[(p1, p2)] — Resolves both the promises, then resolves the output or otherwise if it catches the error, rejects any resolve from any of those promises.
  2. promise.race[(p1, p2)] — Resolves the promises which complete first, then resolves the output or otherwise if it catches the error. Do not wait for the other ongoing promise and moves on.

7. Async & Await

Async & Await are new methods from Node.js and we shall rewrite our movie and director fetching async methods.

They are actually written on top of Promises but gives a feel like synchronous programming.

But there are some changes that need to be done on the function level, as we need to decorate the functions involved with the async keyword.

So we create a new async function that executes all our Promises, also we wrap them into a try & catch block to display an error if at all.

VII. CRUD Operations with Mongoose

CRUD is an acronym for CREATE, READ, UPDATE, DELETE and we will be performing CRUD on a NoSQL Database like Mongoose.

Point to note, as that is a wider topic to cover, I have decided to write it separately and publish it as an independent article.

You can find the article here.

But worry not it again built on our examples related to our movie database.

VIII. Mongo Data Validation

1. Validation

In our movie database, if we try to save a new document with all the required fields based on our defined schema, it will accept it. But if we try to save it without providing any of the fields, it will again save it, because we didn’t mention whether a particular entry is valid or not. This scenario is important sometimes, as we can’t afford to leave it to chance.

So let’s modify our schema to accommodate data validation.

const movieSchema = new mongoose.Schema({movieName : {type: String, required: true}, // Validation Checkdirector: String,imdbRating: Number,cast: [String],releaseDate: Date,genre: String,sequel: Boolean});

Later we also need to handle the rejections, just in case if the user didn’t pass the required parameter, otherwise, our code will result in an error and won’t go through. So we shall use a simple try-catch block to handle the promise rejections if any.

async function insertMovie(name_, director_, rating_, cast_, date_, genre_, sequel_){const movieObject = new movieClass({movieName : name_,director: director_,imdbRating: rating_,cast: cast_,releaseDate: date_,genre: genre_,sequel: sequel_});// Handling rejected promises here.
try{
   result = await movieObject.save()
   return result
}catch (ex){
   console.log('Error in Promises', ex.message)
}
}

2. Built-in Validators

We previously used a built-in validator for String, required: true.

There are a lot more that we can add for string and number for validation, without going for third-party libraries. Also, there is a conditional validator that passes only if the condition is met.

Then, let’s see the String & Number validators and most of them are self-explanatory.

// String ValidatorsmovieName : {   type: String,   required: true,   minlength: 2,   maxlength: 255,   // match: /pattern/ // Some Regex Pattern},genre: {   type: String,   required: true,   enum: ['horror', 'Action', 'science-fiction', 'supernatural-   horror', 'comedy', 'romance', 'adventure']
// Input must be one of them enumerated above.},// Number ValidatorsreleaseDate: Date,imdbRating: {   type: Number,   min: 0,   max: 10,   required: function() { return this.releaseDate(); } 
   // Conditionally make the property required}

So on the top, in the imdbRating, instead of passing the required field to be true or false, we have passed the condition, that only if the releaseDate is present, then we wanted the rating to be filled. This is called conditional validation, the property this. refers to the class object which is created during the function call.

3. Custom Validators

Sometimes we need to set up a condition that is not available in the built-in validators.

So take for the case, the cast field in our movie database must have at least 2 cast people specified while creating an entry in the database, but we cannot simply use required to true, as it will take even if there is a single cast entry.

So we shall modify the schema with custom validation, also specify a custom message if it fails.

cast: {   type: Array,   validate: {   validator: function(v){      return v && v.length > 2; 
      // Should not be a null and should have atleast two cast},message: 'The Cast must contain atleast two character names'
}
}

4. Async Validators

Sometimes, our validator logic needs to read something from the database or perform an HTTP call, etc., to validate the logic, so in those cases, we use async validators.

So we use callbacks logic, as described in the previous sections.

cast: {  type: Array,  validate: {  isAsync: true,  validator: function(v, callback){  // Do some async work or here I simulated timeout  setTimeout(()=>{      const result = v && v.length > 2; 
      callback(result)
  }, 5000);
},
message: 'The Cast must contain atleast two character names,
}
}

5. Validation Errors

So presently in our design, only if there is promise rejection, then we get an error message, so it’s hard to know if any such fields in our database have any error. So we shall display the error messages if any by iterating through the fields available.

async function insertMovie(name_, director_, rating_, cast_, date_, genre_, sequel_){const movieObject = new movieClass({   movieName : name_,
   director: director_,
   imdbRating: rating_,
   cast: cast_,
   releaseDate: date_,
   genre: genre_,
   sequel: sequel_});   try{
      result = await movieObject.save()
      return result
   }catch (ex){
      // Iterate throught the fields for error message      console.log('movieName Error', ex.errors.movieName)
      for (field in ex.errors){
         console.log(`${field} - Errors ${ex.errors[field]}`)
}}}

6. Schema Type Options

Already we have seen some of the schema types. So we shall add two more for String and Number.

// Type - Stringgenre: {   type: String,   required: true,   enum: ['horror', 'Action', 'science-fiction', 'supernatural- horror', 'comedy', 'romance', 'adventure'],   lowercase: true, // Converts the entry to lower case   uppercase: true, // Converts the entry to lower case   trim: true // Removes padding or extra spaces},// Type - Number
imdbRating: {   type: Number,   min: 0,   max: 10,   get: v => Math.round(v), // Rounds the value during fetching.   set: v => Math.round(v), // Rounds the value during saving.   required: function() { return this.releaseDate; } 
   // Conditionally make the property required}

XI. Authentication & Authorization

1. Introduction

Authentication is the process of identifying the user who they claim to be.

Authorization is determining if the user has the right permission to perform the given operation.

So we shall upgrade our movie database to enable this authorization and authentication.

So we shall create new endpoints in our application, with users who can register and login, using REST APIs.

2. Creating the User Model Schema

So let’s create a user database and a schema for enabling fields like name, email, password.

// MongoDb Schemaconst userSchema = new mongoose.Schema({username : {   type: String, required: true,   minlength: 2, maxlength: 50,},email: {   type: String,   required: true, unique: true,   minlength: 2, maxlength: 255,},password: {   type: String, required: true,   minlength: 2, maxlength: 1024,},});const userClass = mongoose.model('userClass', userSchema);
function validateUser(user){
   const schema = Joi.object({
   name: Joi.string().min(3).max(50).required(),
   email: Joi.string().min(3).max(255).required().email(),
   password: Joi.string().min(3).max(255).required()
});   return schema.validate(user);
}

3. Registering Users

So once we create the user schema of how it should be, next we can add routes containing, preferably a POST request to add user, along with the checks of name, email, password condition and already registered or not, etc.,

router.post('/', async (req, res) => {
  const { error } = validateUser(req.body)
  if (error) return res.status(400).send(error.details[0].message);  let user = await userClass.findOne({email: req.body.email})
  if (user) return res.status(400).send('User Already Registered.');  user = new userClass({
    name: req.body.name,
    email: req.body.email,
    password: req.body.password
});try{
  await user.save()
  res.send(user)
}catch (ex){
  console.log('Error in Promises', ex.message)
  res.status(400).send( ex.message)
}
})

4. Using Lodash

It’s cumbersome to actually read the req.body within the routes every time, by indexing each element manually. So we shall use a utility library called lodash to make our job easier and save time.

Lodash makes JavaScript easier by taking the hassle out of working with arrays, numbers, objects, strings, etc.

So I have replaced few codes from the above user registration using lodash utility library. Especially the .pick from lodash library to pick elements automatically from the req.body

router.post('/', async (req, res) => {
  const { error } = validateUser(req.body)
  if (error) return res.status(400).send(error.details[0].message);  let user = await userClass.findOne({email: req.body.email})
  if (user) return res.status(400).send('User Already Registered.');  user = new userClass(_.pick(req.body, ['name', 'email', 
                                         'password']));
  try{
    await user.save()
    res.send(_.pick(user, ['_id', 'name', 'email']))
  }catch (ex){
    console.log('Error in Promises', ex.message)
    res.status(400).send( ex.message)
}
})

More on that here.

5. Hashing Passwords

We cannot store the passwords as it is when a hacker hacks the database, then all the user's account and their passwords will be visible.

So we use a popular library called bcrypt to hash their passwords.

We need to understand the concept of SALT, which is used in the bcrypt library to encrypt and decrypt the password.

SALT — Random string added before or added password.

As mentioned, it adds a random string to the password to prevent the patterns from being guessed by the hackers.

Thus, before saving the password, we simply hash the password and save it in the database.

const salt = await bcrypt.genSalt(10);user.password = await bcrypt.hash(user.password, salt);

More on that here.

6. Authenticating Users

Now that we have successfully stored the users in our database, now we must receive the user login and need to send authenticate them by checking them across our database.

So I have added new POST routes for the action.

router.post('/', async (req, res) => {
  const { error } = validateUser(req.body)
  if (error) return res.status(400).send(error.details[0].message);  let user = await userClass.findOne({email: req.body.email})
  if (!user) return res.status(400).send('Invalid Email.');  const validPassword = await bcrypt.compare(req.body.password, 
                                             user.password)
  if (!validPassword) return res.status(400).send('Invalid 
                                                   Password.');
  
  try{
    res.send(true) // Sending True
  }catch (ex){
  console.log('Error in Promises', ex.message)
  res.status(400).send( ex.message)
}
})

As of now, if we successfully authenticate the user, we just send a true response back. But we need to send something more elegant so that user can continue to do some API operations etc.,

7. JSON Web Token (JWT)

A JSON Web Token JWT is a long string that identifies the user.

Head over to the link specified below to see how it looks like.

JWT Generation

On the left side, we have the JWT token, with tri-coloring red, purple, blue.

On the right side, we have its corresponding color properties, the red color indicates the header i.e the algorithm specified, purple indicates the payload section i.e which contains the user-related data and blue indicates the digital signature property i.e which is created based on the content of the payload body along with a secret key which is only available on the server.

So the hacker cannot generate this digital signature, because it is stored on the webserver

More on that here.

8. Generating Authentication Tokens

Now instead of sending a simple true response to the clients when authenticated, we will generate a JWT token to authenticate the user and later verify the same from the jwt.io site to check our tokens are working properly.

router.post('/', async (req, res) => {
   const { error } = validateUser(req.body)
   if (error) return res.status(400).send(error.details[0].message);
   let user = await userClass.findOne({email: req.body.email})   if (!user) return res.status(400).send('Invalid Email.');
   const validPassword = await bcrypt.compare(req.body.password, 
                                              user.password)
   
   if (!validPassword) return res.status(400).send('Invalid 
       Password.');
   try{
      const token = jwt.sign({ _id: user._id, email: user.email },       'jwtPrivateKey') // Payload + Private Key
   res.send(token)
   }catch (ex){
      console.log('Error in Promises', ex.message)
      res.status(400).send( ex.message)
      }
})

Let me show you how we hit the API and later verify the JWT token we have received.

JWT Token on the left
The payload on the left for the received JWT token

Now when we paste the JWT token we received via POST request here, we can see the corresponding payload is what we sent previously to generate the JWT.

9. Setting Response Headers

So previously when we create a new user, we only sent the name, email in the body. Now we can also send the JWT token also, but it would be appropriate to store the JWT token in the header response of the received POST request.

So that the browser can store that JWT token in the header variable and we can authenticate it directly.

So let’s make some modifications to the new user creation and sent the token to the return header with x-auth-jwtoken variables.

router.post('/', async (req, res) => {
  const { error } = validateUser(req.body)
  if (error) return res.status(400).send(error.details[0].message);  let user = await userClass.findOne({email: req.body.email})
  if (user) return res.status(400).send('User Already Registered.');  user = new userClass(_.pick(req.body, ['name', 'email', 
'password']));
  const salt = await bcrypt.genSalt(10);
  user.password = await bcrypt.hash(user.password, salt);  try{    await user.save()
    const token = jwt.sign({ _id: user._id, email: user.email }, 'MySecurePrivateKey')
    res.header('x-auth-jwtoken', token).send(_.pick(user, ['_id', 'name', 'email']))
  }catch (ex){
    console.log('Error in Promises', ex.message)
    res.status(400).send( ex.message)
  }
})
So in the headers section, we see the new variable. x-auth-jwtoken

10. Authorization Middleware

Now that we have a separate user base and we send the identification key of the JWT token to their browser headers directly upon account creation. Now when these users perform any operation on our movie databases like Get or POST — create a new entry, we need to authorize them based on their JWT token in their headers

So if you remember the middleware function from the previous section, we used a middleware which authenticates almost all of the request without specifying them explicitly in all codes.

So will create a new function in the middleware folder for authorization.

// CUSTOM FUNCTIONS - For Authfunction auth(req, res, next){
   console.log('Authentication Starts...');
   const token = req.header('x-auth-token');
   
   if (!token) return res.status(401).send('Access Denied. No Token Provided')   try{
     const decoded = jwt.verify(token, 'MyPrivateKeyJWT');
     req.user = decoded;  
     next();
   }catch(ex){
     res.status(400).send('Invalid Token')
   }
}

Now we don’t want to authorize users in every place, so let’s say only when the user is performing any operation near to the movie database, then we add the authentication to perform.

So in the movies route handlers, we shall add the code.

const auth  = require('../middleware/auth')// Route, middleware, function
router.get('/', auth, async (req, res) => {
  const allMovies  = await movieClass.find().sort('movieName')
  res.send(allMovies)
})

Now if we try to hit the GET API, we must get an error. Because we haven’t sent the auth token in the headers.

Response for no token at all
Response for invalid token
Passing the correct auth token for the user

Likewise, you can protect most of the routes you need to secure.

11. Getting the Current User

Sometimes, we need to record who is performing which API request, so in the route handlers, we would like to know the details of the user. So let’s enable the user identification inside the routes.

const auth  = require('../middleware/auth')
const {userClass, validateUser} = require('../models/users')router.get('/me', auth, async(req, res) => {
   const user = await userClass.findById(req.user._id).select('-
password')
   res.send(user);
});

X. Handling and Logging Errors

Sometimes in a real-world scenario, we might get into issues. So we cannot send the user, that we have an issue, instead, we can send a friendly error message to the user and also log the error, so that we can check it and resolve it at the earliest. So in the upcoming section, we might learn how we can handle and log the errors.

1. Handling Rejected Promises

When we are dealing with promises, we need to either put a .then, .catch blocks to handle those promises.

But if we are using async and await functionalities, then we must use try, catch blocks to handle the rejected promises, otherwise, the process will be terminated.

So while getting the movie's operation, if the database is not ready, we shall send a friendly error, stating we are not ready not serve.

// Route, mware, functionrouter.get('/', auth, async (req, res) => {
  try{
    const allMovies  = await movieClass.find().sort('movieName')
    res.send(allMovies)
  }catch (ex){
  // Log the error
  res.status(500).send('Something failed. Please try again')
}})

2. Express Error Middleware

There could be a thousand routes we have developed with custom error messages and try-catch blocks, we cannot modify each of them independently, so we need to find a way to handle them all in one place, particularly in middleware.

So this implementation will be tricky, so I would prefer you see the GitHub repository attached to this article for more clarity.

We create an error handling function so that we can utilize it everywhere.

// Error handlingfunction errorHandling(err, req, res, next){  // Log the Exception  res.status(500).send('Something failed. Please try again')}module.exports = errorHandling

And use it as middleware and call it in the index.js file, app.use(errorHandling).

Also, we are using a lot of try-catch blocks in almost every route, so we define a generic function to handle it and pass out arguments to our routes.

function asyncMiddleware(handler){
  return async (req,res, next) => { 
  // Standard express route handler
  try{
    await handler(req, res);
  }catch(ex){
    next(ex)
}}}module.exports = asyncMiddleware

Then wrap our routes and pass the handler to the function.

router.get('/', auth, asyncMiddleware(async (req, res) => {  const allMovies  = await movieClass.find().sort('movieName')  res.send(allMovies)}));

3. Using express-async-errors

As we wrap all our routes inside the asyncMiddleware function, it is again cumbersome to do it, luckily there are a package express-async-errors while does the job for us.

To install it and call it in the index.js like require('express-asunc-errors') and it will take care of all that routes, so you can remove the asyncMiddleware wrap from the routes.

4. Logging Errors

For the purpose of logging errors, we can use a package called Winston.

It has Transport methods like console, HTTP, file to either just display the error or send the error to an HTTP address or save it to a file.

winston.add(new winston.transports.File({ filename: 'combined.log' }))

Also, it has extension plugins for storing it in MongoDB, Redis, etc.,

So, we modify the error handler to log the message in a file.

function errorHandling(err, req, res, next){   // Log the Exception   winston.log('error', err.message, err) 
   // Logging level - error,    warning, info, verbose, debug, silly   res.status(500).send('Something failed. Please try again')}module.exports = errorHandling
combined.log

More on that here.

5. Logging to MongoDB

To store the logs in MongoDB directly without manually creating a CRUD operation, we can make use of another library from Winston.

// npm install winston-mongodb@3.0.0winston.add(new winston.transports.MongoDB({
  db: 'mongodb://localhost/movies',
  level: 'error' 
}))

That’s it, it will log the errors directly to MongoDB if we face any errors.

Errors in MongoDB

6. Handling Uncaught Exceptions

Not all the errors occur inside the routes. Sometimes we might get errors outsides and the present implementation cannot find them, so we need to add a new method to Winston to log these errors as well.

process.on('uncaughtException', (ex) => {
  console.log('Some uncaught Excpetion')
  winston.error(ex.message, ex)
})// throw new Error('Something went down')

7. Unhandled Promise Rejections

Here we handle both the unhandled exceptions and unhandled promise rejections. Caution: It will stop the node server.

winston.handleExceptions(
new winston.transports.File({filename: 'uncaughtExceptions.log'})
)process.on('unhandledRejection', (ex) => {
  throw ex;
})// Error simulation
const p = Promise.reject(new Error('Promise failure'))
p.then( () => console.log('Done'))

8. Code Refactoring

The startup file index.js should not contains all the parameters we use in our project, so we must separate the routes, logging, database, etc., in a different file for easier debugging.

So I have done that and pushed the new codes in the Github Repository, where you can pull it and play along.

Congratulations on reaching the end.

References & Repositories

Contains all the codes and dependencies related to this article can be found here: https://github.com/bala-codes/A-Primer-on-Node.js

Until then, see you next time.

Article By:

BALAKRISHNAKUMAR V

Nodejs
Node Js Tutorial
Crud
Mongodb
Programming

--

--

Balakrishnakumar V
Towards AI
Follow

Written by Balakrishnakumar V

116 Followers
Writer for

Machine and Deep Learning Engineer

Follow

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams