The Modern API, Part 1

At decent labs we know a thing or two about building software. In our pursuit of client happiness paired with using solid software fundamentals to build resilient, scalable systems, we've iterated through the process of building APIs maaaany times.

I took the time yesterday to (start to) take all of my current best practices around API design and put them into one "boilerplate" repo.

TL;DR: https://github.com/decentorganization/decent-api

This post kicks off a series of blog posts aimed at walking through and explaining this repository in detail.

To let readers know which technologies, libraries, and concepts we'll be going over (and to help with SEO):

• highly-supported runtimes via <code>nodejs<code>
• simple servers via <code>express<code>
• postgres via <code>docker-compose<code>
• programmatic sql via <code>knex<code>
• extreme access via <code>cors<code>
• es6 support via <code>babel<code>
• testing via <code>mocha<code> and <code>chai<code>
• logging via <code>debug<code> and <code>morgan<code>
• error handling via custom errors
• linting via <code>eslint<code>
• hot reloading via <code>nodemon<code>

Let's go!

Development Dependencies

We need to install some tools on our system to support the project and tests actually running locally. Then we'll need to install some packages into our project to support the various repetitive tasks of developing code.

System Development Dependencies

We'll be using a couple of tools to execute our development environment, namely <code>git<code>, <code>nodejs<code>, <code>yarn<code> (instead of <code>npm<code>), and <code>docker<code>. I'll leave this up to the reader.

Tip: for <code>nodejs<code> and <code>yarn<code>, use some sort of version manager, like <code>nvm<code> and <code>yvm<code>. It's a good idea for when you're working with different projects with different runtime requirements on your computer. This isn't necessary though, installing from your favorite package manager, or by downloading from their websites, will work just fine.

Project Development Dependencies

If you're creating a new project, go ahead and <code>yarn init<code> your way into it. If you're adding the contents of this guide into an existing project, good luck!

Now we're getting into it! Let's install a bunch of packages to aid with development. Run this in your terminal:

Quickly:

• <code>@babel/*<code> packages are for transpiling our fancy modern Javascript into a more vanilla Javascript
• <code>mocha<code> and <code>chai<code> are for the test suite
• <code>dotenv<code> is for our environment variables
• <code>eslint-*<code> is for linting
• <code>nodemon<code> is for watching for file changes on the system
• <code>rimraf<code> is for deleting shit, like when we're creating a production build and want to clean the project first

Now we've got a <code>node<code> project with some tools installed. Neat.

Configuration

I like to build projects in a way that use as much "default" configuration options from third party tools as possible. This just keeps things as simple as possible.

Sometimes, though, that's just not possible. Let's create some new config files for some of our tools.

git

If you've ever worked in a <code>git<code> repository before, this should be of no surprise. Need that <code>.gitignore<code>. I like to use https://gitignore.io for my <code>.gitignore<code> needs. Here's what gitignore.io has for the standard <code>node<code> project: https://gitignore.io/api/node.

There's one more thing to add to our <code>.gitignore<code>, which will be the directory that production builds get transpiled into. We'll be calling that directory <code>dist<code>, so add an entry for <code>dist<code> in your <code>.gitignore<code>, too.

Here's the <code>.gitignore<code> from the final repo: https://github.com/decentorganization/decent-api/blob/master/.gitignore

I've also got and entry for <code>.vscode<code> in there, too, because I like to tell VSCode to auto-format on save, for certain projects, and I don't want those options part of the repo: they're really just my preference.

dotenv

To manage environment variables on our development machines, we'll use <code>dotenv<code>. This package relies on a file called <code>.env <code> to read environment variables from, and inject them as actual system environment variables into the running <code>node<code> process.

<code>.env<code> is <code>.gitignored<code>, so I like to create a file called <code>.env.example<code> and commit that into the repository. In the README, instruct developers to copy the contents from <code>.env.example<code> into a new (ignored) file named <code>.env<code>.

For our project, the <code>.env(.example)<code> file will hold a port number on which the API should be exposed (<code>API_PORT<code>), database connection information (<code>DB_*<code>) and a "base" logger string (<code>LOGGING_BASE<code>), which can be anything. It's used to prefix logs.

https://github.com/decentorganization/decent-api/blob/master/.env.example

All of the values in this <code>.env.example<code> are defaulted to in the actual codebase. A developer really only needs to create their own <code>.env<code> file if they want to override anything here.

babel

We'll use <code>babel<code> to transpile our nice ES6-style Javascript into vanilla Javascript. To configure <code>babel<code>, we'll use a <code>.babelrc<code> file.

Honestly, I haven't taken much time trying to understand the nuances of <code>babel<code>, so I'll just show you what works for our project. This works for our needs:

https://github.com/decentorganization/decent-api/blob/master/.babelrc

eslint

We're using ESLint to lint our code, and because we're writing new style Javascript with ECMAScript 9 (2018), we'll need to configure ESLint appropriately with an <code>.eslintrc<code> file.

https://github.com/decentorganization/decent-api/blob/master/.eslintrc

nodemon

<code>nodemon<code> is a simple package that monitors for changes on your filesystem, and kicks of scripts when they do. We'll use <code>nodemon<code> to watch for changes to our source code and test code, then to run the linter and run the tests and restart the development server.

We will ignore the specific directory that holds migration files, because things get weird if those get executed before we're done writing them. (The app executes migrations automatically upon startup, so if we're restarting it in the middle of writing a migration via <code>nodemon<code>, a half-formed migration will be executed against the development database and sucks.)

https://github.com/decentorganization/decent-api/blob/master/nodemon.json

docker

We're using docker in this project for one very specific purpose: to provide us with a database. We're not dockerizing our project (yet lol).

We'll use <code>docker-compose.yml<code> to specify that we just need a basic <code>postgres<code> service.

https://github.com/decentorganization/decent-api/blob/master/docker-compose.yml

The Non-Development-Specific Dependencies

Now we'll need to install the packages which will support and enable the actually running API process. Do this in your terminal:

Quickly:

• <code>express<code> is the minimalist web framework
• <code>cors<code> is a little <code>express<code> middleware package that enables CORS for our API
• <code>express-list-endpoints<code> is a nice package that enables us to output a list of all the endpoints of our API. Good for self-documentation.
• <code>knex<code>, <code> knex-db-manager<code>, <code>pg<code>, and <code>pg-escape<code> are all to support interacting with our <code>postgres<code> database from the codebase
• <code>debug<code> and <code>morgan<code> are for logging

Now we've got all of the tools and packages installed that we'll need to support writing some actual code.

Development Scripts

One more thing to do before we can start writing code. We'll add a bunch of scripts to <code>package.json<code> to help us do some common tasks. Add the following <code>scripts<code> object into your <code>package.json<code>:

• <code>build<code>: uses <code>babel<code> to transpile the <code>source<code> code into a <code>dist<code> directory
• <code>clean<code>: deletes the <code>dist<code> directory
• <code>dev<code>: sets the <code>NODE_ENV<code> environment variable to <code>development<code>, then loads the rest of the environment variables from <code>.env<code> and kicks off the <code>dev:logs<code> script
• <code>dev:logs<code>: sets the <code>DEBUG<code> environment variable to whatever we had set in <code>.env<code> via <code>LOGGING_BASE<code>, then kicks off the <code>server<code> script
• <code>lint<code>: lints the source code
• <code>migration<code>: creates a new <code>knex<code> migration and sticks it in the <code>./src/database/migrations directory<code>
• <code>prod<code>: cleans and builds the project using existing scripts then starts the server via <code>server:prod<code> script
• <code>server<code>: uses <code>babel-node<code> to kick off a development server using the <code>src<code> directory
• <code>server:prod<code>: sets the <code>NODE_ENV<code> environment variables to <code>production<code> then starts node by pointing at the <code>dist<code> directory, which is where our built production code lives
• <code>start<code>: simply an alias for the <code>prod<code> script
• <code>test<code>:sets the <code>NODE_ENV<code> environment variables to <code>test<code> then runs <code>mocha<code> with some <code>babel<code> configuration options to allow for our modern Javascript syntax
• <code>watch<code>: starts <code>nodemon<code>

See the final <code>package.json<code> here: https://github.com/decentorganization/decent-api/blob/master/package.json

PHEW

And there we have it, our project is finally set up.

Without yet writing a single line of code, we've:

• installed some system development dependency tools
• installed some project development dependency packages
• configured our development tools
• installed project dependency tools
• created some scripts to help us do common development tasks

Join me in the next post and we'll start diving into some real code!

Questions? Comments? Ask away over at the Medium post