CRUD with the MongoDB Node.js SDK

Justin G.
Justin G.
Retool

May 6, 2022

Welcome, readers. If you’ve made it this far, you’re probably building an app on top of MongoDB and want some help navigating your selectOnes, understanding projections, figuring out the difference(s) between insertMany and bulkWrite, and deciphering the loads of other methods they didn’t teach you in SQL class. Well, we stewed up this tutorial just for you and your loved ones, so read on.

Installing the Node driver and connecting to your Atlas cluster

Step one to querying your database is connecting to it, and thankfully Atlas makes this rather easy.

Install MongoDB’s Node.js driver

First, make sure the mongodb package is installed (whether you’re making queries from the client or the server). A simple npm install in your project directory does it:

npm install mongodb

Note that if you’re using TypeScript in your project you’ll need to use the driver’s definitions:

npm install -D @types/node

With the driver installed, you’ve only got a few steps left before you can write queries: configuring database access, figuring out your unique database URI, and creating an instance of the MongoDB client with it in your application.

Create a database user

Like other databases, MongoDB requires you to create a database user (not to be confused with the user on your Atlas account) to be registered for different access patterns. This makes things more secure as you can allocate individual permissions, see which users did what to which data, etc. To create a database user, head over to your “Database Access” tab under the “Security” section. Click “add a new database user” and you’ll be presented with a lot of options:

You can choose from types of auth, specific roles and privileges, and even restrict users to specific datasets.

Whitelist the IP address of the server or client you want to connect

Head over to the “Network Access” setting in that same security tab, and there you’ll be able to add a whitelisted IP address. The UI is pretty simple and allows for comments—and a sort of self-destruction feature:

Get your unique connection URI

Your connection URI is how MongoDB knows what cluster to connect to among a sea of Atlas instances. Here’s the basic structure:

You can find it by heading to your Atlas homepage, finding your database, and clicking “connect.”

Choose “Connect your application” and configure the driver and version. Since we’re using the Node SDK, we’ll choose “Node.js” and the most recent version (4.0 or later as of May 2022).


Copy that URI, and we’re ready for the final step. I usually trip up on the fine print: note that both the username/password and database are placeholders, and you’ll need to update them with your desired values. For simplicity’s sake, I personally like to remove everything after the final slash (the database name, and retry writes setting) and just deal with that in code. So then my simpler, final URI looks like:

1mongodb+srv://:@retool-test.betz7.azure.mongodb.net/ 

Create an instance of the MongoDB client in your app

We’ll just need to import that MongoDB client we installed, and instantiate it with the URI:

1const { MongoClient } = require("mongodb");
2
3const uri = "mongodb+srv://<username>:<password>@retool-test.betz7.azure.mongodb.net/"
4
5const client = new MongoClient(uri);
6
7console.log(client)

Make sure to put your database user’s username and password in those placeholders as well! And with that, you should be successfully connected to your Atlas cluster and ready to run some queries, which is what you came here for.

Read ops: selecting, cursors, sorting, and search

Retrieving basic, unaggregated data in MongoDB is done via the find() command.

Basic selections and queries with findOne()

To read data from MongoDB, we’ll first want to construct some sort of query that gives the database specific parameters around what we want. For example, let’s say we want to look at all listings with more than 2 bedrooms on Airbnb. Here’s the JavaScript we’d need:

1async function run() {
2    try {
3        await client.connect() const db = client.db("sample_airbnb");   
4        const listingsAndReviews = db.collection('listingsAndReviews');
5        const query = {
6            "bedrooms": {
7                "$gt": 2
8            }
9        }
10        const results = await listingsAndReviews.findOne(query)
11        console.log(results)
12    } finally {
13        await client.close()
14    }
15}
16
17run() 
18
19// {_id: '10006546', listing_url: 'https://www.airbnb.com/rooms/10006546', name: 'Ribeira Charming Duplex', summary: 'Fantastic duplex apartment with three bedroo…ed, without losing their original character.', space: 'Privileged views of the Douro River and Ribei…nerosity, wrapped in a only parochial spirit.', …}

Since we’re using async-await, all of these statements need to be nested in an async function; code examples from here on out won’t include that extra boilerplate, but keep in mind that you’ll need to run them inside of an async function. After some boilerplate for choosing our database and collection, we define a query that tells MongoDB we want any documents that match our criteria–more than 2 bedrooms–using the $gt or greater than operator. Note that we’ve chosen the findOne() method here, which returns just a single result that matches our query.

If we wanted to add multiple filter conditions to our query, that’s no problem:

1const query = {
2    "room_type": "Entire home/apt",
3    "bedrooms": {
4        "$gt": 4
5    }
6}
7
8// {_id: '1066648', listing_url: 'https://www.airbnb.com/rooms/1066648', name: 'Colonial Mansion in Santa Teresa', summary: 'A beautiful colonial mansion for office or r…ion, with five bedrooms and three bathrooms.', space: 'Built before the turn of the century, this fu…ops and lounges are all located nearby as wel', …}

Since MongoDB is NoSQL after all, some of these records have mismatching schemas. We can write a query that simply checks that a certain field exists:

1const query = {
2    "reviews": {
3        "$exists": true
4    }
5}
6
7// {_id: '10006546', listing_url: 'https://www.airbnb.com/rooms/10006546', name: 'Ribeira Charming Duplex', summary: 'Fantastic duplex apartment with three bedroo…ed, without losing their original character.', space: 'Privileged views of the Douro River and Ribei…nerosity, wrapped in a only parochial spirit.', …}

Thankfully, our charming Ribeira duplex does indeed have some reviews.

There are seemingly endless ways to customize your queries: check out MongoDB’s full documentation here.

Working with cursors

All the above examples use findOne() for simplicity, but chances are you want to run queries that have more than one result. When you run a regular old find() that returns multiple results, MongoDB gives them to you in a navigable cursor object. To preview the results, you can run a forEach loop and print each result:

1const query = {
2    "room_type": "Entire home/apt",
3    "bedrooms": {
4        "$gt": 4
5    }
6}
7
8const cursor = await listingsAndReviews.find(query)
9await cursor.forEach(console.dir);

You can easily convert the cursor to an array as well:

1console.log(await cursor.toArray())
2

Or, if you want to get a count of total results:

1const cursor = await listingsAndReviews.find(query)
2console.log(await cursor.count())

Technically, this is actually bad practice: there’s a special type of query for counting the number of documents in a collection that circumvents the need for a cursor in the first place:

1const result = await listingsAndReviews.countDocuments(query)
2

Working with cursors via the MongoDB Node.js driver can be a bit confusing, since the documentation is rather sparse and the MongoDB shell documentation explicitly says that those docs aren’t relevant to drivers. So Mongo, if you’re reading this, please take note.

Projections with .project()

You probably don’t need all of the fields from every document returned in each query. MongoDB allows you to specify which fields you want with projections:

1const cursor = await listingsAndReviews.find(query).project({ “bedrooms”: 1 })
2

Adding a field with a 1 means explicitly including it, and using a 0 explicitly excludes it. By default, every field gets returned, so use projections when you can to minimize the amount of data transfer.

Sorting results with .sort()

The easiest way to sort query results is to attach a sort() method chained to your find() method. Here’s how you’d sort results by price, descending:

1const cursor = await listingsAndReviews.find(query).sort({ "price": -1 })
2await cursor.forEach(console.dir)
3

Replacing that -1 with 1 sorts results ascending. You can add multiple levels of sorting by adding new entries to that sort object, like so:

1const sort = {
2    "price": -1,
3    "cleaning_fee": -1,
4    "extra_fee": -1
5}
6
7const cursor = await listingsAndReviews.find(query).sort(sort)
8await cursor.forEach(console.dir)

Now we have a better sense of how much these places actually cost with extra fees included.

By the way, if you’re doing both projections and sorts, it eventually gets simpler to just add a single configuration object to your query instead of chaining all of these methods together:

1const options = {
2    sort: {
3        "bedrooms": -1
4    },
5    projection: {
6        "bedrooms": 1
7    }
8}
9
10const cursor = await listingsAndReviews.find(query, options)
11await cursor.forEach(console.dir)

Keep in mind that you need to pass your query object before your options object.

Searching with text indexes

There are two ways to add search to your read queries in MongoDB: for text and for geospatial data. I’m going to make the (hopefully) reasonable assumption that it’s probably text you’re most interested in. To text search in MongoDB, you first need to create an index on the text field that you want to search. Here’s how we’d create a text index for each listing’s name field:

1db.listingsAndReviews.createIndex({ name: "text" });
2

Note that you can only create one text index per collection. So if we wanted to index multiple text fields, we’d do it like so:

1db.listingsAndReviews.createIndex({ name: "text", description: “text” });
2

That would mean that every text search would search both of these fields.

Once we’ve created our index, we can search for a specific word or set of words easily:

1const query = {
2    "$text": {
3        "$search": "beach house"
4    }
5}
6
7const cursor = await listingsAndReviews.find(query).project({
8    "name": 1
9})
10await cursor.forEach(console.dir)
11
12// 
13{
14    _id: '5549057',
15    name: 'BONDI BEACH - Beach & Ocean Views'
16} {
17    _id: '4774436',
18    name: 'Charming Beach Cottage in Ewa Beach'
19} {
20    _id: '24187105',
21    name: 'Room in beach house 200m to beach'
22}

There’s a lot more you can do than just text search: check out the docs to go deeper.

Write ops: inserting and the ubiquitous _id

You’ve got 2 methods for inserting data into your MongoDB collection: insertOne() and insertMany(). In MongoDB, each document in a collection has a unique identifier, the _id. You can manage it yourself (set a specific one for each document), or allow MongoDB to do that behind the scenes for you. For this tutorial, we’ll assume the latter, which generally is best practice.

Inserting a single new record with insertOne()

This is pretty basic. You just define the document you want to insert, and then, well, insert it.

1const doc = {
2    "name": "cozy studio in the heart of brooklyn",
3    "square_feet": 100,
4    "bedrooms": 0
5}
6
7const result = await listingsAndReviews.insertOne(doc)
8console.log(result)
9
10// 60c79c0f4cc72b6bb31e3836

MongoDB returns the _id of the newly inserted document by default.

Inserting multiple new records with insertMany()

The insertMany () method inserts multiple documents, one at a time, until your list of docs is exhausted or an exception occurs.

1const docs = [{
2        "name": "cozy studio in the heart of brooklyn",
3        ...
4    },
5    {
6        "name": "charming bathroom in new york city",
7        ...
8    },
9    {
10        "name"
11        "alcove in bushwick collective",
12        ...
13    }
14]
15
16const result = await listingsAndReviews.insertMany(docs)
17console.log(result.insertedIds)

It’s worth couching some of this in catching logic to make sure that everything was inserted as planned. The main reason there would be an error is mismanaging the _id – if you provide IDs yourself and accidentally have duplicates, the insert will fail on the first duplicated ID.

Update ops: simple updates and array updates

Updating documents in MongoDB is perhaps surprisingly a lot more complex than inserting them. There are two main concepts to focus on:

A query so the database knows which document(s) to update
An update document to replace the existing one(s) with

There are a bunch of options for what you want the callback to be. An interesting option is the findOneAndUpdate() method, which returns the entire original or updated document.

In terms of what you can actually do to existing documents, there are a bunch of operators. The most popular one is $set which just replaces the entire document or field with a new value that you pass through. But you can also remove fields, perform atomic operations like adding and multiplying, or update a date to the current date (specific, I know).

Updating a single document with updateOne()

Let’s say we want to update the name of a specific listing in our database (imagine this is tied to some sort of edit functionality in a user-facing UI). Here’s how we’d do that with updateOne(), including finding the document we need and then passing in a new value for the name:

1const filter = {
2    "name": "Ribeira Charming Duplex"
3}
4
5const updateDoc = {
6    "$set": {
7        "name": "Riviera Charming Duplex"
8    }
9}
10
11const result = await listingsAndReviews.updateOne(filter, updateDoc)
12console.log(result.matchedCount)
13
14// 1

Remember that the filter gets passed before the update document. Here we’re using $set to replace the old value for name with our new, slightly better one. The .matchedCount property of the result tells you how many documents matched that filter, which is important in cases where the filter you provided isn’t entirely unique. In our case, it was, so it returns 1. And since this is updateOne(), even if more than 1 result is matched, MongoDB will only actually perform the update on the first matched document.

We can remove a field with $unset:

1const filter = {
2    "name": "Ribeira Charming Duplex"
3}
4
5const updateDoc = {
6    "$unset": {
7        "reviews": ""
8    }
9}
10
11const result = await listingsAndReviews.updateOne(filter, updateDoc)
12console.log(result.matchedCount)

A kind of funny thing here is that it doesn’t actually matter what value you put in your JSON object for the unset operation. It needs to be something, so the object is valid, but since you’re removing the specific field, the value itself doesn’t matter.

You can also upsert by adding an option to the query:

1const options = { upsert: true };
2const result = await listingsAndReviews.updateOne(filter, updateDoc, options)
3

The options param gets passed last, after the filter and updated document.

Updating a multiple documents with updateMany()

Even though updateOne() may match multiple documents, it will only update one; whereas updateMany() allows you to update multiple documents at once. If we wanted to apply a uniform price increase to every cleaning fee on our Airbnb listings with more than 2 bedrooms, we could do the following:

1const filter = {
2    "bedrooms": {
3        "$gt": 2
4    }
5}
6
7const updateDoc = {
8    "$inc": {
9        "cleaning_fee": 5
10    }
11}
12
13const result = await listingsAndReviews.updateOne(filter, updateDoc)
14console.log(result.matchedCount)

Here we’re using the $inc update operator to increment the cleaning fee by $5. Damn inflation.

Remember that you can pass a custom callback to write functions in MongoDB. So if you want to log some information to the console about what happened with your multiple updates, you can do that:

1const result = await listingsAndReviews.findOneAndUpdate({
2        name: "Rebeira Charming Duplex"
3    }, {
4        $set: {
5            name: "Riviera Charming Duplex"
6        }
7    }, {},
8    function(error, result) {
9        if (!error) {
10            console.log(`Things went smoothly: ${result.ok}`);
11        } else {
12            console.log(`Things did not go smoothly: ${error}`);
13        }
14    },
15);

Updating arrays

If you have a document with a field that contains (or is) an array, updating that field requires some special syntax. This deserves its own section as it’s a bit more on the complex side.

In our sample Airbnb dataset, each listing has an array of reviews. Here’s what a single review object in one of those arrays looks like:

1{
2    "_id":"58663741",
3    "date": "2016-01-03T05:00:00.000+00:00",
4    "listing_id": "10006546",
5    "reviewer_id": "51483096",
6    "reviewer_name": "Kate",
7    "comments": "A casa da Ana e do Gonçalo foram o local escolhido para a passagem de ..."
8}
9

Our application team wants to build a feature that allows users to update their review comments on a specific listing. So we’ll need to craft a query that finds the user’s review in the array, and then updates just the comments field. First, let’s figure out our query for finding the array entry in question:

1const query = { "name": "Riviera Charming Duplex", "reviews.reviewer_name": "Kate" }
2

This query filters for the specific listing we want to look at, and pulls only the review with the name of our user. This is obviously an oversimplification–you’ll need to make sure that the query values actually return unique results, as there may be multiple listings with the same name, or reviews with the same reviewer name. We’re using normal dot notation to reference the reviewer_name field.

With our document specified, we can pass a new value for that comments value:

1const query = {
2    "name": "Riviera Charming Duplex",
3    "reviews.reviewer_name": "Kate"
4}
5
6const updateDocument = {
7    "$set": {
8        "reviews.$.comments": "Changed my mind, it was fine!"
9    }
10}
11
12const result = await listingsAndReviews.updateOne(query, updateDocument)
13console.log(result.matchedCount)

That $ is called the positional operator, and it tells MongoDB that in the case of multiple matching array elements, it should update only the first one. If you want to update all matching elements in the array, you can use the all positional operator ($[]):

1const updateDocument = {
2    "$set": {
3        "reviews.$[].reviewer_name": "Cate"
4    }
5}

There’s a lot more to cover here: check out the docs for a more in depth look.

Delete ops: this one is pretty simple

If you can find a document, you can delete it:

1const query = { "name": "Riviera Charming Duplex" }
2
3const result = await listingsAndReviews.deleteOne(query)
4console.log(result)
5
6// {acknowledged: true, deletedCount: 1}

As with other operations we’ve covered, deleteOne() deletes the first document that matches your condition, while deleteMany() deletes all documents that match your condition:

1const query = { "bedrooms": 0 }
2const result = await listingsAndReviews.deleteOne(query)

Improving performance with bulkWrite()

The bulkWrite() function allows you to batch lots of different write operations together in a sort of recipe instead of performing all of them separately. So if there are any workflows that you want to kick off when a user takes an action, you can batch them all together.

Let’s say we’re working on some functionality for handling when a user deletes a review that they’ve posted. In addition to deleting that review from the reviews array, we also need to decrement the number_of_reviews field, update the first_review and last_review timestamps, and change the review_scores object to match the new data. A lot to do! We could do all of these separately, or put them together into a single bulkWrite() call.

The first thing we’ll do is define that recipe of operations:

1const operations = [
2    // Step one: delete Kate's review
3    {
4        deleteOne: {
5            filter: {
6                "name": "Riviera Charming Duplex",
7                "reviews.$.name": "Kate"
8            }
9        }
10    },
11    // Step two: update the last review timestamp
12    {
13        updateOne: {
14            filter: {
15                "name": "Riviera Charming Duplex"
16            },
17            update: {
18                "$set": {
19                    "last_review": "[some new timestamp]"
20                }
21            }
22        }
23    },
24    // Step three: update the number of reviews
25    {
26        updateOne: {
27            filter: {
28                "name": "Riviera Charming Duplex"
29            },
30            update: {
31                "$inc": {
32                    "number_of_reviews": -1
33                }
34            }
35        }
36    }
37    // Step four: update review scores
38    ...
39    ...
40    ...
41]

Note that there’s a special syntax for each operation (read more in the docs here). With our operations array defined, we can now pass it into a bulkWrite() call:

1const result = await listingsAndReviews.bulkWrite(operations)
2

Now, all of these operations get performed with only one round trip to the database. Note that bulk writes in MongoDB are not equivalent to ACID transactions in a traditional database. You can work with transactions in MongoDB, but according to their official documentation, “80%-90% of applications that leverage the document model will not need to utilize transactions in MongoDB.” You can read more here.

Thoughts or comments? Get a discussion going on Twitter: we’re at @retool.

Reader

Justin G.
Justin G.
Retool
May 6, 2022
Copied