Maybe you’ve heard the phrase “Data is the new oil.”

Data is big business and with it come challenges to take that crude data and refine it so that it can be used in a number of highly valuable ways—similar to oil.

One of the biggest challenges is being able to scale databases as the volume of data grows. It must handle increasing loads of requests, deliver the data quickly, and be reliably available.

So what exactly does it mean to scale a database?

It reminds me of the original Starcraft PC game. If you are not familiar with it, don’t worry about that, I’ll explain in a moment.

Scaling is two things:

1. Handling more data
2. Handling more concurrent users

And there are 2 primary ways to do this:

1. Vertical scaling
2. Horizontal scaling

Vertical scaling is the job of adding more resources to an existing server—more RAM, more hard drives, faster CPUs. Make the single machine better. You may have it heard it said scaling up—that’s what this is.

Horizontal scaling is the job of just throwing more machines at the problem. As you know, my IT people, this is distribution. And this is called scaling out—sideways if you may.

Vertical Scaling

This is the traditional approach and it has its advantages:

• It’s simple. You just add/upgrade the hardware in the machine. There’s no need to change architecture or distribute data.
• The results are immediate. Anything leveraging that server will now run faster.
• The underlying structure of the database is still exactly the same, so no changes to application logic are required.

BUT…

There’s only so much room on a single motherboard to add hardware. There’s a ceiling to how much you can upgrade a single system.

If you want to max the components in a machine, you gotta pay. The high-end stuff isn’t cheap.

AND, you have to take the server down to upgrade it. That’s not really convenient.

Horizontal Scaling

The main advantage to horizontal scaling is that you can simply buy more computers and have them run together in a single system.

Most of the time, spending the same amount of money on multiple, less powerful machines will perform better than the same amount of money spent on one or two UBER machine.

After all, this is the foundation of what cloud service providers do.

Have Heard Of ‘Zerging’ Something?

“What’s this have to do with Starcraft?” you might be wondering.

The purpose of the game was to build up an army and destroy your opponent. Each match would last anywhere between 5 minutes and several hours and there were 3 factions you could play as: The Terran (humans), The Protoss (alien) and The Zerg (Bug aliens).

One of the most effective approaches of the Zerg was to obtain a LOT of weak creatures for the same resource cost as 1-2 soldiers that the other factions had to spend and then you could overwhelm your opponent—with more numbers.

That’s what it means to Zerg something (and usually quickly).

Horizontal scaling is a bit like the Zerg: throw in more servers into the mix in order to scale.

I could have used the tried-and-true metaphor of highway lanes (one super fast lane vs 20 lanes), but that wasn’t as fun.

Back To Reality

The main drawback, however, is that running applications and managing data across multiple computers can get complex. Your software needs to support it.

MongoDB was built for horizontal scaling. Its architecture handles data distribution natively without compromising performance or reliability.

Sharding is a notable feature of MongoDB. It distributing data across multiple servers, ensuring that each shard (or partition) holds a subset of the data. This allows for load distribution and ensures that if one server gets Zerged by say a DDOS attack and goes offline, the entire system doesn’t come to a halt.

MongoDB also uses replica sets, which maintain multiple copies of data. If one server goes down, another can take its place without interruption.

The bottom line is that MongoDB was built for the Zerg approach.

Relational Database Management Systems (RDBMS) like SQL have come a long way to support horizontal scaling, but without getting into the nitty gritty and with the caveat that this isn’t always the case—their approaches tend to be more manual, tend to have steeper trade-offs, and tend to require changes to the applications running on them.

Considerations In Horizontal Database Scaling

When horizontal scaling, there’s a tension between maintaining data consistency/correctness and maintaining data availability. You want the data to be consistent while at the same time you want the data to be available and it can be challenging to strike the right balance.

MongoDB leans towards availability, which means there might be moments when the data that gets read is out of sync across different nodes (computers), though it eventually syncs up.

SQL often employs a technique called two-phase commits to maintain strict consistency at all times. This introduces additional complexity and performance overhead.

When it comes to latency, MongoDB handles it natively with built-in mechanisms while SQL often needs third-party tools like load balancers.

And the big one is with sharding. MongoDB does it dynamically, but that doesn’t mean you can ignore a sharding strategy. With SQL, it almost always requires changes to application logic and is a manual process.

GG

“Good game,” let’s wrap things up.

The type of database you choose will be based on a number of factors.

There are many use-cases where consistency at all times is necessary and if that is a high priority, that data begins in a valid state and ends in a valid state after a write, read, whatever, at all times, then SQL will make more sense.

If speed and scaling are a priority, then a NoSQL database that natively handles horizontal scaling will shine.