tnpl.me

Background

I've been telling everyone to avoid Node.js for more than 3 years because I have suffered hard-to-impossible issues when using it in large-scale real-world systems. Issues are included

1. Head-of-line (HOL) blocking — it is a performance issue where one bad request or task in a queue blocks all others behind it. This is because the nature of Node.js is that JS code execution is single-threaded. Cluster mode helps a bit but it still blocks if one bad request hits a single process in a cluster.
2. Losing error stack trace context in async task — when an error happens in async context deep down in the library, it's very hard to trace back to the application layer where it starts.

I recommend using Node.js if you really need to, for example, you have to develop a web application using React, Vue, Next.js, NuxtJS, etc.

However, I've found a way to mitigate the Head-of-line blocking issue which solves the biggest pain of mine.

Head of line blocking causes unpredictable performance

Imagine there's a single Nodejs process running to serve API requests. The API endpoint queries data from the database, loops it, processes it, and returns the response. One of the users has 100,000 rows returned from the database. JS code takes around 10s to process this single request, blocking other users who have only a few rows. If we observe each API response time, all requests blocked by the bad one will have at least 10s of latency — equal to the time it takes to process that bad request.

We have many ways to mitigate this issue, for instance, doing jobs in the background, limit amount of rows returned, etc. However, if you really need to process a request that has unpredictable data size, and wants to keep the application flow the same, you could use my technique.

Cooperative scheduling

Create yieldJs function as below.

async function yieldJs() {
    return new Promise((resolve) => setImmediate(resolve))
}

Then in the tight loop, you can put this function in the middle to yield JS execution back to Node.js event loop.

async function handleRequest(req, res) {
    const rows = ... query database ...
    for (const r of rows) {
        ... process the row ...
        await yieldJs()
    }
    ... return response ...
}

What happens here is when await yieldJs() is executed, Node.js will pause the execution of this function, process something else, and then for the next event loop cycle, it will resume the execution.

I've experimented with 2 other functions: setTimeout and process.nextTick.

For setTimeout, this function has a higher overhead because of the timer. This makes JS code to be able to utilize only 15-20% of CPU.

For process.nextTick, this function is a microtask and it keeps executing the callback before everything else which still suffers from head-of-line blocking issues.

Lastly, setImmediate can utilize 80-85% of CPU in a tight loop scenario which I think is acceptable.

Final words

Even though we have a technique to mitigate head-of-line blocking, however, I still feel that Node.js has many issues preventing us from developing a maintainable and predictable performance of production web applications. Also please avoid thread-per-request language/framework for IO intensive. I recommend using Golang instead or Rust or Zig if you have experience.