How to execute JavaScript concurrently while specifying the number of concurrent executions

There are times when you have many tasks that you want to process in parallel, but executing all of them in parallel would be too heavy, so you want to specify the number of tasks to be executed concurrently.

I am ashamed to admit that I did not know how to do this in JavaScript, but as an alternative, I divided the tasks I wanted to process into a number of simultaneous executions, and executed each batch as Promise.all().

However, this means that the next batch of tasks is processed after all the tasks in a batch Promise.all() are finished, so it is not always a fixed number of concurrent executions, which was bothering me.

qiita.com

I see\~. So you can start as many Promises as there are concurrent executions, and then execute tasks within those Promises.

Well, I learned a lot!

I have been bothered by this for a long time and now it is resolved and clear! Thank you.

copying of a sutra

I rewrote the article as a reference because you can't learn it without actually working with your hands.

function task(workerId: number, dataId: number, input: number) {

    return new Promise((resolve, reject) => {
        console.log(`TASK START : workerId->${workerId} dataId->${dataId} input->${input}`);
        setTimeout(() => {
            console.log(`----- TASK END : workerId->${workerId} dataId->${dataId} input->${input}`);
            resolve({
                workerId,
                dataId,
                input,
                result: input * 10,
            });
        }, input * 1000);
    });
}

(async () => {

    // Data to be shared between promise
    const data = [4, 3, 2, 1, 0, 3, 2, 1]; // Data to be processed
    let dataIdx = 0;　// processing position

    const workerNum = 3; // Number of concurrent executions
    const workers = [];
    for (let workerIdx = 0; workerIdx < workerNum; workerIdx++) {

        const p = new Promise(async (resolve, reject) => {

            const results: any = [];
            while (dataIdx < data.length) {
                const idx = dataIdx;
                dataIdx++;  // Increment before await
                const res = await task(workerIdx, idx, data[idx]);
                results.push(res);
            }

            resolve(results);

        });

        workers.push(p);
    }

    // waiting for the end (of something)
    const res = await Promise.all(workers);

    // Result obtained
    for (const p of res) {
        for (const v of (p as any)) {
            console.log(v);
        }
    }
})();

The key point is to create as many Promises as there are concurrent executions and share input data among them.

Note that tasks are executed asynchronously, so remember to advance the processing position before calling the task function.

Impressions, etc.

This example is effective only when the number of input data is fixed, and cannot be used for streams or other cases where the number of data is undetermined. I tried various ways to make it work, but was frustrated...

Welcome to new things

[Technical] [Electronic work] [Gadget] [Game] memo writing

How to execute JavaScript concurrently while specifying the number of concurrent executions

copying of a sutra

Impressions, etc.

copying of a sutra

Impressions, etc.

Related Category Articles