How to use web worker inside a for loop in javascript?

Question

Following is the code to create a 2d matrix in javascript:
function Create2DArray(rows) {
  var arr = [];

for (var i=0;i<rows;i++) {
     arr[i] = [];
  }

return arr;
}

now I have a couple of 2d matrices inside an array:
const matrices = []
for(let i=1; i<10000; i++){
   matrices.push(new Create2DArray(i*100))
}

// I'm just mocking it here. In reality we have data available in matrix form.

I want to do operations on each matrix like this:
for(let i=0; i<matrices.length; i++){
   ...domeAnythingWithEachMatrix()
}

& since it will be a computationally expensive process, I would like to do it via a web worker so that the main thread is not blocked.
I'm using paralleljs for this purpose since it will provide nice api for multithreading. (Or should I use the native Webworker? Please suggest.)
update() {
for(let i=0; i<matrices.length; i++){
   var p = new Parallel(matrices[i]);

p.spawn(function (matrix) {
                                return doanythingOnMatrix(matrix)
    // can be anything like transpose, scaling, translate etc...
                            }).then(function (matrix) {
                                return back so that I can use those values to update the DOM or directly update the DOM here.
    // suggest a best way so that I can prevent crashes and improve performance.
                            });
}
requestAnimationFrame(update)
}

So my question is what is the best way of doing this?
Is it ok to use a new Webworker or Parallel instance inside a for loop?
Would it cause memory issues?
Or is it ok to create a global instance of Parallel or Webworker and use it for manipulating each matrix?
Or suggest a better approach.
I'm using Parallel.js for as alternative for Webworker
Is it ok to use parallel.js for multithreading? (Or do I need to use the native Webworker?)
In reality, the matrices would contain position data & this data is processed by the Webworker or parallel.js instance behind the scenes and returns the processed result back to the main app, which is then used to draw items / update canvas
UPDATE NOTE
Actually, this is an animation. So it will have to be updated for each matrix during each tick.
Currently, I'm creating a new Instance of parallel inside the for loop. I fear that this would be a non conventional approach. Or it would cause memory leaks. I need the best way of doing this. Please suggest.
UPDATE
This is my example:

blex · Answer

Following our discussion in the comments, here is an attempt at using chunks. The data is processed by groups of 10 (a chunk), so that you can receive their results regularly, and we only start the animation after receiving 200 of them (buffer) to get a head start (think of it like a video stream). But these values may need to be adjusted depending on how long each matrix takes to process.

That being said, you added details afterwards about the lag you get. I'm not sure if this will solve it, or if the problem lays in your canvas update function. That's just a path to explore:

/*
 * A helper function to process data in chunks
 */
async function processInChunks({ items, processingFunc, chunkSize, bufferSize, onData, onComplete }) {
  const results = [];
  // For each group of {chunkSize} items
  for (let i = 0; i < items.length; i += chunkSize) {
    // Process this group in parallel
    const p = new Parallel( items.slice(i, i + chunkSize) );
    // p.map is no a real Promise, so we create one
    // to be able to await it
    const chunkResults = await new Promise(resolve => {
      return p.map(processingFunc).then(resolve);
    });
    // Add to the results
    results.push(...chunkResults);
    // Pass the results to a callback if we're above the {bufferSize}
    if (i >= bufferSize && typeof onData === 'function') {
      // Flush the results
      onData(results.splice(0, results.length));
    }
  }
  // In case there was less data than the wanted {bufferSize},
  // pass the results anyway
  if (results.length) {
    onData(results.splice(0, results.length));
  }
  if (typeof onComplete === 'function') {
    onComplete();
  }
}

/*
 * Usage
 */
// For the demo, a fake matrix Array
const matrices = new Array(3000).fill(null).map((_, i) => i + 1);
const results = [];
let animationRunning = false;

// For the demo, a function which takes time to complete
function doAnythingWithMatrix(matrix) {
  const start = new Date().getTime();
  while (new Date().getTime() - start < 30) { /* sleep */ }
  return matrix;
}

processInChunks({
  items: matrices,
  processingFunc: doAnythingWithMatrix,
  chunkSize: 10, // Receive results after each group of 10
  bufferSize: 200, // But wait for at least 200 before starting to receive them
  onData: (chunkResults) => {
    results.push(...chunkResults);
    if (!animationRunning) { runAnimation(); }
  },
  onComplete: () => {
    console.log('All the matrices were processed');
  }
});

function runAnimation() {
  animationRunning = results.length > 0;
  if (animationRunning) {
    updateCanvas(results.shift());
    requestAnimationFrame(runAnimation);
  }
}

function updateCanvas(currentMatrixResult) {
  // Just for the demo, we're not really using a canvas
  canvas.innerHTML = `Frame ${currentMatrixResult} out of ${matrices.length}`;
  info.innerHTML = results.length;
}

<script src="https://unpkg.com/[email protected]/lib/parallel.js"></script>

<h1 id="canvas">Buffering...</h1>
<h3>(we've got a headstart of <span id="info">0</span> matrix results)</h3>

How to use web worker inside a for loop in javascript?

One Answer

Add your own answers!

Ask a Question