I use Azure DevOps Server 2020 with self hosted agents and I created a CI pipeline which executes all my tests in parallel on one agent. The ~5000 tests (no UI tests) need around 7min to complete.Too speed things up I added 3 extra agents, put the task into another job in the same pipeline with parallel: 4. All 4 agents first download the build artifacts and then run a slice of the tests. But unfortunately the test run isn’t faster but instead slower, every agent needs around 8 min.

My vstest yaml for 1 agent

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            ***test*.dll
            !***TestAdapter.dll
            !***TestFramework.dll
            !**obj**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      rerunFailedTests: false

My vstest yaml for 4 agents

- task: VSTest@2
  displayName: 'Run tests'
  inputs:
      testSelector: 'testAssemblies'
      testAssemblyVer2:
            ***test*.dll
            !***TestAdapter.dll
            !***TestFramework.dll
            !**obj**
      searchFolder: '$(System.ArtifactsDirectory)'
      runInParallel: true
      codeCoverageEnabled: false
      distributionBatchType: 'basedOnExecutionTime'
      rerunFailedTests: false

I also tried batching by assembly and based on number of tests + number of agents, but nothing seems to work.

What makes it even stranger is that I got an old UI based CI pipeline, with multi-config and multiplier on a variable with 4 TestCategories, which distributes even more tests ~10000 (with my 5000 too) but as mentioned by TestCategory on the same 4 agents (Cat1 on agent1, Cat2 on agent2 and so on) and the slowest one finishes in less than 4 minutes.

The yaml of the UI based one looks like this:

steps:
- task: VSTest@2
  displayName: 'Run tests'
  inputs:
    searchFolder: '$(Build.BinariesDirectory)'
    testFiltercriteria: 'TestCategory=$(Tests)'
    runInParallel: true
    codeCoverageEnabled: false

I think I have to be missing something obvious.

Thanks in advance!

Edit 1:

I connected to my agents with RDP and inside the task manager there are multiple instances of testhost.x86 running, up to 8 simultaneously but not constantly. If I start my tests locally the 8+ instances of testhost.x86 are up almost all the time and rarely vanish at all. If that’s any help.