Understanding concurrency in evaluation tasks and scorers

When running evaluations on datasets, it's important to understand how concurrency works to optimize performance. Here's how parallelism is handled in the evaluation framework:

Default behavior

By default, the evaluation framework launches all tasks as independent asynchronous jobs that run concurrently, with no pre-defined limit on concurrency. Each data input is processed as a single unit that includes:

1. Running the evaluation task

2. Executing all scorers for that task's output

Controlling concurrency

You can control parallel execution using the max_concurrency parameter when running an evaluation. This parameter limits how many input-task-scorer blocks can run simultaneously.

How scorers are executed

Within each input-task-scorer block:

• The evaluation task runs first

• Once the task completes, all scorers for that input run concurrently

The thread pool size can impact when scorers begin execution. A larger thread pool may allow scorer execution to begin sooner as tasks complete.

For more details on limiting concurrency, see our documentation on running evaluations.