Controlling Concurrency to Prevent Resource Exhaustion During Evaluations

Last updated: January 29, 2026

Summary

Issue: Evaluations fail with various errors or produce incomplete results when running with unlimited concurrency.

Cause: By default, maxConcurrency is undefined, allowing unlimited concurrent tasks which can exhaust system resources or hit external service limits.

Resolution: Set maxConcurrency to 10-20 to limit concurrent execution and prevent resource exhaustion.

Applicable To

Plans: Free, Pro, Enterprise

Deployments: Braintrust Hosted, Hybrid

Use case: Running evaluations that may hit resource limits or external service constraints

Steps

For Python SDK

Step 1: Add maxConcurrency parameter to Eval

Set max_concurrency between 10-20 when calling Eval().

from braintrust import Eval

Eval(
    "my-project",
    data=my_dataset,
    task=my_task,
    scores=[my_scorer],
    max_concurrency=10  # Limit concurrent execution
)

Step 2: Verify issues are resolved

Run the evaluation and confirm all tasks complete without resource or service limit errors.

For TypeScript SDK

Step 1: Add maxConcurrency parameter to Eval

Set maxConcurrency between 10-20 when calling Eval().

import { Eval } from "braintrust";

await Eval("my-project", {
  data: myDataset,
  task: myTask,
  scores: [myScorer],
  maxConcurrency: 10  // Limit concurrent execution
});

Step 2: Verify issues are resolved

Run the evaluation and confirm all tasks complete without resource or service limit errors.

Additional Information

Recommended maxConcurrency values

  • Start with 10 for most use cases

  • Increase to 20 if performance is adequate and no errors occur

  • Reduce to 5 if still experiencing resource issues

When concurrency limits are needed

  • API rate limits: OpenAI/Anthropic APIs have RPM limits

  • Memory constraints: Large documents/images per task

  • Database limits: Connection pool exhaustion

  • Cost control: Spread API usage over time

  • Self-hosted infrastructure: CPU/memory resource limits

  • File descriptor limits: System resource exhaustion

Common symptoms requiring concurrency control

  • Evaluations with incomplete or failed tasks

  • System resource exhaustion errors

  • Timeout errors during task execution

  • API rate limit (429) errors

  • Out of memory errors with large datasets

Reference