Eval timeout errors for initial scorers

Summary

Issue: Eval scorers timeout during execution with errors indicating function execution exceeded time limits. This occurs when scorers take longer than the default 30-second timeout in self-hosted deployments.

Cause: The CODE_FUNCTION_EXECUTION_TIMEOUT_S environment variable defaults to 30 seconds, which is insufficient for complex scorers.

Resolution: Set CODE_FUNCTION_EXECUTION_TIMEOUT_S on both APIHandler and AIProxy services in your deployment configuration.

Applicable To

Plans: Enterprise

Deployments: Hybrid

Use case: Self-hosted evals with scorers requiring longer execution time

Resolution Steps

For Terraform AWS deployments

Step 1: Update service_extra_env_vars

Add CODE_FUNCTION_EXECUTION_TIMEOUT_S to both APIHandler and AIProxy in your Terraform configuration.

service_extra_env_vars = {
  APIHandler = {
    CODE_FUNCTION_EXECUTION_TIMEOUT_S = "120"
  }
  AIProxy = {
    CODE_FUNCTION_EXECUTION_TIMEOUT_S = "120"
  }
}

Step 2: Apply configuration

Run terraform apply to redeploy the data plane with updated timeout settings.

For other deployment methods

Step 1: Set environment variable

Add CODE_FUNCTION_EXECUTION_TIMEOUT_S as an environment variable to both APIHandler and AIProxy services with matching values (maximum 900 seconds).

Step 2: Restart services

Restart both services for the configuration to take effect.

Internal Links

• Braintrust limits documentation

• Terraform AWS module