Securing remote eval endpoints: authentication and deployment best practices

Summary

Remote eval endpoints return "Hello, world!" on GET / as a public health check, which can cause confusion about security when deploying to staging environments exposed to the internet.

While protected endpoints (/list, /eval) require Braintrust API key authentication, the dev server runs HTTP only without built-in HTTPS, so production deployments need additional security layers including --dev-org-name restrictions and HTTPS proxies.

Configuration Steps

Step 1: Understand endpoint security

Remote eval endpoints have different authentication requirements:

• GET / - Public health check, returns "Hello, world!"

• GET /list - Lists evaluators, requires Braintrust API key

• POST /eval - Executes evaluations, requires API key and org verification

Step 2: Restrict to your organization

Use --dev-org-name to limit access to your Braintrust organization:

braintrust eval your-eval.ts --dev --dev-host 0.0.0.0 --dev-org-name "your-org-name"

Step 3: Add HTTPS layer

The dev server runs HTTP only. For staging/production, front it with a secure proxy that provides HTTPS (AWS ALB, Cloudflare Tunnel, nginx). Register the secure URL in your Braintrust project's Remote eval sources settings.

Step 4: Configure additional security

Add AWS security layers for production deployments:

• Security Groups: Restrict inbound traffic to trusted IP ranges

• AWS WAF: Add rate limiting and request filtering

• VPC Configuration: Deploy in private subnets with network isolation

Step 5: Control host binding

Use --dev-host localhost for local development. Only use --dev-host 0.0.0.0 when behind secure proxies in controlled environments.

Helpful Links

• Remote Evaluations Documentation

• Cookbook: Secure Tunnel Example