API rate limits during backfill operations

Summary

Issue: High-volume operations like backfill jobs produce spikes of 401 authentication errors (invalid x-api-key) when using https://braintrustproxy.com/v1, with errors lasting 1-2 minutes before resolving automatically.

Cause: The current proxy gateway infrastructure has capacity constraints that trigger temporary rate limiting during high load, manifesting as 401 errors even though authentication is valid.

Resolution: Throttle concurrent requests during backfills or schedule them during off-peak hours until the new gateway is deployed.

Resolution Steps

If running backfill jobs

Step 1: Throttle request rate

Reduce concurrent requests to stay below gateway capacity limits.

# Add rate limiting to your backfill
import time
from braintrust import log

for batch in data_batches:
    log(batch)
    time.sleep(0.1)  # Throttle requests

Step 2: Schedule during off-peak hours

Run backfills when customer traffic is lowest to reduce contention.

Step 3: Implement retry logic

Handle transient 401 errors with exponential backoff.

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def log_with_retry(data):
    return log(data)

Otherwise, if experiencing errors during production traffic

Step 1: Monitor error duration

Errors should resolve within 1-2 minutes as load decreases.

Step 2: Implement client-side retry logic

Use exponential backoff to handle temporary failures without data loss.

Step 3: Contact support

If errors persist beyond 2 minutes or affect customer requests, contact Braintrust support with timestamps and error details.