Configure scorer pass threshold via API (workaround for Ruby SDK)

Last updated: March 6, 2026

Summary

Goal: Configure pass/fail thresholds for scorers in Ruby using the Braintrust API.

Features: Function endpoint, scorer metadata, pass threshold configuration.

Applicable To

Plans: Any

Deployments: Any

Configuration Steps

Step 1: Create the scorer registration function

POST to the /v1/function endpoint with scorer metadata.

def register_pass_threshold(project_id, scorer_name, pass_threshold)
  return unless project_id

  api_state = Braintrust::State.global
  return unless api_state&.logged_in

  payload = {
    project_id: project_id,
    name: scorer_name,
    slug: scorer_name.to_s.downcase.gsub(/[^a-z0-9]+/, "-"),
    function_type: "scorer",
    metadata: { "__pass_threshold" => pass_threshold }
  }

  uri = URI("#{api_state.api_url}/v1/function")
  request = Net::HTTP::Post.new(uri)
  request["Authorization"] = "Bearer #{api_state.api_key}"
  request["Content-Type"] = "application/json"
  request.body = JSON.dump(payload)

  response = Braintrust::Internal::Http.with_redirects(uri, request)
  if response.code.to_i < 400
    puts "[Scorer] Registered #{scorer_name} with pass threshold: #{pass_threshold}"
  end
rescue StandardError => e
  puts "[Warning] Failed to register pass threshold for #{scorer_name}: #{e.message}"
end

Step 2: Call the function with your scorer details

Pass the project ID, scorer name, and desired threshold value.

register_pass_threshold("your_project_id", "accuracy", 0.8)

Step 3: Verify in the Braintrust UI

The pass threshold will appear in the eval results, marking scores as pass/fail based on the threshold.

Notes

  • The __pass_threshold feature is not directly supported in the Ruby SDK yet

  • This workaround uses the same API endpoint that Python and TypeScript SDKs use internally

  • The slug field must be lowercase alphanumeric with hyphens only

  • Registration errors won't fail your eval; they're logged as warnings

Internal Links