LLM as Judge scorer limitations with custom scoring

Last updated: March 6, 2026

Summary

LLM as Judge scorers in the Braintrust UI are limited to predefined Choice scores and cannot automatically parse custom numeric values, percentages, or complex scoring logic directly from LLM responses. When you need dynamic scoring based on calculations or prompt-defined scales, implement a Python or TypeScript scorer that calls an LLM and returns the computed score.

Applicable To

Plans: Any

Deployments: Any

Limitation Details

The built-in LLM as Judge scorer requires you to configure specific Choice scores (like "Good", "Bad", "Excellent") in the UI. The LLM must return exactly one of these predefined labels, which Braintrust then maps to a numeric score. This approach doesn't work when you need:

  • Percentage-based scoring (e.g., "5 out of 7 items correct = 71%")

  • Custom numeric scales defined in the prompt

  • Dynamic calculations based on response content

  • Complex scoring logic that returns structured data

Workaround: Custom Scorers with LLM Calls

For flexible scoring requirements, create a custom scorer that calls an LLM and processes the response programmatically. This approach gives you full control over score calculation and prompt design.

Implementation Pattern

function custom_scorer(output, expected):
    // Option 1: Direct calculation
    calculate score based on output and expected values
    return score between 0-1 with metadata
    
    // Option 2: LLM-based scoring
    construct custom prompt with scoring instructions
    call LLM with prompt and context
    parse numeric score from LLM response
    handle parsing errors with fallback score
    return score between 0-1 with metadata

Example Use Cases

  • Percentage accuracy: Calculate items_found / total_expected directly

  • Custom scale judgment: Prompt LLM to score on 0-10 scale, then normalize to 0-1

  • Multi-criteria scoring: Have LLM evaluate multiple aspects and return weighted average

  • Structured evaluation: Parse JSON responses with scores and reasoning

When to Use Each Approach

Use the UI LLM as Judge scorer for simple categorical evaluations with fixed labels (Good/Fair/Poor). Use custom scorers with LLM calls when you need percentage calculations, dynamic scoring scales, or any logic that requires parsing numeric values from LLM responses.

Internal Links

  • Ticket: LLM as Judge: Score Definition via Prompt or UI Choice Scores