Online Scorers Missing Production Traces Due to Time Filters

Last updated: January 27, 2026

Summary

Issue: Online scorers configured with time-based filters like created > now() - 1 day fail to score production traces, showing missing scores in the UI.

Cause: Time-based filters in online scorer queries can interfere with trace discovery due to timing mismatches between trace creation and scorer evaluation.

Resolution: Remove time-based filters from online scorer SQL queries, as scorers automatically apply to new traces without needing time conditions.

Applicable To

Plans: Free, Pro, Enterprise

Deployments: Braintrust Hosted, Hybrid

Use case: Online scorers with time-based SQL filters in production environments

Resolution Steps

Remove Time-Based Filters

Step 1: Edit scorer rule

Navigate to the Online Scoring modal and remove any created > now() - X conditions from your SQL filter.

-- Before
WHERE span_attributes.name IN ('agent_run', 'turn_start') 
AND created > now() - 1 day

-- After
WHERE span_attributes.name IN ('agent_run', 'turn_start')

Step 2: Set sampling to 100%

Temporarily increase sampling rate to 100% to verify the scorer works on all matching traces.

Step 3: Test the rule

Click Test rule to confirm your SQL filter matches recent spans.

Step 4: Verify scoring

Create new traces and confirm scores appear. Some lag (seconds to minutes) is normal.

If No Results Appear After Testing

Step 1: Simplify filter

Temporarily remove span_attributes.name IN (...) to check if any spans match.

Step 2: Verify scorer output

Run the scorer against a recent trace in the scorer page to confirm it returns a numeric value.

Step 3: Check LLM-judge scores

For LLM-judge scorers, ensure each choice maps to a unique numeric score in [0,1] range.

# Incorrect - both choices set to 0
clear: 0
showed_frustration: 0

# Correct
clear: 0
showed_frustration: 1

Additional Context

Online scorers automatically apply to new traces as they are created. Time-based filters are unnecessary and can cause the scorer to miss traces due to timing issues between trace creation and query evaluation. Some delay in score appearance is expected behavior.

Links