Troubleshooting Data Retention Setup in Hybrid Deployments

Last updated: January 26, 2026

This issue is applicable to customers matching these conditions

Plans: Enterprise

Deployments: Hybrid

Use case: Data Retention Policies

Issue

Users may notice the following indicators that retention is not functioning as expected:

  • Searching logs for sourcetype=braintrust* retention returns no results

  • Retention policies appear configured but old logs are not deleted.

  • Log messages like:

    Time-based retention will not run, no object IDs provided

  • No evidence of the retention cron job running in logs.

Cause

  • Missing or Invalid Data Plane Service Token

  • Missing Environment Variables in API or Brainstore Hosts

  • Missing Encryption Keys

Resolution(s)

Before attempting any of the below steps ensure you are on the latest version of the Data Plane, you can confirm the latest released version here:

https://www.braintrust.dev/docs/data-plane-changelog

Missing or Invalid Data Plane Service Token:

Step 1: Navigate to your service token settings

Step 2: Click Create next to the data plane manager token. This will:

  • Provision a data plane manager (read-only service account).

  • Generate the bt_data_plane_service_token.

  • Store it within your data plane for use by retention workers.

Step 3: Once completed, the warning message should disappear.

Missing Environment Variables in API or Brainstore Hosts:

Step 1: Ensure the following environment variables are set for the API Host:

Variable

Description

PRIMARY_ORG_NAME

Must be set to your primary org name. Required for multi-org deployments.

SERVICE_TOKEN_SECRET_KEY

Used to encrypt and verify service tokens. Must be a secure, randomly generated string.

FUNCTION_SECRET_KEY

Used as a fallback if

SERVICE_TOKEN_SECRET_KEY

is not defined.

BRAINSTORE_ENABLED

Must be set to true to enable Brainstore features.

Step 2: Ensure the following environment variables are set for the Brainstore Host:

Variable

Description

BRAINSTORE_LICENSE_KEY

Enables Brainstore features.

BRAINSTORE_LOCKS_URI, BRAINSTORE_INDEX_URI, BRAINSTORE_REALTIME_WAL_URI

Configure S3 URIs for storage.

BRAINSTORE_TIME_BASED_RETENTION_OBJECT_ALL

Enables retention for all objects.

FUNCTION_SECRET_KEY

Must match the one set in the API host. Used for encryption/decryption.

Missing Encryption Keys

If your logs show errors like:

Time-based retention will not run, failed to resolve control plane context: Neither SERVICE_TOKEN_SECRET_KEY nor FUNCTION_SECRET_KEY is set

This indicates Brainstore cannot read or decrypt secure data.

To resolve add either or both keys to the Brainstore Host:

FUNCTION_SECRET_KEY=<secure-random-string>
SERVICE_TOKEN_SECRET_KEY=<secure-random-string>

Ensure these match the API Host configuration.

Verification Steps

  • Check for Cron Logs

    • Search logs for messages containing:

      • Automation Cron loop

  • Confirm Retention Worker Execution

    • Search logs for messages containing:

      • Launching time-based retention worker for all objects

  • Validate that data older than the retention window is no longer present