Experiment Fetch Limited to 1000 Events - Export All Experiment Events Using BTQL and Pagination

Summary

Goal: Export complete experiment data beyond the 1000 event limit using BTQL with cursor pagination.

Features: BTQL endpoint, cursor-based pagination via x-bt-cursor header, native query format.

Configuration Steps

Step 1: Understand the limitation

The /v1/experiment/{id}/fetch endpoint has a hard limit of 1000 events per request and does not support cursor pagination.

Step 2: Use BTQL endpoint for full export

The /btql endpoint supports cursor-based pagination to retrieve all experiment events.

import requests
import json

def fetch_all_experiment_events(experiment_id, api_key):
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Content-Type": "application/json"
    }
    all_events = []
    cursor = None

    while True:
        query = {
            "query": {
                "from": {
                    "op": "function",
                    "name": {"op": "ident", "name": ["experiment"]},
                    "args": [{"op": "literal", "value": experiment_id}]
                },
                "select": [{"op": "star"}],
                "limit": 1000
            },
            "fmt": "jsonl"
        }
        
        if cursor:
            query["query"]["cursor"] = cursor
        
        resp = requests.post(
            "https://api.braintrust.dev/btql",
            headers=headers,
            json=query
        )
        
        # Extract cursor from response header
        cursor = resp.headers.get("x-bt-cursor")
        
        # Parse JSONL response
        rows = [
            json.loads(line)
            for line in resp.text.strip().split("\n")
            if line.strip()
        ]
        
        all_events.extend(rows)
        
        # Stop when no cursor returned
        if not cursor:
            break
    
    return all_events

# Usage
events = fetch_all_experiment_events(
    "your-experiment-id",
    "your-api-key"
)
# Print example, handle event data however you need from here.
print(f"Total events: {len(events)}")

Step 3: Extract pagination cursor from headers

The pagination cursor is returned in the x-bt-cursor response header, not in the response body.

Step 4: Pass cursor in subsequent requests

Add the cursor to the query.cursor field in subsequent requests until x-bt-cursor header is null.

Fetch vs BTQL Endpoint Differences