Context Curation: Preliminary REALM Tests

• Run 1: gpt-4o-mini - February 3, 2026 - 04:33:52 UTC
• Run 2: gpt-5-nano - February 3, 2026 - 04:40:19 UTC
• Scope: read loop only (navigate + accumulate context)
• Queries: Authentication and rate limiting
• Max iterations: 5

REALM as Context Curation

This post treats REALM as a context curation system. The model does not receive the full document up front. Instead, it progressively pulls in only the sections that appear relevant to the current question. The mechanism is intentionally simple: a table of contents, a constrained list of candidate next sections, and a loop that appends selected sections into the working context.

This matters because common alternatives do not scale cleanly. Full-text prompting scales linearly with content size. Embedding-based retrieval can flatten or ignore the hierarchy that documentation already provides. Manual curation does not scale when both content volume and query volume increase. REALM attempts to preserve document structure while keeping per-call context bounded.

This first log focuses on navigation accuracy and context growth. The next phase is to expand toward the full loop and its operational pitfalls, including stop conditions, confidence estimation, guardrails, and richer tool and document graphs.

What I Tested

I compared two approaches across five document sizes (9 to 260 sections) and two queries, with a maximum of 5 iterations.

• Iterative REALM: select a section, read it, and repeat until the relevant section is located
• Full-text: attempt a one-shot selection from the entire document

Queries Used

• How do I authenticate API requests?
• What are the rate limits?

Results: multi-size-experiment-2026-02-03.json

Data sources:

• small-api-docs.md
• large-api-docs.md
• xlarge-api-docs.md
• xxlarge-api-docs.md
• xxxlarge-api-docs.md

Basic Loop

available_sections = toc(document)
selected_sections = []

for iteration in 1..5:
  prompt = build_prompt(query, available_sections, selected_sections)
  next_section = llm_select(prompt)
  selected_sections.append(next_section)
  available_sections.remove(next_section)

  if has_answer(selected_sections, query):
    # early-stop candidate
    record(iteration, selected_sections)

The focus of this post is the selection step. Can the model consistently choose a useful next section from a constrained menu, and continue doing so as the document grows?

Result Snapshot

Full-Text Is a Fragile Interface

Full-text prompting appears simpler, but its failure mode is difficult to manage operationally. The model must scan thousands of tokens, select one section, and return it in a strict format. In practice, it can select an irrelevant section, return an invalid identifier, or produce a plausible answer that is not grounded in the document. Because the call is single-shot, there is no built-in recovery step.

Early Stopping Is the Biggest Multiplier

The clearest signal is not the 5-iteration total. It is the iteration at which the loop first reaches the correct section. If the loop stops as soon as it has enough evidence to answer, token usage can drop substantially. To keep the comparison aligned with the earlier charts, the view below compares one-shot full-text against stop-at-first-correct, averaged across both queries.

Per-Iteration Growth Stays Manageable

Context grows each iteration because selected sections accumulate. In a representative medium run, growth remained controlled while output tokens stayed small, since the model was not producing long explanations during selection.

This is promising for smaller local models. Keeping each call bounded to a much smaller context window can make the loop more practical on constrained hardware, especially when stop conditions keep the number of iterations low.

Per-Call Context Windows Stay Small

Takeaways from This First Pass

• The baseline loop is viable. Iterative section selection remains stable as content grows.
• For gpt-4o-mini, iterative becomes token-competitive around 30 KB documents and remains competitive beyond that.
• Full-text selection is operationally fragile because it provides no recovery path when it selects the wrong section.
• Early stopping should be a first-class control. In these runs, it typically reduced a 5-iteration budget to about 2 iterations.
• Next step: evaluate a local small language model on the same loop, then extend toward the full REALM design with explicit stop criteria, confidence signals, and validation.