Context-use evaluation (CUE) studies whether a subject can retrieve and correctly apply earlier information under a specified design and deployment. It does not measure a universal memory percentage or prove that a nominal context-window size is reliable for every task.
This advanced guide targets Sonny 0.12.7, source/build
192e590345c6a4bf1ef8f72b043b318084ea84d6. Installed provider-free checks exercised
planning, strict refusals, incomplete-evidence classification, hypothetical
archive verification/reanalysis and JSON/CSV/SVG rendering. No live CUE scheduler,
empirical model operating range or authorized measurement is qualified here.
Choose the claim and mode
| Mode | What it can establish | What remains separate |
|---|---|---|
research_only planning | A valid recorded design, assumptions, strata and proposed resource ceilings. | No model call, actual budget approval or performance result. |
| Existing data | Which facts survive in retained evidence and which reconstruction inputs are missing. | Missing exposure, truth, configuration or pairing cannot be invented. |
| Hypothetical archive | Whether retained illustrative artifacts verify and deterministic analysis can be recomputed. | Structural verification is not a measured target-model claim. |
| Testing preflight | Whether a complete testing plan has the required current configuration, isolation, access, resources and authorization. | A dry run is not execution or successful qualification. |
| Authorized testing | Measurements within a frozen design through an approved deployment runner. | Broader task, model, workload or operating-range claims need their own evidence. |
Method A is a planning/sensitivity decomposition: v*q + (1-v)*b, separating
sufficient exposure, application given sufficient evidence, and performance
without sufficient evidence. A retained-token fraction or a successful answer
is not evidence for v by itself. Method B uses empirical paired observations,
independent clusters, sufficient/absent/partial controls, fixed strata and
preregistered stopping and uncertainty rules. A later passing length cannot
bridge an earlier failure or an unmeasured cell.
Keep serialized input at the adapter boundary, provider-reported usage and local token estimates distinct. Include framing, system text, tools, retrieved material and summaries when defining the input budget. Pin tokenizer, template, model/deployment, scorer, truth, randomization, memory and isolation settings. See context and usage interpretation .
Inspect a research-only plan
Download the research plan
and its referenced
fixed pilot input suite
.
The plan was serialized with the pinned CUE contract and binds that suite’s
actual byte digest. Its dates, design and ceilings are documentation-fixture
inputs; execution_authorized is false. They are not current access or spending
approval for your deployment.
In a new private working directory, run:
sonny cue plan --plan ./research-plan.json --out ./research-report.json
The installed result was planned, executed: false. The output was owner-only;
a second request for the same output path was refused without overwriting it.
An unknown input field was also refused. Preserve the exact plan and returned
digest with the report.
The same research plan was refused by both real and dry-run cue testing
requests. Those responses listed unresolved configuration, budget and pilot
receipt requirements. Toggling one JSON field is not a supported way to create
an authorized testing plan. A testing deployment needs a current sealed design,
independent roster, target criteria, reviewed budget/access/isolation evidence
and the matching authorization/freeze receipts. Credentials do not belong in
those secret-free declarations.
Audit existing evidence before using it
The incomplete legacy example deliberately records missing tokenizer, request-snapshot, truth and pairing facts:
sonny cue existing-data --input ./incomplete-legacy.json
The installed result was inconclusive, class inconclusive_evidence, with exit
25. It did not run a model. That is an evidence classification, not a successful
measurement or permission to fill missing fields with defaults.
A separate check attempted to import the actual unedited Sonny trace generated
for the diagnostics guide
. Native trace import refused it:
CUE import trace contains more than one header. Normal Sonny tracing had
emitted multiple header updates. The fixture kept those records and the failed
import; it did not remove valid headers to make the command accept the file.
The original trace bytes remained unchanged. This import path is not qualified
as an end-to-end native-trace workflow in the tested release.
Verify and reproduce an illustrative archive
The hypothetical archive
is a separate
renderer/reanalysis fixture. Every artifact and analysis receipt is explicitly
hypothetical, with limitations stating that no target model was measured.
It contains invented illustrative outcomes, not imported observations, an
execution authorization or a reliable operating range.
After downloading it into the working directory:
sonny cue verify --archive ./hypothetical-archive.json
sonny cue reanalyze --archive ./hypothetical-archive.json
sonny cue report --archive ./hypothetical-archive.json --format json --out ./hypothetical-report.json
sonny cue report --archive ./hypothetical-archive.json --format csv --out ./hypothetical-report.csv
sonny cue report --archive ./hypothetical-archive.json --format svg --out ./hypothetical-report.svg
These commands executed successfully. Verification checked the retained
structure/artifacts; reanalysis matched the recorded deterministic result.
Reporting made all three formats without another model call, and the JSON
report had universal_capacity_claim: false. An unknown analysis ID returned
unsupported_analysis instead of silently choosing another method. Passing a
planning document as an archive was refused.
There is a provenance-label limit: although the archive’s artifacts and analysis
are hypothetical, the native report labeled its joint-session and first-failure
rows observed. The fixture retains that mixed-label result as a failed positive
check. Do not treat those rows as empirical model performance or use this
illustration for model selection, a reliable tested prefix or promotion.
Inspect the source receipt and limitations, not just a chart label or a zero
exit code. The output was not relabeled after rendering to hide the mismatch.
Keep execution and promotion reviewable
Before a real study, confirm the approved runner and current preflight with the deployment owner. A valid research plan, an inconclusive import, a hypothetical reanalysis or a rendered figure does not install that runner. Keep raw inputs, scorers, configuration, pairing and costs so another reviewer can reconstruct the claim. Preserve missing cells and denominators; do not replace unknowns with zero or report a model-wide capacity from one task/position.
For broader validation, use Recipe evaluation and optimization with their own grading and promotion gates. Training/export has separate verification and consent rules. Related release capability tracking is not evidence that an unqualified CUE workflow has shipped successfully.