Evaluate a Recipe change

Evaluate a change against a task contract you can inspect. This guide targets Sonny 0.12.7, the research Recipe and an approved provider/model connection. The controlled qualification exercised the actual eval command and two lexical cases; it does not certify a live model’s reasoning.

Decide what a passing result means Share link

The research Recipe contains pack.pack_eval_suite.cases for two inputs:

• Supplied notes: require 75%, 70%, source labels and missing satisfaction evidence; reject a claim that satisfaction improved.
• Missing note B: require an explicit missing-B statement and reject an invented 70% B rate.

Those lexical checks can catch specific regressions. They do not establish that every factual claim is supported, that a citation resolves, or that a tool action was performed. Independently review arithmetic, conflicting dates and the proposed next step against the actual inputs.

Select the evaluation connection explicitly Share link

The installed eval command refused without an explicit provider, even though the Recipe’s normal run could resolve category primary. The passing eval used an explicit provider/model and that provider’s configured authentication/endpoint. Treat this as a separate connection decision; it can send evaluation inputs to a different service from the task’s category route.

The command form is:

sonny recipe eval evidence-brief --file ./brief.yaml --provider <approved-provider-id> --model <approved-model-id> --format jsonl --exit-on-fail

Replace the angle-bracket identifiers with your approved connection. Configure its credentials and endpoint through the provider reference and credential guide before running it. A catalog inspection or stored key is not an authentication test. The command contacts the provider, consumes usage and can write evaluation history when configured.

Inspect every JSONL case row. The controlled fixture returned pass for both case IDs. A second suite with an intentionally impossible expected pattern returned fail and a nonzero exit with --exit-on-fail, confirming the failure gate rather than relying on an empty or successful-looking command output.

Keep the two suite schemas separate Share link

pack_eval_suite and an external bare cases: suite use input, expected_pattern, forbidden_pattern, any_of, min_matches, optional judge, tools_allowed and timeout_seconds. Patterns use RE2; review their actual matching behavior. A bounded timeout is useful for each case.

The separate eval_suites pipeline uses user_input and its own grader/case shape. Copying keys between these schemas can produce an ineffective check. Use the release schema and actual validator diagnostics.

Test tools and permissions through a real run Share link

The lexical eval path does not prove tool execution. The research qualification also ran a tool-enabled headless case whose controlled provider requested write_file. Sonny returned denied_exposure, the target file remained absent, and the conversation later completed. That is evidence of this denial, not a terminal task failure or a TUI consent-dialog qualification.

For your own Recipe, test permitted operations, denied operations, missing inputs, provider failures and conflicting evidence in the client you will use. Record what actually executed. A final answer saying “I wrote the file” is insufficient.

Review model judgments and proposed optimizations Share link

A model judge is advisory unless the release’s evidence policy has independent support for its semantic claim. A JSON outcome’s local status or a higher lexical score does not certify a production task or establish authorization.

recipe optimize can propose prompt changes and run measurement/gates. Review its candidate, holdout inputs, cost, preserved policy and output artifacts before adoption. Optimizer export, variant approval, source publication and runtime authorization are separate decisions. Keep a reviewed baseline and rollback files; rerun representative cases after changing a prompt, reference, model, adapter or policy.

Continue with composition , model variants and trust and the evaluation/optimization command reference .

Work through a baseline and candidate Share link

The pilot-brief baseline , candidate , training suite , demonstration holdout and human rubric form a fixed non-coding example. Save and inspect all five files in one private directory.

The complete-notes input supplies 30/40 and 35/50 with source labels. The missing-B input supplies only A. Both explicitly lack survey evidence. The candidate makes missing-note handling and the separation of facts from interpretation explicit. The holdout changes the supplied notes, labels and rates; it has two cases and is deliberately too small for the optimizer’s default promotion gate. Do not duplicate cases merely to reach that gate’s minimum.

With your approved provider/model configured, use this command form for each Recipe, changing only the file between baseline and candidate:

sonny recipe eval pilot-brief-eval --file ./baseline.yaml --suite-file ./train.suite.yaml --provider <approved-provider-id> --model <approved-model-id> --format jsonl --exit-on-fail
sonny recipe eval pilot-brief-eval --file ./candidate.yaml --suite-file ./train.suite.yaml --provider <approved-provider-id> --model <approved-model-id> --format jsonl --exit-on-fail
sonny recipe eval pilot-brief-eval --file ./candidate.yaml --suite-file ./holdout.suite.yaml --provider <approved-provider-id> --model <approved-model-id> --format jsonl --exit-on-fail

These exact command paths ran against a controlled provider. Its deliberately selected baseline response passed the complete-notes case but invented B’s rate in the missing-B case; the baseline command exited nonzero. The selected candidate responses passed both training and both holdout cases. Every JSONL reply and case status was inspected. This qualifies the command, case and failure-gate wiring; it does not measure improvement on a live model, unseen tasks or the full human rubric.

For a real comparison, retain the same inputs, model revision, adapter, suite, configuration and grading policy. Review the replies against the rubric: recompute rates, check source support, preserve missing evidence and distinguish interpretation. A lexical pass cannot establish those broader properties by itself. If you add a model judge, preserve its advisory/local-grade boundary and measure disagreement rather than treating its confidence as independent verification.

Record baseline and candidate source hashes, case IDs, replies, usage and failure reasons with the comparison. Keep held-out data outside the optimizer prompt. See optimization for call ceilings, retained candidates, source-file behavior and promotion/refusal gates, and diagnostics for trace versus evaluation identifiers.