An optimizer proposes a change to a Recipe and compares its measured responses with a baseline. A higher local score is useful only when the cases and rubric represent the work you intend to do. Adoption is a separate review decision.

This guide targets Sonny 0.12.7 and the fixed pilot-brief evaluation . Installed tests used a controlled provider to exercise the actual commands and artifact paths. They do not establish that the proposed prompt improves a live model or satisfies a production rollout gate.

Prepare a comparison you can inspect

Keep a reviewed baseline, an explicit evaluation suite, a disjoint held-out suite, the model/provider revision and the expected behavior. Include difficult cases, missing inputs and unsupported claims. Use the human rubric alongside lexical checks. Keep a restorable copy or version-control revision of the baseline before allowing a source rewrite.

The demonstration holdout has only two cases. The installed optimizer refused it before any model call with the default --min-holdout-cases=25 requirement. That refusal is retained. A small synthetic example does not become an adequate independent holdout by lowering the threshold or copying its cases.

Inspect the plan and call ceiling

After selecting the approved connection, the command form is:

sonny recipe optimize pilot-brief-eval --file ./baseline.yaml --suite-file ./train.suite.yaml --holdout-suite-file ./holdout.suite.yaml --provider <approved-provider-id> --model <approved-model-id> --optimizer-model <approved-optimizer-model-id> --rounds 1 --max-calls 20 --out ./optimization --dry-run

The tested dry run made no model calls. Its plan named the mutable system-prompt component, edit budgets and worst-case call estimate. Read that estimate: a dry run returning zero did not mean the demonstration could execute with those limits. The reported worst case was 25 calls, while the requested ceiling was 20 and the holdout was also undersized.

A separate training-only preflight with --max-calls=1 refused the estimated 13-call worst case before any model request. The ceiling bounds the command’s call planning; it is not a universal currency budget or a promise that all providers account for tokens the same way. Set costs, rounds, retries, timeouts and access limits appropriate to the approved deployment. See providers and usage interpretation .

Inspect the local loop separately

A separate controlled test omitted the holdout only to exercise the local training loop. It made five provider requests: baseline evaluation, one optimizer proposal and candidate evaluation. The fixture’s lexical score moved from 0.5 to 1.0, and the native decision was accept_new_best. The provider responses deliberately made that branch observable; this is not a measured quality gain on a real service or an independently qualified promotion.

The artifact directory contained:

  • best.yaml — the retained best candidate for this local run.
  • history.jsonl and progress.jsonl — decisions and execution progress.
  • steps/step_1/proposal.json — the proposed component edit and hypothesis.
  • steps/step_1/candidate.yaml and eval.jsonl — the candidate and case results.
  • steps/step_1/decision.json — the compared baseline, candidate and decision.

Review the bytes of the proposal and resulting Recipe, not just the score. The installed source files remained unchanged in the controlled run. A candidate artifact being written does not activate it in another session or publish it to a source, variant or deployment.

Keep promotion and rollback explicit

The command’s source-rewrite switch is --apply; variant export is separate. The fixture requested --apply only on an owned copy with the undersized holdout. It was refused before model calls and the baseline hash was unchanged. There was no source rewrite, variant export or publication.

For an actual adoption, inspect the holdout verdict, scope, missing/infrastructure cases, changed policy and evidence freshness before approving it. Invalid or unmeasured rounds are different from measured rejections. Retain rejected and invalid proposals; do not describe an untried edit as ineffective or retry an external write because a progress record is missing.

Keep the previous approved Recipe and deployment settings available. After an approved change, rerun the relevant representative cases and runtime permission checks. Recover by restoring the reviewed baseline through its supported source workflow, then rerun those checks. A prompt artifact alone is not a rollback of external actions already performed by a session.

Model-judge trust flags change local grading, not the authority of runtime, release or training evidence. Use Recipe trust and variants for activation, training/export for data eligibility, and CUE for context-use studies rather than assuming every evaluation system uses the same identifiers or gates.

Check background analysis availability

See the analysis configuration reference for all generated patch fields, installed defaults and inherited policy limits.

Background analysis, recipe optimize and session optimize are separate interfaces. session optimize --help describes a paired full-session gate with candidate and scenario inputs; the Recipe demonstration above does not qualify that gate or the background analysis worker.

Inspect configuration without starting analysis:

sonny config validate --json
sonny config validate --config ./sonny.yaml --json

On the installed 0.12.7 Linux binary, the isolated default configuration reported analysis.enabled: false and startup state disabled. The release’s optimizer sample parsed successfully and exposed both background_optimizer and eval_candidates sinks, but its startup state was blocked, with reason runtime_unavailable: production runtime assembly was unavailable in this build. The process still returned zero because configuration validation succeeded. Inspect data.startup, runtime_assembly and provider_capabilities before concluding that a pipeline can run. An unsupported analysis version and an automatic promotion value were separately refused with configuration exit 3.

The pinned source defines a background optimizer grant with defaults of one concurrent worker and 500 cents when its protected background_optimizer block is absent. This default grant does not turn on an unconfigured pipeline. In a build with runtime support, activation requires enabled analysis, a pipeline with a background_optimizer sink and durable candidate storage. An explicit protected denial uses enabled: false without concurrency or budget fields; configuring the sink against that denial is a startup blocker. User/invocation patches cannot expand protected authority.

The source worker polls every 30 seconds, persists unverified candidates and requires independent grading, negative controls and promotion evidence. Missing grading capability leaves candidates unverified. Its monetary accounting is charged from actual cycle results: a final cycle can overshoot the remaining cap before later cycles stop. Treat that cap separately from the configured analysis session/request budgets and the Recipe command’s call ceiling.

The installed configuration report also exposed default snapshot retention of 24 hours, result retention of 168 hours and remote_export: false. These are resolved analysis settings; they do not establish physical deletion, trace retention or a live collector’s behavior. Validate the selected deployment’s settings and access rules before enabling capture or sharing findings.

No background worker, independent critic, canary or promotion was executed in this qualification. Use a build whose startup inspection supports the intended surface, bind its approved critic and provider, and inspect the retained verdicts before authorizing an operational rollout. Source support alone does not establish availability in the installed binary.