Eval Corpus

Local Eval Corpus Collection (internal/eval)

• Collection is automatic and default-on through eval.capture_provenance / eval.auto_capture / eval.max_cases / eval.diversified_size in config.yaml, never an environment variable: the daemon's launchd/systemd unit is re-rendered on install and update and preserves only proxy variables (internal/daemon/service.go proxyEnvKeys), so an env-gated corpus silently stops collecting after an update. The keys are global-only - Merge copies them straight from GlobalConfig, and an eval block in a repo's .no-mistakes.yaml is ignored.
• Provenance is unrecoverable: executor.go writes it with the review round or never. A round recorded with capture_provenance off can never be captured, so the rejection names the setting rather than the round's age.
• The trigger is RunManager.autoCaptureEvalCase, called last in the run goroutine after the outcome is already reported: it recovers its own panic (the enclosing recover would otherwise mark a finished run failed), bounds itself with evalAutoCaptureTimeout off the run context, serializes runs on evalCaptureMu (shared pool + registry), and logs rather than propagates. ErrNoCapturableReview separates "nothing to freeze" (DEBUG) from a real fault (WARN). Automatic and manual capture call the same eval.Capture. A merged PR also best-effort relabels already-captured cases via RunManager.relabelEvalRun (same mutex/timeout); eval relabel is the CLI path.
• The unit of truth is finding-level gold, not park/pass, and it is keyed on the round's recorded fix-vs-skip decision plus merge state, never on whether a later round still raises the finding (a fix and a ship both make it disappear): a user-selected Fix is true-positive gold (no merge required); an auto-fix selection on a merged run is true-positive gold even if a later round re-raised or rewrote it; a raised auto-fix/ask-user finding the human did NOT select, on a merged run, is false-positive gold - deliberately reversing the older "never auto-FP from a skip" stance, because in this operator's corpus an approved-and-shipped finding IS a false positive; a human-added finding is false-negative gold; skip/approve/abort without a merge and any round with no recorded decision stay unlabeled / pending; no-op findings are never labeled; unmatched candidate findings stay queued - never inferred as false positives - and a confirmed post-PR miss ingested via eval miss ingest is also false-negative gold (recorded-post-pr-miss). Owner: internal/eval (goldFromRound, hasRecordedDecision, IngestPostPRMiss, ScoreCandidate); user-facing language is docs/src/content/docs/reference/eval.md.
• diversified is gold-only and pinned (empty gold -> empty set + eval sets warning, never unlabeled fill). Those pins are the held-out official set; leftover labeled cases are tune. ListCases trims pins to the live eval.diversified_size cap (at most one per stratum when reconciling to 0 or a lower cap); RefreshDiversified is only for an explicit rebuild. Never fit matcher thresholds or review prompts on diversified. Report F1 as the headline metric only when false-positive gold exists; otherwise recall + precision bounds. RelabelRun recomputes derived merge labels and drops the obsolete ones. Matcher assignment is ONE globally optimal bipartite matching over all gold and candidate findings, weighted so an exact match outweighs any number of fuzzy ones; per-strength-tier greedy assignment understated recall and must not come back. Regressions: TestListCasesDiversified_*, TestGoldFromRoundLabelsByRecordedDecision, TestCaptureWritesAutoFixMergedAsTruePositive, TestCaptureWritesShippedUnfixedAsFalsePositive, TestCaptureWritesShippedUnfixedEvenWhenTheFinalRoundNoLongerRaisesIt, TestCaptureLabelsSelectedAutoFixAsTruePositiveEvenWhenLaterRoundReRaisesIt, TestRelabelReplacesShippedUnfixedWhenTheRoundLaterRecordsAFixDecision, TestMergeGoldClearsStoredShippedUnfixedWhenRecomputedUnlabeled, TestRelabelClearsStoredShippedUnfixedFPWhenRecomputedUnlabeled, TestScoreCandidateDoesNotLetFuzzyEarlierGoldStealExactLaterMatch, TestScoreCandidateRecoversMatchTheTieredMatcherLost, TestMaxWeightAssignmentMatchesBruteForceOptimum, TestEvaluationSummaryWithholdsHeadlineF1WithoutFalsePositiveGold, TestCaptureDoesNotLabelSkipOrApproveAsPass, TestCaptureWritesFalseNegativeGoldForUserAddedFinding, TestCaptureSkipsIncompleteReviewRoundAndKeepsCompletedSibling, TestIngestPostPRMissWritesFalseNegativeGoldOnGreenReview, TestCaptureAndReport*, CLI TestEvalCaptureAndSetsSpeakInFindingGoldTerms, TestEvalMissIngestLabelsFalseNegativeGold.
• A case stores no Git bundle. Bundles were a full history copy per review pass (~8 MB each here) and cannot be trimmed, because a bundle built with negative refs records prerequisites an empty restore gate lacks. Cases of one repository instead share <NM_HOME>/eval/pools/<fingerprint>.git, pinned by refs/no-mistakes/eval/<caseID>/{head,source-head,base,trusted-config}; the marginal case costs ~8 KB. Store.Prune applies max_cases oldest-first but protects active replay reservations and cases with recorded evaluations, so the cap is a retention target rather than a hard bound.
• Capture stays read-only against the gate, so objects reach the pool through a throwaway bare clone plus a refspec fetch - never a bare-object-id fetch, whose want policy is off by default and version-dependent.
• Every eval subcommand is idempotent and tested so (internal/eval/idempotency_test.go, CLI TestEvalCaptureSetsReportAndRelabelAreIdempotentAtTheCLI): capture/relabel converge in place, sets reads self-stabilize their pins, and replay is additive-by-cohort but never rewrites case labels or manifests - queued unmatched-finding counts derive from the evaluations table (Store.pendingFindingCounts), never from a stored counter. The eval sets and eval run dashboards render in internal/cli/eval_render.go, sharing the stats box idioms (renderTitledBox); the diversified headline's instant self-score is SelfScoreRecordedReviews scoring each case's recorded review against its own gold.
• Regressions: TestCaptureDoesNotCopyRepositoryHistoryPerCase, TestPruneBoundsTheCorpusOldestFirstAndKeepsEvaluatedCases, TestDropCaseObjectsReleasesOnlyItsOwnPins, TestAutoCaptureEvalCase* (internal/daemon), TestEvalDefaultsCollectWithoutSetup, TestRepoConfigCannotChangeEvalCollection, e2e TestEvalAutoCaptureJourney.