Outcome Measurement.

Agent Economics (PRD 10) answers "how much did this cost?" but not "how much value did this deliver?" Operators can see token spend, model routing decisions, and per-agent cost breakdowns, but have no way to correlate that spend with actual outcomes. The system tracks inputs (tokens consumed, models invoked) without tracking outputs (tasks completed, quality achieved, time saved).

Without outcome measurement, optimization becomes guesswork. An operator might see that Opus costs 10x more than Haiku for a particular task type, but has no data on whether the Opus run produced a first-pass success while the Haiku run required three rework cycles. The cost comparison is meaningless without outcome context.

Outcome Measurement operates as a passive observation layer. It never modifies conductor state, never blocks agent dispatch, and never introduces latency into the orchestration pipeline. It reads events, computes metrics, and stores results in a dedicated Qdrant collection, completely decoupled from the orchestration path.

8 Outcome Metrics

Every metric is computed from observable data already present in conductor state, Gemini validations, and governance events. No additional instrumentation required from agents.

Core Components

Key architectural and design choices, with rationale for each decision.

• Passive observation, never active instrumentation. The Outcome Collector reads existing data streams (conductor state, Gemini verdicts, governance events) rather than requiring agents to emit outcome telemetry. This eliminates the risk of observation affecting agent behavior and requires zero changes to existing agents.
• Operator-defined baselines, not system-imposed targets. Performance baselines vary dramatically by project type, team size, and domain complexity. The system provides baseline suggestions from historical data but requires operator confirmation. Auto-adjustment happens quarterly with operator override.
• Separate Qdrant collection for outcome data. Outcome metrics are stored in a dedicated outcome_metrics collection rather than co-locating with memory or governance data. This enables independent retention policies, dedicated vector indexes for outcome similarity search, and clean separation of concerns.
• Extend existing dashboard rather than build standalone. The Memory Dashboard (PRD 6) already provides the infrastructure for real-time data visualization. Adding an outcome analytics tab leverages existing authentication, layout, and deployment rather than creating a parallel dashboard that operators must monitor separately.
• Exactly 8 metrics, no more. Every proposed metric was evaluated against three criteria: is it observable from existing data, is it actionable by an operator, and does it measure outcomes (not inputs)? Metrics like "tokens consumed" and "model selection accuracy" were excluded because they measure process, not outcome.
• 90-day granular retention with indefinite aggregates. Per-task granular data is essential for debugging and root cause analysis but grows linearly with workload. The 90-day window provides sufficient history for trend analysis. Aggregated data (per-agent-type, per-tier, per-project) is compact enough to retain indefinitely for long-term trending.

Outcome Measurement connects to 8 other PRDs in the ecosystem. Each integration is directional: Outcome Measurement reads from source systems and writes to presentation layers.

Copy and paste this prompt into Claude Code to begin implementing Outcome Measurement. It references the architecture, metrics, and integration points defined in this PRD.

Build the Outcome Measurement system (PRD 15) for the conductor plugin ecosystem.

## Context
Agent Economics (PRD 10) tracks cost but not value. We need a passive observation
layer that computes outcome metrics from existing conductor data without modifying
any source systems.

## Architecture
- Outcome Collector: passive observer reading conductor-state.json transitions,
  Gemini validation verdicts, governance audit events, and agent completion signals
- Metric Compute: transforms observations into 8 standardized metrics
- Outcome Store: dedicated Qdrant collection (outcome_metrics) with 90-day
  granular retention and indefinite aggregate retention
- Dashboard: extends Memory Dashboard (PRD 6) with outcome analytics tab
- Reports: automated weekly digest and project retrospective

## 8 Outcome Metrics to Implement
1. Task Completion Rate — verified-complete / total dispatched, by tier + agent
2. Time-to-Resolution — dispatch to verified-complete, excluding blocked time
3. First-Pass Success Rate — PASS on initial Gemini validation, no retries
4. Rework Frequency — retry cycles per task, by agent type + category
5. Quality Score Trend — 7-day rolling average of Gemini validation scores
6. Recovery Rate — initially-failed tasks that reach verified-complete
7. Context Efficiency — output tokens in deliverables / total tokens consumed
8. Cost per Successful Outcome — Agent Economics cost / success binary

## Value Attribution Framework
Map each outcome to: time savings, quality improvement, throughput gain,
risk avoidance, knowledge preservation. Quantify estimates per category.

## Key Constraints
- Read-only observation: NEVER modify conductor-state.json or source data
- Operator-defined baselines with quarterly auto-adjustment
- Separate Qdrant collection, not co-located with memory data
- 90-day granular + indefinite aggregate retention
- Dashboard updates within 30 seconds of metric computation

## Integration Points
- Reads: Conductor (PRD 2), Agent Economics (PRD 10), Governance (PRD 5),
  Code Assurance (PRD 8), Self-Healing (PRD 12)
- Writes: Memory Dashboard (PRD 6), Memory System (PRD 4)
- Subscribes: Event-Driven (PRD 13) event bus

## Requirements
14 requirements (REQ-OM-001 through REQ-OM-014). MUST-priority requirements
are non-negotiable: task completion rate, TTR, first-pass success, rework
frequency, quality trend, cost-per-outcome, read-only observation, and
retention policy.