Agentic platforms plan multi-step tasks, call tools and take actions on users’ behalf. Most evaluation practice still measures capability: whether an agent can complete a task at least once on a static benchmark. For platforms that move money or change account state, that question is necessary but not sufficient. What matters in production is whether the agent completes the task correctly every time, fails safely when it can’t, and lets operators detect, diagnose and contain errors before they reach customers. We propose an evaluation framework that treats reliability as a first-class property of agentic systems. I draw on lessons from three mature domains: search serving, self-driving and fintech. Reliability is managed statistically through SLOs and error budgets, traditionally, and in autonomous driving, safety is argued through scenario coverage, operational design domains (ODDs) and incident-to-scenario regression loops. The framework has five parts: - Consistency metrics that measure success across repeated runs rather than a single attempt - Severity-weighted error taxonomies that separate recoverable errors from irreversible ones - Evals for the full stack, covering component, trajectory, end-to-end and adversarial behavior - Release gates tied to an autonomy ladder that unlocks higher-risk actions only as reliability evidence accumulates - A closed loop that turns production incidents, near-misses and disputes into regression cases