add

2026-07-03 00:38:05 +09:00
parent 87eacd39b2
commit 86cad348b4
149 changed files with 24450 additions and 105 deletions
@@ -0,0 +1,31 @@
+---
+title: E2E Coverage Metrics
+created: 2026-06-30
+updated: 2026-07-01
+type: concept
+tags: [quality, reliability, evaluation, workflow]
+sources: [raw/articles/knowledgework-e2e-coverage-metrics-2026.md, raw/articles/github-code-coverage-merge-protection-2026.md, raw/articles/realworld-framework-comparison-spec-2026.md]
+confidence: medium
+---
+
+# E2E Coverage Metrics
+
+E2E coverage metrics are a way to measure whether end-to-end tests actually touch the product surfaces they are supposed to protect. KnowledgeWork's article argues that manually maintained "test cases written / test cases needed" lists drift as features change, so the denominator should be derived from implementation artifacts where possible.
+
+The concrete pattern is to compute **page coverage** from all known product pages versus pages visited during Playwright runs, and **RPC/API coverage** from all service/method definitions versus RPCs observed in test traffic. In their setup, all pages are extracted from Next.js routes, all RPCs from `.proto` definitions, and the test-side observations come from Playwright trace network entries such as page-view and API requests.^[raw/articles/knowledgework-e2e-coverage-metrics-2026.md]
+
+GitHub Code Quality's merge-protection preview shows the same idea being productized as a repository gate: branch rulesets can block pull requests when coverage falls below a minimum percentage, drops too far from the default branch, or both. Its evaluate mode is important operationally because teams can observe the effect of a quality threshold before turning it into a hard merge blocker.^[raw/articles/github-code-coverage-merge-protection-2026.md]
+
+RealWorld adds a benchmark-design angle: many frontend and backend implementations share the same Medium-like app, API specification, backend spec tests, frontend E2E test suite, CSS theme, and hosted demo API. That makes it useful not only as framework learning material, but as a stable surface for comparing generated code, agent-built app variants, and cross-framework regression behavior under one contract. For [[ai-evaluation-infrastructure]], the important part is the common spec/test harness, not the specific app clone.^[raw/articles/realworld-framework-comparison-spec-2026.md]
+
+This is useful for [[ci-cd-runtime-security]] and [[ai-evaluation-infrastructure]] because it treats test execution as observable runtime evidence, not just a green/red result. It also fits [[loop-engineering]]: the loop should store raw observations, compute metrics later, notify people in the place they already work, and keep enough history to change aggregation methods without rerunning old tests.
+
+## Caveat
+
+Implementation coverage is a necessary-condition signal, not a sufficient proof of product quality. Visiting every page or calling every RPC does not guarantee that important user scenarios are asserted. The stronger pattern is to combine implementation-derived coverage with deterministic regression tests for known critical business paths.
+
+## Open Questions
+
+- Which surfaces should define the denominator for non-Next.js or non-RPC products: routes, OpenAPI endpoints, event names, domain actions, or user journeys?
+- When should low E2E coverage block a release, and when should it only produce a review item?
+- How can AI-generated test additions avoid optimizing for easy-to-cover surfaces while missing high-risk workflows?