Methodology Capability: Q Audits Must Execute the Consumer's Read Path
Date: May 15, 2026 Origin: Q's Wave 4 audit of the Compass
/data-modelpage (docs/quality/audits/2026-05-14-compass-data-model-page.md) returned PASS-WITH-ISSUES with 0 sev-1 findings. Mirror's authenticated walk in the same wave caught a sev-1 functional defect Q's audit did not surface: the page rendered no future-state plan content because the Library substrate'sportal_slug AND listing_content_type IN [...]filter chain returned zero records at runtime. Showcase's Wave 5 diagnosis confirmedportal_slugwasnullon the FSDM and CSB Listings at the moment of the live read, even though Q's Wave 4 GET appears to have inspected those same records. Impact: A new audit SOP for any Q audit covering code that consumes HubSpot, Sanity, or Content Vault records via a substrate or filter chain. Static record-existence verification is reclassified as INSUFFICIENT for PASS verdict on read-path audits. The audit must execute the consumer's actual filter chain against live data and confirm the consumer's expected records appear in the response. Q's Wave 4 audit verdict is proposed for retroactive reclassification with a sev-2 methodology finding.
What Was Built
A methodology change to Q's audit standard: when auditing code that reads records via any substrate or filter chain, Q must execute the consumer's actual read path against the live data source and verify the consumer's expected records appear in the response. Static verification of "records exist with the right properties + the right associations" is now classified as INSUFFICIENT to support a PASS verdict on read-path audits.
This change closes the gap that allowed Wave 4 to issue a PASS-WITH-ISSUES verdict (0 sev-1, 0 sev-2, 4 sev-3) on a build that, when Mirror walked it under authentication in the same wave, displayed an empty-state card where 4 fully-authored prose sections were supposed to render. Q's audit checked all the pieces; what Q's audit did not check was the chain that assembles the pieces into a result.
What Happened
The Compass /data-model page is rendered by data-model.astro, which calls data-model-page.ts. That module imports createHubSpotLibraryDataSource from ./hubspot-library and uses it as the substrate for reading both the FSDM (Future-State Data Model) Listing and the CSB (Current-State Brief) Listing for the active client. The substrate filters records by portal_slug (resolved from Astro.params.clientSlug) and by listing_content_type IN [...] before returning anything to the consumer.
| Wave | Agent | Action | Verdict |
|---|---|---|---|
| 4 | Q | Static record audit + property-index integrity + code audit (read-only file inspection in V2/V1; live HubSpot GET against records 556337927839 and 556346234391; cross-reference of property-index against page code) | PASS-WITH-ISSUES, 0 sev-1, 0 sev-2, 4 sev-3 |
| 4 | Mirror | Authenticated walk of compass.valuefirstteam.com/abs-company/data-model |
Sev-1 functional defect: empty future-state card where two fully-authored prose section groups were expected |
| 5 | Showcase | Live GET diagnosis of FSDM (556337927839) and CSB (556346234391) | portal_slug = null on both Listings at time of read; Library substrate filter chain returned zero records; runtime fall-through to empty state was correct given the data |
| 5 | Resolution | Stamp portal_slug = "abs-company" on both Listings |
Filter chain returns expected records; future-state content renders |
Q's Wave 4 audit (F-12 in the audit report) reported the live GET against both records "confirms successful creates with intended payload" and noted, in F-12's evidence, that the associated Company 18819022093 carried portal_slug="abs-company". The audit did not assert that the Listings themselves carried portal_slug. The Library substrate filters Listings by portal_slug on the Listing record, not by traversing the Listing's Company association at read time. The audit verified the wrong slug.
Methodology Gap Identified
Static record verification establishes that records exist, that they carry the intended properties, and that they hold the intended associations. None of those three checks executes the chain the consumer applies after fetching. When the consumer applies a non-trivial filter (a substrate query, a GROQ filter, an FTS5 WHERE clause, a post-fetch JavaScript .filter(...)), the audit must execute that exact filter against live data and confirm the consumer's expected records appear in the response.
| Audit verb | What it answers | Sufficient for PASS on read-path audit? |
|---|---|---|
| "Records exist" | Are the records in the data source? | NO |
| "Records carry intended properties" | Do the records have the field values we expect? | NO |
| "Records carry intended associations" | Are the records linked to the records they should be linked to? | NO |
| "The consumer's filter chain returns the records the consumer expects" | If we run the substrate query / GROQ filter / FTS5 query the consumer code actually runs, do the consumer's expected records appear? | YES (necessary, not sufficient alone) |
The first three are property checks against record state. The fourth is a behavior check against the consumer's runtime read path. Read-path audits require the fourth.
New SOP for Read-Path Audits
When auditing code that reads records via a substrate, GROQ filter, FTS5 query, or any post-fetch filter chain, Q must:
Identify the actual filter chain the consumer uses. Read the consumer code path end-to-end. Note every filter, projection, and join the consumer applies between fetch and render. The substrate header comment (e.g.
hubspot-library.tslines 11-15) is informative but not authoritative — the operative filter is whatever the consumer module actually invokes at runtime.Execute that filter chain against the live data source. Use the same substrate the consumer uses (
createHubSpotLibraryDataSourcefor HubSpot Library reads), the same GROQ query for Sanity reads, the same SQL for Content Vault reads, the same_GETendpoint for HubSpot v3 reads. If the consumer applies a post-fetch JavaScript filter, replicate that filter in a one-off script.Confirm the consumer's expected records appear in the actual response. "Records exist" is not "records appear in the consumer's filtered response." If the substrate filter excludes the records the consumer needs (because of property state, association state, or filter logic), the consumer renders empty even though the records are there.
If the consumer's expected records do not appear, classify as sev-1 functional finding. A read-path audit cannot return PASS while the consumer's runtime read returns nothing the audit was supposed to verify the consumer would render.
Static record-existence verification alone is INSUFFICIENT. A PASS verdict on a read-path audit requires the runtime filter chain check. Without that, the audit may PASS the records and miss the page rendering empty — exactly the Wave 4 outcome.
Coverage
This SOP applies to any Q audit covering code that reads records and applies post-fetch filtering before rendering. The non-exhaustive list:
| Audit class | Example surfaces | Filter chain check required |
|---|---|---|
| Compass surfaces | compass.valuefirstteam.com/[clientSlug]/* (data-model, walkthrough, library, Story So Far, canvas) |
YES — Library substrate portal_slug + listing_content_type |
| V1 portal sections | clients.valuefirstteam.com/[clientSlug]/* (28 sections, 46+ API routes) |
YES — per-section filter (clientSlug, document_type, listing_content_type, project_status, etc.) |
| Content Vault reads | /mnt/d/data/content-vault.db consumers (Vault, Catalog, /shows, /articles, episode pipeline) |
YES — FTS5 WHERE clauses, JOIN conditions, show_slug + status + published_at filters |
| Sanity GROQ-driven pages | valuefirstteam.com/* pages reading Sanity (episodes, articles, methodology, listings, courses) |
YES — GROQ *[_type == "x" && y == $z] filters and projections |
| HubSpot dashboards | /big3, /daily-ops, /weekly-*, /leadership-meeting outputs sourcing from HubSpot search |
YES — search filterGroups + property projection |
| Agent-rendered briefings | Herald, Sentinel, Pulse, Tide, Quorum, Marquee output that filters HubSpot/Vault records | YES — wherever the agent applies a filter to a fetch result before reporting |
The pattern: anywhere the audit verifies "records exist" but the consumer applies post-fetch filtering, the audit must execute the consumer's filter chain or the audit's PASS verdict is structurally weak.
Out of scope (no filter chain check required): pure write audits, agent-definition consistency audits, schema audits where no consumer filtering is involved, and audits where the consumer reads a record by ID with no filter.
Severity Reclassification
Q's Wave 4 audit verdict is proposed for retroactive reclassification:
| Field | Wave 4 as filed | Wave 4 as reclassified |
|---|---|---|
| Verdict | PASS-WITH-ISSUES | PASS-WITH-ISSUES |
| Sev-1 count | 0 | 0 |
| Sev-2 count | 0 | 1 (methodology finding: read-path audit closed without runtime filter-chain verification) |
| Sev-3 count | 4 (process-tracking and SOP-codification gaps) | 4 (unchanged) |
The audit's verdict was technically defensible against its own stated scope ("Read-only file inspection in V2 + V1 monorepos; live HubSpot GET against records 556337927839 and 556346234391; cross-reference of property-index against page code; signature comparison of discoverArchitecture import paths against existing Compass routes"). The verdict was not defensible against the methodology a read-path audit requires, because the scope itself was incomplete. The sev-2 finding records that the scope was the failure point — not that the agent executing the scope made an error within it.
Filing the sev-2 against the audit's methodology rather than against the agent who ran it preserves the principle that scope decisions are auditable, and that Q's own scope decisions are subject to the same standard Q applies to others.
portal_slug Audit-Trail Discrepancy (separate finding)
Q's Wave 4 audit reported (in F-12 evidence) that record 18819022093 (The Abs Company) carried portal_slug="abs-company". Showcase's Wave 5 live GET against the FSDM Listing 556337927839 and the CSB Listing 556346234391 returned portal_slug = null on those Listing records. These are two different objects (Company vs Listing), so the two reads do not directly contradict each other on a per-record basis. However:
- Q's Wave 4 audit DID NOT assert
portal_slugon the Listings themselves. The audit treatedportal_slugon the associated Company as evidence the Library substrate would resolve the Listings, but the substrate filters byportal_slugon the Listing record. - Showcase's Wave 5 read found
portal_slug = nullon both Listings. - The Wave 5 resolution stamped
portal_slug = "abs-company"onto both Listings.
Two questions remain unresolved and are flagged for follow-up investigation by Audit (data integrity agent) or Ledger:
| Question | Investigation owner | Resolution path if confirmed |
|---|---|---|
Did Q's Wave 4 GET inspect portal_slug on the Listings and miss the null, or did it never inspect that field on those records? |
Audit | If Wave 4 inspected and missed: tighten Q's GET-response inspection methodology — require explicit per-property assertions ("portal_slug on Listing 556337927839 = <value>"), not aggregate "intended payload confirmed" claims. If Wave 4 never inspected: the methodology gap is the same one this capability report addresses (the audit asked the wrong questions). |
Did the Listing's portal_slug change between Wave 4 (audit) and Wave 5 (Showcase diagnosis) — i.e., was the property silently cleared by some workflow, schema change, or UI action? |
Audit + Ledger | If the property was cleared between Wave 4 and Wave 5: surface as evidence that critical Listing properties can be silently mutated, and propose an audit-log mechanism on portal_slug and similar consumer-filter-bearing properties. |
This capability report does not resolve which of the two scenarios occurred. The factual ambiguity is itself the finding: an audit trail that cannot distinguish between "the audit was wrong" and "the record changed silently" is an audit trail with insufficient evidence resolution. Either outcome motivates a methodology tightening on Q's side, an integrity check on Audit's side, or both.
Implementation
| File | Purpose |
|---|---|
docs/quality/capabilities/2026-05-15-audit-must-execute-read-paths.md |
This document — methodology change record, SOP definition, severity reclassification, portal_slug discrepancy filing |
docs/quality/audits/2026-05-14-compass-data-model-page.md |
Wave 4 audit subject to retroactive reclassification (proposed sev-2 methodology finding) |
Registration Points
| System | Record |
|---|---|
| Capability report | docs/quality/capabilities/2026-05-15-audit-must-execute-read-paths.md |
| Subject audit | docs/quality/audits/2026-05-14-compass-data-model-page.md |
| Owning agent | Q (instruction-optimizer) |
| Methodology change scope | All Q audits classed as read-path audits per Coverage table above |
| Follow-up investigation owners | Audit (data-integrity), Ledger (HubSpot write gateway) — for portal_slug discrepancy |
Usage
When to Apply
Before opening any Q audit whose scope includes "read code that fetches and filters records," classify the audit as a read-path audit. For read-path audits:
- Identify the consumer's filter chain (substrate query, GROQ, FTS5, post-fetch JS filter)
- Execute that filter chain against the live source
- Confirm the consumer's expected records appear in the response
- Only then evaluate property-existence, association-existence, and code structure
If steps 1-3 cannot be completed (substrate unavailable, source offline, filter chain not yet specified), the audit MUST classify the verdict as PASS-WITH-ISSUES sev-2 minimum (methodology gap) until the filter-chain verification is performed.
What Changes for Audit Reports
Audit reports for read-path audits must now include a section titled Filter-chain verification that documents:
- The consumer module and the filter chain it applies (file:line citation)
- The exact query/filter executed against the live source (command or script)
- The records returned by the live filter
- Whether those records match the consumer's expected set
- Verdict on the filter-chain check (PASS / FAIL / DEFERRED)
A read-path audit without a Filter-chain verification section cannot return PASS — it returns PASS-WITH-ISSUES sev-2 minimum.
What Does NOT Change
This SOP does not require Q to write integration tests, deploy test harnesses, or maintain test suites. The filter-chain verification is a one-off live read at audit time, not a permanent test. Squire and the consuming module's owner remain responsible for whatever automated coverage the codebase warrants under existing policy (memory/feedback_no_local_build_gates_in_briefs.md — Vercel is the test environment).
This SOP does not change Q's authority, scheduling cadence, audit triggers, or report locations. It changes the methodology a single class of audits (read-path) must follow within the existing audit framework.
Why This Capability Matters
Wave 4 issued PASS-WITH-ISSUES on a build that rendered empty in production. The build was not broken — the records were not in the state the consumer's filter chain required. Q's audit checked the records. Q's audit did not check the chain. Mirror's authenticated walk caught the gap that Q's audit was structurally unable to catch.
This is a textbook case of the April 12 problem the QMS framework was created to solve: assumptions compound across sessions. Records exist (verified). Properties carry intended values (verified). Associations are intact (verified). Each link added confidence without adding evidence on the question that mattered: does the consumer's read return what the consumer needs?
The methodology change records the failure mode and prevents recurrence. It also makes Q's own scope decisions auditable — the sev-2 reclassification of Wave 4 is the audit-on-the-auditor that the QMS framework's verification protocol requires. Q audits processes owned by other agents; V reviews Q's own effectiveness at management review. This capability report is the artifact V will reference when reviewing Q's read-path audit performance.
The portal_slug discrepancy filing keeps a second open question visible: an audit trail that cannot distinguish "the audit was wrong" from "the record changed silently" is an audit trail that needs tightening. Filing the discrepancy without claiming resolution preserves the option for Audit and Ledger to investigate, and prevents the convenient closure that would let either failure mode recur.
Revision History
| Date | Change | Author |
|---|---|---|
| 2026-05-15 | Initial filing. Triggered by Q-vs-Mirror gap on Wave 4 audit of Compass /data-model page. Codifies read-path audit SOP, proposes Wave 4 sev-2 reclassification, files portal_slug audit-trail discrepancy for follow-up investigation. | Q |