Corrective Action: Ledger Silent Association Failures (N=3 in one session)
Date: 2026-05-18 Category: Gateway Verification Gap (Governance-Class) Impact: 43 association edges reported "verified complete" by Ledger spawns when zero had actually been written. Records existed; associations did not. Caught by direct
list-assocGET-back; remediated in-session via direct v4 PUT. Resolution Time: ~90 minutes across three incidents in the same session
Incident
What Happened
Within a single working session on 2026-05-18, three separate Ledger spawns returned "verified" reports on association writes where the association edges were never actually persisted. In each case the parent record (Deliverable, Ticket, Note) was created successfully; the associations to Company / Project / Ticket were silently dropped. Ledger's internal verification logic did not catch any of the three.
Timeline
| Time | Incident | Detail |
|---|---|---|
| Afternoon | #1 — Deliverable associations | Ledger created Deliverable 55587822486 ("Compass Design System v1") and reported associations to VFT Company 49241304942 and Compass Project 556916002923 as "pending due to CLI custom-object type resolution issue." Direct list-assoc 2-18484424 55587822486 companies returned []. Same for ... 0-970 returned []. Fixed via direct v4 PUT: typeId 76 (Company, USER_DEFINED) and typeId 572 (Project, USER_DEFINED). |
| Afternoon | #2 — 21 Ticket→Company associations | Same spawn created 21 UAT Tickets in pipeline 897410592 using v3 POST with inline associations field. Ledger reported all 22 associations (21 Tickets + 1 Deliverable from #1) "wired and verified." Sample list-assoc on 4 tickets returned [] for all 4. Fixed via 21 direct v4 PUT calls (typeId 26, HUBSPOT_DEFINED, ticket_to_company). |
| Same session | #3 — 21 Note→Ticket associations | A different Ledger spawn created 21 synthesis Notes using v3 POST with inline associations. Ledger reported all 42 writes (21 Notes + 21 stage transitions) verified with full GET-back table. Sample on 2 Notes via list-assoc notes {id} tickets returned [] for both. Fixed via 21 direct v4 PUT calls (typeId 18, HUBSPOT_DEFINED, note_to_ticket). |
Root Cause
Two failure modes contribute, and both bypass Ledger's existing verification logic.
1. CLI associate command defaults to --category HUBSPOT_DEFINED. Any custom or USER_DEFINED label fails silently or returns a misleading object-type error. This is documented in wiki/conventions.md § HubSpot data conventions, but Ledger's spawn invocation did not pass the correct category for the custom Deliverable associations in Incident #1.
2. The v3 POST inline associations field silently drops on writes. Incidents #2 and #3 both used the documented v3 inline-associations payload shape. The parent records were created. The association edges were not. No HubSpot error returned; the API responded 2xx with full record bodies. Either HubSpot's v3 path does not honor inline associations for the typeIds used (26, 18), or the payload shape is subtly wrong in a way that produces no error.
Why this is governance-class. Per the Governance Activation Rule (2026-04-25, MEMORY.md): a documented protocol is not operational until mechanical enforcement makes the failure mode structurally hard. Ledger's verification protocol exists in skills/hubspot-write-gateway/, but the verification ran against Ledger's own internal state ("did I send the call?"), not against the live HubSpot graph ("did the edge persist?"). Three reports of "verified complete" with zero true associations is the same shape as the Canon over-confident-completion pattern documented in memory/feedback_canon_verify_after_write.md — parallel gateway, same defect.
Category
Gateway Verification Gap. Mirrors 2026-04-03-corrective-sanity-batch-transaction-silent-failure.md (Sanity returning success on dropped writes) and 2026-03-16-corrective-ledger-bypass-slash-commands.md (Ledger governance enforcement). Distinct from those: this is not a bypass and not a payload-size limit — it is a verification scope defect inside the gateway itself.
Containment
All 43 missing association edges were remediated in-session via direct v4 PUT calls. Post-remediation list-assoc queries confirmed each edge. No further data action required.
Corrective Actions
| # | Owner | Action | Deliverable | Due |
|---|---|---|---|---|
| 1 | Hone | Update skills/hubspot-write-gateway/ contract with three additions: (a) mandate direct v4 PUT for any association write — never CLI associate, never v3 inline associations; (b) require post-write list-assoc GET-back as evidence on every association claim; (c) refuse to mark any association "verified" without GET-back evidence in the return payload. |
Updated skill pack + manifest section on association writes | 2026-05-25 |
| 2 | Archivist | Document non-default association categories and forward/reverse typeIds for every object pair Ledger may write, in skills/hubspot/property-index/associations.json (or new file). Ledger should look up the right typeId + category without guessing. Populate from verified-live state (no intended state per feedback_property_index_verified_only.md). |
Verified association reference file | 2026-05-25 |
| 3 | Q | Add gateway-write contract to docs/quality/verification-protocol.md: every Ledger and Canon return payload must include either GET-back evidence or an explicit unverified marker. Add this CAR to the verification protocol's referenced incidents. |
Updated verification protocol | 2026-05-21 |
Preventive Action
The structural fix is verification-as-evidence inside the gateway return contract. Ledger must include, for every claimed association:
{
"association": { "from": {...}, "to": {...}, "typeId": N, "category": "..." },
"verified": true,
"evidence": { "command": "list-assoc ...", "edges_returned": [...], "timestamp": "..." }
}
If evidence.edges_returned does not contain the claimed edge, Ledger returns verified: false and the orchestrating agent halts. This makes silent association failures structurally hard: the gateway cannot report success without proof.
Mechanical follow-on (deferred to Hone): a pre-return assertion inside the Ledger skill that fails closed when evidence is absent on any association claim.
Effectiveness Verification
| Signal | Threshold | Window |
|---|---|---|
| Silent association incidents (Ledger reports "verified" with no actual edge) | 0 | 30 days from 2026-05-18 |
| Ledger return payloads containing GET-back evidence on association claims | 100% | 30 days from skill-pack update |
| CARs filed referencing this root cause | 0 | 60 days |
Q will audit the next 5 Ledger spawns that perform association writes, sampling for GET-back evidence in the return payload. Findings filed to docs/quality/audits/ per the standard Tier 1 audit cadence.
Lessons
Gateway verification scope must extend to the live graph, not the gateway's internal call state. "Did I send the API call?" is not verification — it is intent. "Does the API graph reflect the change?" is verification. The Canon parallel (2026-05-05 feedback) and now Ledger (N=3 in one session) confirm this is a gateway-pattern defect, not a single-agent defect. Both gateways now require explicit post-write GET-back evidence in their return payloads.
The CLI associate category default and the v3 inline-associations silent-drop are both HubSpot-API-side surprises that Ledger should be opinionated about: never use those paths. Direct v4 PUT with explicit typeId and category is the only sanctioned association path.
Related Incidents
| Date | Incident | Relationship |
|---|---|---|
| 2026-04-03 | Sanity batch transaction silent failure | Same shape — API returns success while dropping writes. Resolution required batching + verify-after-write. |
| 2026-03-16 | Ledger bypass via slash commands | Different mode (bypass vs verification gap) — same governance class. Documented Ledger as the sole HubSpot write authority. |
| 2026-05-05 | Canon verify-after-write feedback (memory/feedback_canon_verify_after_write.md) |
Parallel gateway, same defect. Canon now requires GROQ verify post-write; Ledger gets the equivalent here. |
| 2026-04-25 | Governance Activation Rule | Foundational: documented protocol does not equal operational protocol until mechanical enforcement makes failure structurally hard. |