Corrective Action: Morning Show Cross-Contamination — Pipeline Linked Wrong Mux Recordings to AI Daily Episodes

← Back to Corrective Actions 2026-03-30

Corrective Action: Morning Show Cross-Contamination — Pipeline Linked Wrong Mux Recordings to AI Daily Episodes

Date: March 30, 2026 Category: API Behavior Assumption + Verification Failure Impact: 6 AI Daily episode pages displayed Wake Up Customer Platform (or other show) video for 2-4 weeks. Visitor-facing. Show page carousel displayed wrong thumbnails. Resolution Time: ~3 hours across two sessions (Mar 29 inventory + analysis, Mar 30 cleanup)

Incident

What Happened

The content pipeline (run-pipeline.ts) linked Mux recordings from Wake Up Customer Platform, Value-First Humans, Value-First Data, and Value-First Revenue Architecture to AI Daily episode pages. Visitors clicking an AI Daily episode saw a completely different show's video. The AI Daily show page carousel displayed thumbnails from wrong shows, making the page look broken and unprofessional. Chris identified the issue from the show page carousel and episode detail pages.

Timeline

Time Event
~Feb 2026 onward Pipeline begins linking morning Mux recordings using time-window matching
Mar 27 (evening) /media-recap run reveals thumbnail issues on AI Daily show page
Mar 29 (morning) Chris reports Wake Up thumbnails appearing on AI Daily carousel
Mar 29 (afternoon) Full inventory audit across Sanity, Mux, and Google Calendar identifies 6 mismatched episodes
Mar 29 (afternoon) Pipeline matching logic rewritten to use Mux passthrough metadata
Mar 29 (afternoon) muxVideo reference unlinked from 6 episodes — but wrong video persists
Mar 30 (morning) Discovered muxPlaybackId direct string field also contains wrong ID — coalesce() fallback serves it
Mar 30 (morning) Cleared muxPlaybackId, muxAssetId, and hostingProvider on all 6 episodes
Mar 30 (morning) All 6 episode pages verified clean after Sanity CDN cache expiry (~2 min)

Root Cause

Cause 1: Time-Window Matching Cannot Distinguish Back-to-Back Shows

The matchAssetToPreCreatedEpisode() function in run-pipeline.ts matched Mux recordings to Sanity episodes using a 30-minute time buffer around each show's scheduled time slot. AI Daily streams at 7:00 AM CT, Wake Up Customer Platform at 7:30 AM CT. Both shows are within each other's 30-minute buffer window.

When multiple unlinked episodes existed for the same day, the pipeline would match the first Mux asset to the first available episode — regardless of which show the recording actually belonged to. When only one unlinked episode remained (the other already linked), the pipeline's "single episode = safe match" shortcut (line 508) force-matched regardless of show identity.

The Mux passthrough metadata already contained the correct show name — written by the pipeline itself when enriching the Mux asset. The matching logic ignored this metadata entirely and relied solely on time proximity.

Cause 2: Dual Video Field Paths in Sanity

Episode documents store video identity in two independent locations:

  1. muxVideo — A Sanity reference to a mux.videoAsset document (contains playbackId via dereferencing)
  2. muxPlaybackId — A direct string field on the episode document

The episodeBySlugQuery GROQ query uses coalesce() to try both paths:

"muxData": coalesce(
  muxVideo.asset->{ playbackId, assetId, status },
  select(defined(muxPlaybackId) => { "playbackId": muxPlaybackId, "assetId": muxAssetId })
)

When the initial cleanup unlinked only muxVideo (the reference), the coalesce() fell through to muxPlaybackId (the direct string) which still contained the wrong recording's ID. The episode pages continued serving the wrong video because the fallback path was invisible during the first fix attempt.

Category: API Behavior Assumption + Verification Failure

The pipeline assumed time proximity was sufficient to identify show recordings. It was not. The dual-field architecture was not documented or visible — the cleanup assumed clearing one field would remove all video references, without verifying the query's actual resolution path.

Fix Applied

Immediate Resolution

  1. Identified all 6 mismatched episodes via Mux passthrough metadata cross-reference
  2. Cleared ALL video fields: muxVideo, muxPlaybackId, muxAssetId, hostingProvider
  3. Verified each episode page rendered without the wrong video after Sanity CDN cache expiry

Code Changes

File Change
agents/content-pipeline/run-pipeline.ts Rewrote matchAssetToPreCreatedEpisode() — Priority 1: match by Mux passthrough metadata (showName/showSlug). Priority 2: single-episode-per-day. Priority 3: time-slot with tightened 15-min buffer (was 30).
agents/content-pipeline/run-pipeline.ts linkAssetToEpisode() now sets status: 'published' when linking (separate fix from same session)

Data Changes

Episode What Was Cleared
episode-vf-ai-daily-2026-03-26 Had Wake Up Customer Platform recording
episode-vf-ai-daily-2026-03-24 Had Value-First Humans recording
episode-vf-ai-daily-2026-03-23 Had Wake Up Customer Platform recording
episode-vf-ai-daily-2026-03-17 Had Wake Up Customer Platform recording
episode-vf-ai-daily-2026-03-12 Had Value-First Revenue Architecture recording
episode-vf-ai-daily-2026-03-10 Had Value-First Data recording

All six: muxVideo, muxPlaybackId, muxAssetId, hostingProvider set to null.

Verification

# Confirmed each page no longer contains mux-player or playback-id references
for slug in vf-ai-daily-2026-03-26 vf-ai-daily-2026-03-24 vf-ai-daily-2026-03-23 \
            vf-ai-daily-2026-03-17 vf-ai-daily-2026-03-12 vf-ai-daily-2026-03-10; do
  curl -sL "https://valuefirstteam.com/media/episodes/$slug" | grep -c "mux-player"
done
# All returned 0

Prevention Measures

Rules Added

Layer File Rule
Critical Lessons MEMORY.md NEVER match Mux recordings to episodes by time window alone. Always use Mux passthrough metadata (showName/showSlug) as the primary match key. Time-window matching is fallback only for assets without metadata.
Critical Lessons MEMORY.md Sanity episodes have TWO video paths: muxVideo (reference) and muxPlaybackId (direct string). The episodeBySlugQuery uses coalesce() to try both. When clearing video from an episode, ALWAYS clear all four fields: muxVideo, muxPlaybackId, muxAssetId, hostingProvider.
Pipeline Code run-pipeline.ts matchAssetToPreCreatedEpisode() now checks passthrough metadata FIRST, logs match source, refuses to match cross-show when metadata is present.
Pipeline Code run-pipeline.ts Time buffer tightened from 30 min to 15 min for fallback matching.

Detection Triggers

If the pipeline reports "Matched by time slot" for a morning show (AI Daily or Wake Up), this should be treated as suspect. The passthrough metadata path should be the norm. A "Skipped: Metadata says X but no matching unlinked episode" log is expected and correct behavior — it prevents cross-contamination.

Lessons

  1. When two systems write the same identity (Mux playback ID) to two different fields, cleanup must address both. The coalesce() pattern in GROQ is invisible until you read the query — you can clear one path and the other silently takes over. Before clearing any Sanity field that affects rendering, read the GROQ query that fetches it and verify all resolution paths.

  2. Metadata-based matching is always more reliable than time-based matching. The Mux passthrough metadata existed from the start — the pipeline wrote it when enriching assets. Using it for matching was the obvious approach; the time-window was a shortcut that became a liability. When you have identity data, use it. Don't approximate.

  3. Back-to-back shows sharing infrastructure (StreamYard, Mux live stream) will always create collision risk. Any pipeline that processes recordings from shared infrastructure must have a show-identity signal that's stronger than timing. This applies to future shows added to the morning block.

Related Incidents

  • Substring date matching (Feb 21, 2026): "Feb 2" matching "Feb 20" — same category of insufficient matching specificity. Fixed in run-pipeline.ts with trailing comma delimiter.
  • HubSpot Broadcast API no draft mode (Feb 18, 2026): API behavior assumption — assumed omitting triggerAt would create a draft. Same pattern of not verifying actual behavior before batch operation.
  • Portal Documents filter exact match (Mar 2, 2026): listing_content_type === 'document' exact match — invisible filter causing content to not appear. Same pattern of invisible resolution path.