{"incident_id": "INC-20260716-f1c4e6", "prefix": "Hary_Part1-Gateway-Auth-Rejection", "title": "hary-frontend failures after hary-ai change (hary-platform)", "description": "Synthetic demo fixture. hary-ai is pointed at the shared LLM gateway with a freshly configured virtual key. Every chat completion call comes back 401 Unauthorized on both auth schemes, the circuit breaker opens, the BFF times out, and the assistant panel goes dark for users. The gateway's own logs are not available, so whether the virtual key is unprovisioned or the Basic credential is wrong stays an open question \u2014 exactly the kind the video hands to the on-call engineer.", "started_at": "2026-07-16T09:09:00+00:00", "origin_service": "hary-ai", "failing_module": "hary.models.llm_factory", "failing_symbol": "hary.models.llm_factory.get_llm_for_tier", "blast_radius": 6, "evidence_count": 16, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown", "unknown"], "missing_evidence_count": 3, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Hary_Part1-Gateway-Auth-Rejection/replay.mp4", "keys": ["Hary_Part1-Gateway-Auth-Rejection/replay.mp4", "Hary_Part1-Gateway-Auth-Rejection/analysis.json", "Hary_Part1-Gateway-Auth-Rejection/poster.jpg", "Hary_Part1-Gateway-Auth-Rejection/briefing.md", "Hary_Part1-Gateway-Auth-Rejection/code-graph.mmd", "Hary_Part1-Gateway-Auth-Rejection/provenance.genblaze.json", "Hary_Part1-Gateway-Auth-Rejection/narration.genblaze.json"]}
{"incident_id": "INC-20260719-e1ca69", "prefix": "Hary_Part2-PII-Guardrail-Crash", "title": "hary-frontend failures after hary-ai change (hary-platform)", "description": "Synthetic demo fixture. Minutes after a hary-ai rollout bumps the text analyser engine, a user pastes a block copied out of an external editor: an unpaired surrogate followed by four thousand combining marks. The input PII pass raises IndexError instead of handling it, the input-guardrail node re-raises, and from then on every turn carrying that same byte sequence returns HTTP 500 while well-formed messages sail through. That makes it a partial failure \u2014 chat success falls to 71.4%, not to zero \u2014 so volume and latency dashboards stay flat and only the affected sessions notice. The third-party analyser engine emits no telemetry of its own, so whether this is a library defect in the rewritten span merge or a missing input-validation step in hary.guardrails.pii is exactly the question the video hands to the on-call engineer.", "started_at": "2026-07-19T09:38:00+00:00", "origin_service": "hary-ai", "failing_module": "hary.guardrails.pii", "failing_symbol": "hary.guardrails.pii._get_presidio", "blast_radius": 4, "evidence_count": 20, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown"], "missing_evidence_count": 2, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Hary_Part2-PII-Guardrail-Crash/replay.mp4", "keys": ["Hary_Part2-PII-Guardrail-Crash/replay.mp4", "Hary_Part2-PII-Guardrail-Crash/analysis.json", "Hary_Part2-PII-Guardrail-Crash/poster.jpg", "Hary_Part2-PII-Guardrail-Crash/briefing.md", "Hary_Part2-PII-Guardrail-Crash/code-graph.mmd", "Hary_Part2-PII-Guardrail-Crash/provenance.genblaze.json", "Hary_Part2-PII-Guardrail-Crash/narration.genblaze.json"]}
{"incident_id": "INC-20260722-b70c1c", "prefix": "Hary_Part3-Agentic-Retry-Exhaustion", "title": "hary-frontend failures after hary-ai change (hary-platform)", "description": "Synthetic demo fixture. A reworded tool-calling prompt template ships to hary-ai and the agent node starts receiving a function-call payload it cannot decode. AgentNode._is_retryable_llm_error treats the decode failure as retryable, so AgentNode.__call__ burns all three attempts on an error that can never succeed, then falls back to _partial_answer_result and returns a text-only prefix. Users get HTTP 200 with a plausible but truncated reply. Nothing 5xxs: success rate holds at 99.6% while agent p95 goes from 2.1 s to 6.9 s, the partial-answer salvage rate goes from 0.3% to 44.8%, and thumbs-down on the assistant panel climbs. The llm-gateway and model-provider supply no telemetry of their own, so whether the malformed payload is a model-side output regression or a defect in the new template cannot be settled from this evidence.", "started_at": "2026-07-22T14:08:10+00:00", "origin_service": "hary-ai", "failing_module": "hary.graph.nodes.agent", "failing_symbol": "hary.graph.nodes.agent.AgentNode.__call__", "blast_radius": 3, "evidence_count": 18, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown"], "missing_evidence_count": 2, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Hary_Part3-Agentic-Retry-Exhaustion/replay.mp4", "keys": ["Hary_Part3-Agentic-Retry-Exhaustion/replay.mp4", "Hary_Part3-Agentic-Retry-Exhaustion/analysis.json", "Hary_Part3-Agentic-Retry-Exhaustion/poster.jpg", "Hary_Part3-Agentic-Retry-Exhaustion/briefing.md", "Hary_Part3-Agentic-Retry-Exhaustion/code-graph.mmd", "Hary_Part3-Agentic-Retry-Exhaustion/provenance.genblaze.json", "Hary_Part3-Agentic-Retry-Exhaustion/narration.genblaze.json"]}
{"incident_id": "INC-20260708-553388", "prefix": "Other-Architectures/Cache-Stampede", "title": "web failures after results-cache change (search-platform)", "description": "Synthetic demo fixture. A results-cache node restarts cold during a scheduled maintenance window and comes up with an empty keyspace. The read-through path's single-flight registry is in-process, so it comes up empty too: every miss opens its own backfill, the fan-out floods the index tier, index query admission saturates, database p95 climbs from 14 ms to 480 ms, and the search page returns 504s until the prewarm loop refills the ring. No deployment involved. No index-tier or database source is supplied, so whether the index tier could have absorbed the fan-out with more executor threads stays an open question.", "started_at": "2026-07-08T14:02:00+00:00", "origin_service": "results-cache", "failing_module": "search.cache.readthrough", "failing_symbol": "search.cache.readthrough.ReadThroughCache.get_or_backfill", "blast_radius": 3, "evidence_count": 26, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown"], "missing_evidence_count": 2, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Other-Architectures/Cache-Stampede/replay.mp4", "keys": ["Other-Architectures/Cache-Stampede/replay.mp4", "Other-Architectures/Cache-Stampede/analysis.json", "Other-Architectures/Cache-Stampede/poster.jpg", "Other-Architectures/Cache-Stampede/briefing.md", "Other-Architectures/Cache-Stampede/code-graph.mmd", "Other-Architectures/Cache-Stampede/provenance.genblaze.json", "Other-Architectures/Cache-Stampede/narration.genblaze.json"]}
{"incident_id": "INC-20260710-4f8199", "prefix": "Other-Architectures/Checkout-Secret-Rotation", "title": "frontend failures after payment-api deployment (checkout-platform)", "description": "Synthetic demo fixture. A payment-api deployment repoints the service at a new database secret reference; the credential resolver hands the pool a password the payment-db role rejects with 28P01, the result is memoised so retries amplify, order-api's worker pool saturates and checkout starts failing for shoppers. No database-side or secret-store audit logs are supplied, so whether the new secret was never applied to the database role or the wrong value was stored stays an open question.", "started_at": "2026-07-10T02:01:00+00:00", "origin_service": "payment-api", "failing_module": "payments.db.credentials", "failing_symbol": "payments.db.credentials.get_db_credentials", "blast_radius": 10, "evidence_count": 21, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown"], "missing_evidence_count": 3, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Other-Architectures/Checkout-Secret-Rotation/replay.mp4", "keys": ["Other-Architectures/Checkout-Secret-Rotation/replay.mp4", "Other-Architectures/Checkout-Secret-Rotation/analysis.json", "Other-Architectures/Checkout-Secret-Rotation/poster.jpg", "Other-Architectures/Checkout-Secret-Rotation/briefing.md", "Other-Architectures/Checkout-Secret-Rotation/code-graph.mmd", "Other-Architectures/Checkout-Secret-Rotation/provenance.genblaze.json", "Other-Architectures/Checkout-Secret-Rotation/narration.genblaze.json"]}
{"incident_id": "INC-20260724-9a5fb9", "prefix": "Other-Architectures/Queue-Poison-Message", "title": "member-portal failures after checkout-service change (messaging-platform)", "description": "Synthetic demo fixture. A producer promotes an order-event payload schema in which the 'recipient' field stops being a string and becomes a contact object. The stream consumer's decoder raises on the first such record, the worker exits and re-reads the very same offset, the dead-letter topic that should have quarantined it is not provisioned, partition backlog and consumer lag climb, notification delivery stalls and members see an empty notification history. The queue broker and the schema registry supply no telemetry of their own, so whether the producer's schema change was deliberate, reviewed or announced cannot be established from this evidence.", "started_at": "2026-07-24T09:08:30+00:00", "origin_service": "stream-consumer", "failing_module": "stream.codec.decoder", "failing_symbol": "stream.codec.decoder.decode_record", "blast_radius": 7, "evidence_count": 18, "leading_confidence": 0.99, "hypothesis_statuses": ["confirmed", "inferred", "inferred", "unknown"], "missing_evidence_count": 2, "video_url": "https://s3.us-east-005.backblazeb2.com/Hackproject/Other-Architectures/Queue-Poison-Message/replay.mp4", "keys": ["Other-Architectures/Queue-Poison-Message/replay.mp4", "Other-Architectures/Queue-Poison-Message/analysis.json", "Other-Architectures/Queue-Poison-Message/poster.jpg", "Other-Architectures/Queue-Poison-Message/briefing.md", "Other-Architectures/Queue-Poison-Message/code-graph.mmd", "Other-Architectures/Queue-Poison-Message/provenance.genblaze.json", "Other-Architectures/Queue-Poison-Message/narration.genblaze.json"]}