diagnose-why-work-stopped

**Why it stopped:** The "Assign owner" node's condition is `assignee.email is not empty`, and the request's assignee field is blank — so the condition evaluates to false. The tree has no branch for that case, so execution ends there with no further nodes running and no error surfaced. That's a silent stall: the tree "completes" from the engine's perspective, but the onboarding flow never continues.

**Product rule to add:** Every conditional node must have a defined fallback (default/else) branch so that when the condition is false, execution continues on a specified path instead of stopping.

Concretely for this tree:

1. Add an **else branch** on "Assign owner" — e.g., if `assignee.email is empty`, route to a fallback node: "Assign to default onboarding queue/owner" (or "Notify admin: unassigned request"), then continue to "Send welcome email."
2. Optionally add a **stall alert rule**: if a tree reaches a node and no branch executes within X minutes, emit a notification/error event so stalls are never silent.

Flow becomes: Start → Validate request → Assign owner → (assignee set → Send welcome email | assignee empty → Assign default owner → Send welcome email).

## Forensics — exact stop point

- **Stop node:** `Assign owner` (condition: `assignee.email is not empty`).
- **Evidence:** Logs show the tree reached `Assign owner` and no further node transitions; the request's `assignee` field is blank, so the condition evaluates **false**.
- **Mechanism:** `Assign owner` is a conditional gate with no false-branch. When the condition fails, the node has no outgoing action path — it's neither terminal, explicitly waiting, nor explicitly live. Downstream nodes (`Send welcome email`) never get queued. This is a **silent pseudo-stop**: the tree looks "running" but has no next action.

## Classification

| Node | State | Classification |
|---|---|---|
| Start, Validate request | completed | Already covered |
| Assign owner | condition false, no false-branch | **Agent-actionable but not routed** — needs a rule that routes the false case |
| Send welcome email | never queued | Blocked by the above |

## General product rule (contract, not a patch)

> **Every conditional node must finish each run with a defined disposition for *every* branch outcome.** A condition that evaluates false must route to either (a) an explicit alternate action path (e.g., fallback assignee, escalation to an owner queue), or (b) an explicit waiting state with a named human owner and a bounded re-check. A node may never end a run with an unhandled branch — the engine must detect "condition false, no false-branch" at validation time and reject the tree config, and at runtime surface it as a visible blocked state with an owner, never as silent inactivity.

**Invariant check:**
1. *Productive work continues* — the false branch keeps the tree moving (fallback assignee or bounded re-check after assignment), so onboarding doesn't need a human to wake it.
2. *Only real blockers stop work* — a missing assignee is a genuine human dependency, so stopping is legitimate **only** if it's explicit and owned; silent stops are eliminated.
3. *No infinite loops* — the re-check is bounded (e.g., max N re-evaluations, then escalate to a named owner); no recursive recovery.

**Doc note:** `doc/execution-semantics.md` should gain an explicit "unhandled branch = invalid disposition" entry under post-run disposition, so this is a documented contract rather than an engine behavior.

## Phased plan (no code yet)

- **Phase 0 (live tree):** Assign an owner to the affected onboarding requests (or set a fallback assignee), re-run the tree; leave an audit comment — don't hide the stall.
- **Phase 1 (docs):** Update `doc/execution-semantics.md` with the unhandled-branch contract. — *ClaudeCoder*
- **Phase 2 (detection):** Config-time validation rejecting conditional nodes without both branches; runtime detection surfacing a blocked-with-owner state. — *CodexCoder*
- **Phase 3 (UI):** Visible "blocked: unhandled condition" state with named owner. — *UXDesigner*
- **Phase 4 (QA + review):** Validation tests incl. the onboarding tree regression; CTO sign-off. — *QA*, blocked by Phases 2–3.

Next step per process: open a `request_confirmation` on this plan revision before any child issues or code.

**Why it loops:** Your tree has no exit path for persistent failure. Node 2 only branches on "API error → wait 5 minutes," and node 3 unconditionally returns to node 1. So a permanently failing invoice (e.g., invalid data, duplicate ID, expired auth token) retries forever: 3 hours ÷ 5 min ≈ 36–40 cycles, which matches your logs exactly. The workflow treats every error as transient, and nothing distinguishes "retryable" from "fatal."

**No-code rule to add:** Add a retry-limit + escalation rule on the API-error branch:

- **Condition:** "API error occurred **AND** retry count for this invoice ≥ 3"
- **Then:** Stop the loop for this invoice, mark it as "Failed – needs review," and send an alert (email/Slack) to the on-call person with the invoice ID and error message.
- **Else:** Wait 5 minutes and retry (existing behavior).

Optionally, also add a second alert rule: "Same invoice failed N times within 1 hour → alert immediately," so silent loops like this 40-retry run get caught early. And consider a "duplicate invoice detected → skip and log" branch, since same-invoice repetition often means the error is deterministic, not transient.

## Why it loops

The tree has no exit condition on failure. Node 2 treats "API error" as a transient retryable state, and node 3 unconditionally returns to node 1 — so the loop is a retry loop with **no retry cap, no error classification, and no escalation path**. Evidence: 40 identical failures over 3 hours (every ~4.5 min, matching the 5-min wait) for the *same* invoice means the error is persistent (e.g., validation rejection, duplicate invoice, bad credentials), not transient — but the tree can't tell the difference. And because the loop is technically "still running," it never hits a stop condition, so no alert fires. This is a pseudo-live state: it looks productive but makes zero progress.

## No-code rule to add

**Contract:** *Every retry loop must be bounded and must distinguish transient from persistent failure. After N consecutive identical failures (suggest N=3) on the same item, the tree must exit the loop and route to a named owner — it may not silently continue retrying.*

Concretely, add to the tree:

1. **Retry cap:** max 3 attempts per invoice.
2. **Error classification:** on each failure, tag it transient (timeout, 5xx, rate limit → retry) vs. persistent (4xx validation, duplicate, auth → no retry).
3. **Escalation node:** when the cap is hit or the error is persistent, create a blocked state assigned to a human owner (accounting/ops) with the error detail attached — this is a real blocker, so it legitimately stops work and fires an alert.
4. **Alerting rule:** any item failing ≥3 consecutive times raises a notification; "still retrying" is never a substitute for surfacing failure.

This preserves all three invariants: productive work continues for transient errors (bounded retries), only genuine blockers (persistent failure needing a human) stop the tree, and the loop is bounded so it can't spin indefinitely.

## Why it over-recovered

The loop is self-perpetuating because the recovery path and the failure path form a cycle with no exit condition:

1. Refund fails → creates manual review task
2. Review task done → re-runs Refund
3. Refund fails again → creates *another* manual review task
4. Goto 2

Nothing in the config distinguishes a *transient* failure (worth retrying) from a *deterministic* one (never going to succeed). If the refund fails for a persistent reason — card expired, gateway rejects the refund, refund exceeds original charge amount, account closed — every retry fails identically, so the system just manufactures a new review task each cycle. The review task is doing real human work (someone marks it "done"), but "done" only means "I looked at it," not "I fixed the underlying cause." So the loop burns 14 review cycles on a refund that was never going to succeed, and each cycle looks like progress.

The structural problem: **recovery is keyed to task completion, not to a state change in the underlying failure.** The system treats "a human touched it" as evidence the failure was resolved.

## Product rules to set

1. **Cap retries per refund attempt chain** — e.g., max 2–3 refund attempts per original refund request. After the cap, stop creating review tasks and move the refund to a terminal `failed_refund` state requiring a distinct resolution path (manual payout, support escalation).

2. **Classify failures before retrying.** On refund failure, inspect the error: transient (timeout, 5xx, gateway unavailable) → auto-retry with backoff, no review task. Deterministic (card invalid, amount mismatch, policy rejection) → go straight to terminal state / human escalation queue, never re-run Refund automatically.

3. **Require the review task to change something.** "Done" should only trigger a re-run if the reviewer recorded a corrective action (e.g., updated payment method, corrected amount). A plain "done" with no change closes the case instead of re-triggering.

4. **Dedupe/idempotency key.** One open review task per refund request at a time — a new review task can't be created while a prior one for the same refund is unresolved.

The minimal fix if you only do one thing: the retry cap (#1). It converts an infinite loop into a bounded one and forces failures into a terminal state where a human with actual authority (not a "done" checkbox) resolves them.

## Root cause: recursive recovery

Your config is a self-feeding recovery loop:

1. **Refund fails → creates manual review task.** That's a legitimate recovery step.
2. **Review done → re-runs Refund.** Also legitimate — a retry after human review.
3. **Refund fails again → creates *another* manual review task.** This is the bug.

Nothing in the chain distinguishes "first failure" from "retry after recovery also failed." The recovery path (review → re-run) uses the same failure handler as the original path, so each cycle spawns a fresh recovery task. Recovery is recovering its own recovery — the canonical recursive-recovery loop. With no depth counter, no escalation, and no terminal state for "refund failed N times," the tree just keeps deepening: 14 reviews, 14 refund attempts, and it would have kept going indefinitely until the refund happened to succeed or someone intervened manually.

## The product rule (stated as a contract, not a patch)

**"Every recovery path must be bounded and must escalate rather than recurse. A recovery action may re-trigger the original work at most once per issue; a second failure on the same work item must terminate the recovery chain and route to a named human owner."**

Concretely for refunds:

- **Depth bound:** tag each retry with an attempt count (or parent-recovery link). If a refund fails *after* a manual-review retry, do **not** create another review task — the review already happened and didn't fix it.
- **Escalation, not recursion:** on the second failure, mark the refund attempt **terminal-failed** and open a single escalation task assigned to a named owner (e.g., payments on-call), with the full attempt history attached. No further automated retries.
- **Audit trail preserved:** keep all 14 existing review tasks and attempts visible; don't delete or mark them done to clean up. New rule applies going forward.

## How the three invariants hold

1. **Productive work continues:** first-failure recovery (review → one retry) still runs automatically — no user wake needed.
2. **Only real blockers stop work:** the chain doesn't silently stall; it ends in an explicit terminal state plus a routed escalation with a named owner.
3. **No infinite loops:** the retry-after-recovery path is bounded at one re-trigger; deeper failures escalate to humans instead of spawning more recovery tasks.

**Immediate cleanup on the affected order:** cancel the pending review tasks beyond the first, mark the refund terminal-failed, and assign one escalation task to the payments owner with the attempt log.

## Problem-by-problem diagnosis

**Order 4821 — duplicate fulfillment + double tracking**
- The "if timeout → Retry" branch assumes a timeout means failure. But the first create actually succeeded — the timeout was just a slow response. The retry ran anyway, creating a duplicate fulfillment order, and each one sent tracking.
- Root cause: no idempotency. The workflow treats "no response" as "not done," and nothing checks whether a fulfillment order already exists before retrying.

**Order 4822 — 30 rechecks over 30 days**
- The unpaid loop (Wait 1 day → Recheck → if still unpaid → Wait) has no exit condition other than payment succeeding. A permanently unpaid order loops forever.
- Root cause: unbounded retry loop — no max attempts and no escalation path.

**Order 4823 — blank warehouse location, silent failure**
- The condition "warehouse location not empty" simply evaluated false, so Create fulfillment order was skipped. There's no else branch and no alert, so the order silently stalled.
- Root cause: a guard condition with no fallback — false paths that aren't handled just dead-end.

## No-code rules to add

1. **Idempotency key on fulfillment creation** — before retrying, check "does a fulfillment order already exist for this order ID?" If yes, skip creation and go straight to Send tracking.
2. **Verify-before-retry** — on timeout, add a status-check step (query fulfillment order status) before triggering Retry; only retry if status = not created.
3. **Deduplicate Send tracking** — only send tracking if no tracking notification was already sent for this order ID.
4. **Loop limit on payment recheck** — max 3 rechecks (or 3 days). After the limit, branch to "Cancel order / notify customer / alert ops" instead of looping back to Wait.
5. **Else branch on the warehouse condition** — if warehouse location is empty → send alert to ops ("Order {id} missing warehouse location") and pause the order, rather than silently skipping.
6. **Data validation at Start** — reject or flag orders with missing warehouse location up front, before entering the fulfillment branch.

Net effect: retries become safe (idempotent), loops terminate, and every "false" condition has an explicit, alerted path instead of a silent dead end.

## Why each problem happened

**Order 4821 — duplicate fulfillment + double tracking.** The retry rule fires on *timeout*, but a timeout is ambiguous: the create may have succeeded while the response was lost. The tree has no idempotency key or "check before retry" step, so the retry blindly re-created the order, and each fulfillment order then sent its own tracking notification.

**Order 4822 — 30-day unpaid loop.** The Wait → Recheck → Wait cycle has no bound and no exit condition other than payment succeeding. Nothing classifies "still unpaid after N checks" as a real blocker needing a human, so the tree loops forever — an unbounded continuation loop.

**Order 4823 — silent skip on blank warehouse location.** The condition "warehouse location not empty" failed, so Create fulfillment order never ran — but there's no else-branch or alert. The order fell off the tree silently: a pseudo-stop that looks like progress but has no action path.

## No-code rules to add

1. **Idempotency rule (4821):** Every fulfillment create carries an idempotency key (order ID). Before any retry, first *query* whether the fulfillment order already exists; retry only if it does not. Send tracking only once per order ID, not per fulfillment attempt.
2. **Bounded retry rule (4821/4822):** Any retry/wait loop must have a max attempts and a max total duration. On exhaustion, the order moves to an explicit "needs human" state with a named owner — never loops silently.
3. **No silent skip rule (4823):** Every condition node must have both branches. If a precondition fails (e.g., blank warehouse location), route to an alert/escalation path (notify ops, mark order blocked) — a failed condition is a real blocker, not a quiet end.
4. **Timeout ≠ failure rule (4821):** Treat timeouts as "unknown," not "failed": verify outcome before any compensating action.

These preserve all three invariants: productive work continues (valid orders flow through), only real blockers stop work (blank location and exhausted retries escalate to humans instead of stalling), and no infinite loops (all loops are bounded with explicit exits).

**Walkthrough**

1. **Start** → proceeds to **Get ticket** — succeeds (ticket exists, just missing the `priority` field).
2. **If ticket.priority = 'high'** → this is the exact stop point. With no `priority` field present, the condition can't evaluate to true or false — it's undefined/null, so the branch fails and the tree halts here.
3. Neither **Assign to tier 2** nor **Assign to tier 1** is ever reached; **End** is not reached.

**Exact stop point:** the `If ticket.priority = 'high'` condition node.

**No-code product rule to prevent the stop**

Add a **default/fallback branch** on the condition node:

- Rule: "If `ticket.priority` is empty or missing → treat as `low` (or `medium`) and route to **Assign to tier 1**."
- Implementation, no-code: on the If node, add an "Else if field is empty" branch (or set a "Default value when missing = low" on the field check), so every ticket always matches exactly one branch and the tree can never stop on a missing field.

Optionally pair it with a second rule: "If `ticket.priority` is missing → set field to default and flag ticket for priority backfill," so data quality improves over time without changing the flow logic.

## Walk of the tree

1. **Start** → passes through fine.
2. **Get ticket** → succeeds; the ticket exists, just lacks a `priority` field.
3. **If ticket.priority = 'high'** → **STOP POINT.** The condition evaluates against a missing field. `ticket.priority` is `undefined`/`null`, so the comparison is neither true nor false — it's unevaluable. The node can't pick a branch, so it emits a "condition could not be evaluated" stop rather than routing.
4. **Assign to tier 2 / tier 1 / End** → never reached.

**Exact stop point:** the If node on the `ticket.priority = 'high'` condition, caused by a missing field, not by a wrong value. Note the tree has no default/else-catch-all for *unevaluable* conditions — the Else branch only covers `priority ≠ 'high'` when a priority exists.

## No-code product rule

**Rule (stated as a contract):** *Every condition node must define a default branch that fires whenever the condition cannot be evaluated (missing field, null, type mismatch). No condition node may end a run without routing to exactly one branch.*

Concretely, in the tree builder UI:

- When a user adds an If node, require a **"Default (unevaluable)" branch** — e.g., "If priority = 'high' → tier 2; Else → tier 1; **If priority is missing/unset → tier 1**" (safe low-severity default for triage).
- Optionally pair it with a sibling no-code rule: **"Field default rule"** — any field referenced by a condition must have a declared fallback value at tree configuration time (e.g., `priority ?? 'normal'`), so the condition is always evaluable.

**Invariant check:**
1. **Productive work continues** — the ticket is always assigned (tier 1) instead of stopping; no human wake needed.
2. **Only real blockers stop** — a missing optional field is no longer a blocker; it's routed.
3. **No infinite loops** — the default branch routes once to a terminal End; no recovery/continuation loop is introduced.

**Phase 0 for the live ticket:** manually assign it to tier 1 (or set `priority = 'normal'` and re-run), and leave the original stop record intact for the audit trail.