eval-harness

# Eval-Driven Development Plan: JWT Auth for Flask

## Step 0: Baseline (before writing any code)

Capture current behavior so you can prove nothing regresses:

```python
# tests/eval_baseline.py
import pytest
from app import app

@pytest.fixture
def client():
    app.config['TESTING'] = True
    return app.test_client()

def test_baseline_login_correct_password(client):
    r = client.post('/login', data={'password': 'secret'})
    assert r.status_code == 200

def test_baseline_login_wrong_password(client):
    r = client.post('/login', data={'password': 'wrong'})
    assert r.status_code in (401, 403)  # record actual behavior
```

Run `pytest tests/eval_baseline.py` and **record the actual outputs** (status codes, response bodies) — this is your ground truth.

## Capability Evals (new behavior — write these first, they should FAIL)

**1. Token issuance**
```python
def test_login_returns_token(client):
    r = client.post('/login', data={'password': 'secret'})
    assert r.status_code == 200
    body = r.get_json()
    assert 'token' in body
    assert body['token'].count('.') == 2  # JWT structure: header.payload.signature
```

**2. Token verification on protected route**
```python
def test_protected_route_with_valid_token(client):
    token = client.post('/login', data={'password': 'secret'}).get_json()['token']
    r = client.get('/protected', headers={'Authorization': f'Bearer {token}'})
    assert r.status_code == 200

def test_protected_route_without_token(client):
    assert client.get('/protected').status_code == 401
```

**3. Token claims correctness**
```python
import jwt, time

def test_token_claims(client):
    token = client.post('/login', data={'password': 'secret'}).get_json()['token']
    claims = jwt.decode(token, options={'verify_signature': False})
    assert claims['exp'] > time.time()
    assert 'iat' in claims
```

**4. Expiry behavior**
```python
def test_expired_token_rejected(client, monkeypatch):
    # issue token, then freeze time past exp, expect 401
```

**5. Tampered/invalid tokens**
```python
@pytest.mark.parametrize("bad", ["garbage", "a.b.c", valid_token[:-2] + "xx"])
def test_invalid_tokens_rejected(client, bad):
    r = client.get('/protected', headers={'Authorization': f'Bearer {bad}'})
    assert r.status_code == 401
```

**6. Wrong secret / wrong signing key** — token signed with a different key must be rejected.

## Regression Evals (existing behavior must not change)

- `test_baseline_login_correct_password` — still 200 (or whatever you decide the new contract is; if the response shape changes, update this eval *deliberately*, not accidentally)
- `test_baseline_login_wrong_password` — still rejected with same status code
- Missing `password` field in form → same error behavior as today
- Any other existing routes/endpoints: add smoke tests hitting each one

## Security-specific evals (worth adding for auth code)

- Password comparison is not vulnerable to timing/`==` on secrets (consider `hmac.compare_digest`)
- Token doesn't leak in logs or error responses
- `alg: none` / `alg` confusion attacks rejected (decode with algorithms pinned to `HS256`/`RS256`)
- No secret material hardcoded (assert `app.config['SECRET_KEY']` comes from env)

## How to run them before coding

1. Write all evals above in `tests/`.
2. Run `pytest tests/ -v` — capability evals should **all fail**, regression evals should **all pass**. If a capability eval accidentally passes, it's testing the wrong thing — fix it.
3. Commit the failing suite as your spec.
4. Implement JWT issuance/verification.
5. Iterate: `pytest tests/` until green. Never delete a failing eval to make the suite pass — fix the code or consciously revise the eval with a note.

## Suggested layout

```
tests/
  eval_baseline.py      # regression: current behavior
  eval_capability.py    # new JWT behavior (fails initially)
  eval_security.py      # adversarial cases
  conftest.py           # client fixture, token helpers
```

This gives you a red→green workflow where the eval suite *is* the specification for the feature.

## EVAL DEFINITION: jwt-authentication

Save as `.claude/evals/jwt-authentication.md`.

### Capability Evals (new behavior — must pass before merge)

All graded with code-based graders (pytest + Flask test client) — deterministic, fast.

1. **token-issuance-valid-credentials**
   - Task: `POST /login` with correct password returns a JWT.
   - Criteria:
     - [ ] Response 200 with `access_token` in JSON body
     - [ ] Token decodes with app's secret key; `exp` claim present and in the future
   - Grader: `pytest tests/test_jwt.py::test_login_issues_token`

2. **token-verification-protected-route**
   - Task: A protected endpoint accepts `Authorization: Bearer <token>`.
   - Criteria:
     - [ ] Valid token → 200
     - [ ] Missing header → 401
     - [ ] Malformed token → 401
     - [ ] Expired token → 401
     - [ ] Token signed with wrong key → 401
   - Grader: pytest parametrized test

3. **token-payload-claims**
   - Criteria:
     - [ ] Token contains `sub` (user identity), `iat`, `exp`
     - [ ] Expiry configurable (e.g., 15 min default), not hard-coded forever

4. **invalid-credentials-rejected**
   - Criteria:
     - [ ] Wrong password → 401, no token issued
     - [ ] Response does not leak whether user vs password was wrong (if applicable)

5. **security-hardening** (⚠ human review required — never fully automate security)
   - [ ] Secret key loaded from config/env, not hard-coded
   - [ ] Passwords not compared in plaintext (hashing planned or noted)
   - [ ] `'secret'` literal removed from source
   - [ ] No token in URL/query strings; logs don't contain tokens

### Regression Evals (existing behavior must not break)

Baseline: current `main` SHA (record it in `.claude/evals/baseline.json`).

1. **legacy-login-path**: `POST /login` with correct password still returns 200 (even if response shape changes, status stays 200)
2. **wrong-password-still-401**: incorrect password → 401 as before
3. **public-routes-unchanged**: all currently public routes remain accessible without a token
4. **existing-test-suite-green**: `pytest` (full suite) passes; no new failures vs baseline
5. **api-contract**: any endpoint response schemas unchanged except `/login` (documented diff)

### Metrics

- Capability: pass@3 > 90% (retry up to 3 attempts per eval)
- Regression: pass^3 = 100% (every run, every trial — zero tolerance)

### How to run before coding

```bash
# 1. Record baseline
git rev-parse HEAD > .claude/evals/baseline.json

# 2. Write the eval tests FIRST (they will fail — that's expected)
pytest tests/test_jwt.py          # capability evals: FAIL (red)
pytest tests/test_regression.py   # regression evals: PASS (green baseline)

# 3. Implement JWT auth, then re-run
pytest tests/test_jwt.py tests/test_regression.py

# 4. Generate report
```

Report format after implementation:

```markdown
EVAL REPORT: jwt-authentication
Capability: 5/5 (pass@3: 100%)
Regression: 5/5 (pass^3: 100%)
Human review: security-hardening — PENDING
Status: READY FOR REVIEW
```

Key principle: the capability evals are red before you write code — that's the point. The regression suite is green before and must stay green after.

# Eval Report — URL Shortener Feature

## Capability Evals

| Eval | Results | Pass@1 | Pass@5 |
|------|---------|--------|--------|
| cap1 | T,F,T,T,T | 0.80 | 1.00 |
| cap2 | F,F,F,F,F | 0.00 | 0.00 |
| cap3 | T,T,T,T,T | 1.00 | 1.00 |
| cap4 | F,T,T,T,T | 0.80 | 1.00 |
| cap5 | T,F,F,F,F | 0.20 | 0.00 |

**Aggregate:** pass@1 = 14/25 = **0.56** · pass@5 = 2/5 = **0.40**

## Regression Evals

| Eval | Results | Pass@1 | Pass@5 |
|------|---------|--------|--------|
| reg1–reg4 | T,T,T,T,T | 1.00 | 1.00 |

**Aggregate:** pass@1 = **1.00** · pass@5 = **1.00** ✅ No regressions.

## Evals Needing Attention

1. **cap2 — critical.** 0/5 passes. The capability is completely broken; investigate first.
2. **cap5 — high priority.** Only 1/5 passes (pass@1 = 0.20). Highly flaky/unreliable; likely a real bug or unstable behavior, not noise.
3. **cap1 & cap4 — moderate.** 4/5 each (pass@1 = 0.80). One flaky failure apiece — could be nondeterminism, timing, or edge-case handling. Worth a retry-and-triage pass.

cap3 and all regression evals are healthy.

# EVAL REPORT: URL Shortener Feature

## Capability Evals

| Eval | Trials | Pass@1 | Pass@5 | Status |
|------|--------|--------|--------|--------|
| cap1 | T,F,T,T,T | 80% (4/5) | ✅ 100% | Needs attention (flaky) |
| cap2 | F,F,F,F,F | 0% (0/5) | ❌ 0% | **Failing** |
| cap3 | T,T,T,T,T | 100% (5/5) | ✅ 100% | Healthy |
| cap4 | F,T,T,T,T | 80% (4/5) | ✅ 100% | Needs attention (flaky) |
| cap5 | T,F,F,F,F | 20% (1/5) | ✅ 100% | **Failing** |

## Regression Evals

| Eval | Trials | Pass@1 | Pass^5 | Status |
|------|--------|--------|--------|--------|
| reg1–reg4 | T,T,T,T,T each | 100% (4/4) | 100% (4/4) | Healthy |

## Summary Metrics

- **Capability pass@1 (avg): 56%** (9/16 trials)
- **Capability pass@5: 80%** (4/5 evals pass at least once)
- **Regression pass@1: 100%** — no regressions introduced ✅

## Evals Needing Attention

1. **cap2 — Critical.** 0/5 passes. Completely broken; the feature or eval is fundamentally failing. Investigate first.
2. **cap5 — High.** Only 1/5 passes. Unreliable implementation; pass@5 masks this, but pass@1 of 20% means users will usually hit failure on first attempt.
3. **cap1 & cap4 — Medium.** 4/5 each (flaky). Likely nondeterminism, race conditions, or edge cases in the shortener (e.g., URL validation, collision handling). Worth stabilizing before trusting the feature.

**Overall status: NOT READY** — fix cap2 and cap5, then re-run to confirm cap1/cap4 stabilize. Regression suite is clean, so the failures are isolated to new capability behavior.

No — don't trust this change. It's not an optimization; it's a behavior change.

**Why it fails:** `set(items)` loses insertion order (sets are unordered). `dict.fromkeys(items)` preserves the order of first occurrences, which is exactly what your regression eval checks. Your 3 capability evals pass because they only test that duplicates are removed — they don't catch the ordering regression.

**What to do:**
1. **Reject the change.** The regression eval is doing its job.
2. If you want a real speedup, keep order-preserving semantics. Options:
   - Keep `dict.fromkeys` (it's already fast — O(n), C-implemented).
   - Only use `set` when order doesn't matter, e.g. add a separate `deduplicate_unordered` for callers that don't care.
3. Consider strengthening the suite: if order preservation is part of the contract, it arguably deserves a capability eval too, not just a regression guard.

Rule of thumb: any "optimization" that changes observable output is a bug, not a speedup.

**No — don't trust this change.** The failing regression eval caught a real behavioral break, not a fluke.

## What happened

- Old: `dict.fromkeys(items)` — deduplicates **and preserves insertion order** (first-occurrence order).
- New: `set(items)` — deduplicates but **loses ordering entirely** (set iteration order is arbitrary and depends on hash values).

Your regression eval is doing exactly its job: it encoded a guaranteed behavior of the old version, and the new version violates it. Capability evals passing doesn't matter here — they test new abilities, not preserved ones. Under EDD, regression evals should be treated as a hard gate (pass^k = 100%).

## What to do

1. **Revert or fix.** If order preservation is a contract of `deduplicate`, keep the original implementation. If you need speed, a correct fast version is:
   ```python
   def deduplicate(items):
       return list(dict.fromkeys(items))  # already O(n) — this IS the fast one
   ```
   Note: `dict.fromkeys` is already O(n), so the "optimization" likely gains nothing anyway.

2. **If order genuinely doesn't matter** for your use case, then the regression eval is wrong, not the code. In that case:
   - Confirm with callers/docs that ordering was never a requirement.
   - Update or delete the regression eval **deliberately**, with a note in the eval log explaining why — never silently delete a failing eval to make the suite green.

3. **Record the run** in your eval history (e.g., `.claude/evals/deduplicate.log`): capability 3/3, regression 0/1, status: BLOCKED until resolved.

Bottom line: a failing regression eval means the change isn't shippable as-is. Either fix the code or formally retire the requirement — don't ship with a red eval.

# Eval Harness Design: NL-to-SQL CLI

## 1. Config (updated)

```yaml
evals:
  capability: [simple_select, join_query, group_by]
  regression: [legacy_schema]

metrics:
  - exact_match      # normalized SQL text equality
  - execution_match  # result-set equality on a fixture DB
  - latency          # p50/p95 ms per query

trust_threshold:
  pass@3: 0.90        # capability evals
  pass^3: 1.00        # regression evals (all 3 consecutive runs)
  execution_match: 0.95   # primary correctness metric
  exact_match: 0.80       # secondary (style variance tolerated)
  latency_p95_ms: 2000
```

## 2. Eval Categories

| Category | Type | Current | Target | Grader |
|---|---|---|---|---|
| simple_select | capability | 10/10 | pass@3 ≥ 90% | execution_match |
| join_query | capability | 8/10 | pass@3 ≥ 90% | execution_match + exact_match |
| group_by | capability | 6/10 | pass@3 ≥ 90% | execution_match (exact_match too strict for GROUP BY ordering/aliasing) |
| legacy_schema | regression | 10/10 | pass^3 = 100% | execution_match |

**Key change:** make `execution_match` the primary metric. `exact_match` fails on semantically-correct SQL (column order, alias names, implicit vs explicit joins) — that's likely why group_by scores 6/10. Re-score those failures with execution_match before assuming the model is weak.

## 3. pass@k Calculation

Run each eval case k=3 times (fresh sessions, temperature > 0):

```
pass@k = 1 - C(n-c, k) / C(n, k)     # unbiased estimator
```
where n = trials, c = successes. For small k, approximate: pass@3 = fraction of cases with ≥1 success in 3 trials.

Also track **pass^3** (all 3 trials succeed) for regression suites — reliability, not just capability.

Current state (assuming 10 cases each, 1 trial shown):
- simple_select: pass@1 = 1.00 → trust ✅
- join_query: pass@1 = 0.80 → below 0.90 threshold → **not trusted**, needs 3-trial runs to see if pass@3 recovers
- group_by: pass@1 = 0.60 → **failing**; diagnose with execution_match first

## 4. Trust Threshold Policy

A category is **trusted** only when:
1. pass@3 ≥ 0.90 (capability) or pass^3 = 1.00 (regression)
2. execution_match ≥ 0.95
3. latency p95 ≤ 2000ms

Untrusted categories gate merges: any change touching SQL generation requires the failing category to improve or stay flat; a regression suite failure blocks the change outright.

## 5. Flakiness Policy

- **Detection:** run each case 3×; flag cases with mixed results (e.g., 1/3, 2/3) as `flaky`.
- **Quarantine:** move flaky cases to a `flaky` list; they don't count toward pass/fail until resolved, but are reported separately.
- **Triage:** for each flaky case, classify cause:
  - *Nondeterministic generation* → lower temperature or pin seed for evals
  - *Nondeterministic grading* (row order in unordered queries) → sort result sets before comparison; compare as multisets
  - *Environment* (DB state, timing) → fixture reset between runs; use p50 not mean for latency
- **Re-baselining:** a case flaky in ≥2 consecutive runs gets rewritten or removed with a changelog entry — never silently deleted.
- **Latency flakiness:** treat latency as a distribution; alert only when p95 exceeds threshold in 2 of 3 runs.

## 6. Run Report Format

```
EVAL REPORT: nl2sql-cli @ <commit>
Category        pass@1  pass@3  exec_match  exact_match  p95(ms)  Status
simple_select   1.00    1.00    1.00        1.00         340      TRUSTED
join_query      0.80    0.93    0.90        0.80         610      TRUSTED (pass@3)
group_by        0.60    0.87    0.80        0.60         520      NOT TRUSTED
legacy_schema   1.00    1.00^3  1.00        1.00         290      TRUSTED
Flaky: 1 case (group_by#7 — unordered rows, grader fixed)
Status: BLOCKED — group_by below threshold
```

**Immediate next steps:** (1) add execution_match grader with multiset comparison, (2) run 3-trial pass@3 baseline for join_query and group_by, (3) quarantine and triage flaky cases, (4) enforce the threshold gate in CI.

**Per-case pass@k** (unbiased estimator: 1 − C(n−c, k)/C(n, k)):

| Case | Passes | pass@1 | pass@3 | pass@5 |
|------|--------|--------|--------|--------|
| 1 | 4/5 | 0.80 | 1.00 | 1.00 |
| 2 | 5/5 | 1.00 | 1.00 | 1.00 |
| 3 | 0/5 | 0.00 | 0.00 | 0.00 |

**Suite averages:**
- pass@1 = (0.8 + 1 + 0)/3 = **0.60**
- pass@3 = (1 + 1 + 0)/3 = **0.667**
- pass@5 = (1 + 1 + 0)/3 = **0.667**

**Ship decision:** pass@5 = 0.667 < 0.8, so **not ready to ship** under your criterion. Case 3 fails deterministically (0/5), which suggests a real bug in `slugify` rather than flakiness — fixing that case would bring pass@5 to 1.0.

## Results

**pass@1** (first attempt success rate): case1 T, case2 T, case3 F → **2/3 ≈ 0.67**

**pass@3** (at least one success in first 3 attempts): case1 ✓ (T,T,F), case2 ✓, case3 ✗ (all F) → **2/3 ≈ 0.67**

**pass@5** (at least one success in all 5 attempts): case1 ✓, case2 ✓, case3 ✗ (F F F F F) → **2/3 ≈ 0.67**

## Ship decision

**Not ready.** pass@5 = 0.67 < your 0.8 threshold. Case 3 fails deterministically (0/5), so no amount of retries will help — this is a real bug in `slugify` for that input, not flakiness. (Case 1's single failure looks like flakiness worth noting, but case 3 is the blocker.) Fix case 3 and re-run.