Error Handling & Rollback

What this lesson covers

Bulk Operations built a runner that can recover — checkpoints, retries, idempotent transforms. This lesson is the other half: the operator judgment for when a run has actually gone wrong. Production automation rarely fails cleanly. It partially succeeds — 480 posts updated, 20 in an unknown state — and the money question is no longer "how do I catch exceptions" but "which posts are in which state, and do I resume, roll back, or fix forward?"

The failure taxonomy — classify before you react

The first move in any automation incident is not an action. It's a classification. Acting before you know what failed is how a recoverable situation becomes a corrupted one.

The dangerous quadrant is the last one: a transform that "succeeded" on every post but wrote the wrong thing. HTTP was happy; your content is corrupted. Only the post-run verification pass catches it — which is why Bulk Operations made that pass mandatory.

Retryable vs non-retryable — the decision that saves the run

Retries are only correct for failures that a later attempt could fix. Retrying the rest wastes time, burns rate limit, and buries the real error.

# Runner retries EVERYTHING on failure:
for attempt in range(4):
  try: patch(post_id)
  except: time.sleep(2 ** attempt)   # backoff on any error

# Post has a bad auth token → 401 on every attempt.
# Result: 4 slow retries per post × 500 posts of 401s,
# rate-limit tripped, the real problem (expired token)
# hidden under 2000 backoff sleeps. Run "fails" after 40 min.

RETRYABLE = ("429", "500", "502", "503", "504", "timeout")
def is_retryable(err: str) -> bool:
  return any(code in str(err) for code in RETRYABLE)

for attempt in range(4):
  try: patch(post_id); break
  except Exception as e:
      if not is_retryable(str(e)):
          log_fatal(post_id, e); break     # 401/404 → fail fast, surface it
      time.sleep(2 ** attempt)

# The 401 surfaces on post #1 in seconds. You refresh the
# token and resume. The run is fixed in 2 minutes, not 40.

The recovery decision tree — resume, roll back, or fix forward

When a batch half-completes, there are exactly three moves. Pick by asking two questions: were the applied changes correct? and is the operation idempotent?

Did the writes that succeeded produce CORRECT content?
├─ YES → the run just didn't finish
│        └─ RESUME from checkpoint (safe because the transform is idempotent).
│           Re-running touches only the unprocessed posts.
│
└─ NO → the transform itself was wrong
       ├─ Few posts affected, applied changes are wrong
       │   └─ ROLL BACK the affected posts to their backup, fix the
       │      transform, then re-run clean.
       │
       └─ Applied changes are fine but INCOMPLETE (e.g. added a block,
          now need a second field on the same posts)
           └─ FIX FORWARD: a new idempotent patch over the same set,
              NOT a rollback. Rolling back correct work wastes it.

Resume is the common case and the cheapest — it's why the runner checkpoints. Rollback is for when the change itself was wrong. Fix-forward is the one operators miss: if the applied changes are correct and you just need more, another idempotent patch is safer than undoing good work and redoing it.

Reading the audit log to drive recovery

The runner's JSONL log is the incident record. You don't guess which posts are in which state — you compute it. Every recovery starts by partitioning the target set into three groups.

import json
from pathlib import Path

def triage(target_ids: list[int], log_path: str):
  """Partition the target set using the run's JSONL log."""
  succeeded, failed = set(), {}
  for line in Path(log_path).read_text().splitlines():
      e = json.loads(line)
      if e["success"]:
          succeeded.add(e["post_id"])
      else:
          failed[e["post_id"]] = e.get("error", "unknown")

  never_attempted = [p for p in target_ids if p not in succeeded and p not in failed]

  print(f"succeeded:       {len(succeeded)}")
  print(f"failed:          {len(failed)}  {list(failed)[:10]}")
  print(f"never attempted: {len(never_attempted)}  {never_attempted[:10]}")
  return succeeded, failed, never_attempted

Now recovery is precise, not a full re-run:

• never_attempted → feed straight back into the runner; the checkpoint already skips succeeded.
• failed → look at the errors. All 429/timeout? A paced resume clears them. A 401? Fix auth first. A 422 on specific posts? Those posts have a data problem to fix individually.
• succeeded → leave them alone. The cardinal rule of partial-success recovery: re-run only the failed and unattempted items — never re-touch what already succeeded. This is exactly how the real bulk work recovered — the per-post run report was the source of truth for which posts still needed the operation, and only those were re-run.

Rollback done right

Rollback is not a panic button — it's its own bulk operation, and it can fail the same way the forward run did. Three rules keep it safe:

1. Roll back only the affected set. If a bad transform hit 20 posts, restore those 20 from backup — not the whole run. Restoring correct posts just risks corrupting them.

2. Verify after rollback. Restoring is a write; a write must be verified. Re-GET each rolled-back post and assert it matches the backup.

def rollback_and_verify(patcher, backup_dir, affected_ids):
  restored, mismatched = 0, []
  for pid in affected_ids:
      original = json.loads(Path(f"{backup_dir}/post_{pid}_backup.json").read_text())
      want = original["content"]["raw"]
      patcher.patch_post(
          pid,
          transform_fn=lambda p, w=want: {**p, "content": {"raw": w}},
          description=f"ROLLBACK post {pid}",
          dry_run=False,
      )
      # Verify: re-read and confirm the original content is back
      got = patcher.get_post(pid)["content"]["raw"]
      if got.strip() == want.strip():
          restored += 1
      else:
          mismatched.append(pid)
  print(f"rolled back: {restored}/{len(affected_ids)} verified")
  if mismatched:
      print(f"ROLLBACK MISMATCH on: {mismatched}")   # do NOT walk away — these are still wrong
  return mismatched

3. Rollback is idempotent by nature — writing the original content twice is harmless, so a rollback that itself gets interrupted is safe to re-run. That's the payoff of having backed up before the run: undo is always available, and always repeatable. A real large operation on this project — noindexing several hundred stale posts — shipped its rollback script in the same commit for exactly this reason: the decision was reversible with one command, verified.

The production incident workflow

When a run breaks — especially at 2am — the instinct is to re-run it immediately. That instinct corrupts data. Follow the sequence instead.

The difference between an operator and a beginner is not that the operator's runs never break — they do. It's that the operator's broken run is a five-minute procedure with a known-good end state, because the backup, the checkpoint, and the log were in place before the run started.