Skip to content
Quarry Docs

Runbook: Incident response (brief)

A concise path for handling production incidents on quarry.team.

Severity

Section titled “Severity”
Sev Meaning Examples
SEV-1 Site down or data at risk Site unreachable; lead data loss/corruption; suspected breach or exposed secret
SEV-2 Major function broken Contact/careers forms failing; large SEO regression (redirects broken)
SEV-3 Degraded / minor Single page broken; slow but up; cosmetic regression

Roles

Section titled “Roles”
  • Incident lead — coordinates, decides, owns comms. First responder until handed off.
  • Responders — investigate and remediate.
  • Keep a running timeline (what was seen, what was done, when).

Flow

Section titled “Flow”
  1. Detect & declare. Note time, symptom, severity. Declare in the team incident channel.
  2. Assess blast radius. Prerendered pages/assets vs the on-demand form routes and D1 data tier (all on the one Cloudflare Worker) vs infra (Terraform/DNS/WAF). Static delivery and the dynamic form routes can fail independently.
  3. Stabilize. Prefer the fastest safe mitigation:
    • Bad site build → roll back to the previous Worker version (wrangler rollback).
    • Failing forms → roll back the Worker; verify Turnstile, secrets, CSP, and the D1 binding.
    • Infra/DNS/WAF/redirect issue → revert the infra change and terraform apply.
    • Data corruption → roll forward a corrective migration, or restore the form_submissions table via D1 Time Travel / wrangler d1 time-travel restore (treat as SEV-1).
    • See deploy.md → Rollback for exact steps.
  4. Verify. Re-run the post-deploy verification checklist (deploy.md).
  5. Communicate. Update stakeholders on status and ETA; confirm resolution.
  6. Post-incident review. For SEV-1/SEV-2, write a blameless review: timeline, root cause, what went well, action items (with owners). File follow-ups.

Security incidents

Section titled “Security incidents”
  • Suspected breach or exposed secret: treat as SEV-1. Rotate the affected credential immediately (provider secret stores / GitHub Actions secrets — never the repo), then assess exposure.
  • Report/coordinate per SECURITY.md; external disclosures go to security@quarry.team.

Key levers

Section titled “Key levers”
  • Cloudflare: Worker versions (wrangler rollback), WAF, DNS, cache purge, Web Analytics, Worker logs.
  • Cloudflare D1: Time Travel point-in-time restore (wrangler d1 time-travel), migrations.
  • GitHub Actions: re-run/redeploy, secret rotation.
  • Terraform: revert + apply to reconcile infrastructure (single Cloudflare provider).