Phase 4 Go-Live Runbook

Incident: Webhook delay

• Check payment_webhook_events_total and consumer lag metrics.
• Verify callback URLs and provider status pages.
• Run reconciliation cycle and inspect RECONCILIATION_PENDING_TIMEOUT transactions.
• If backlog grows, scale payment-service replicas and monitor queue drain.

Incident: Duplicate callback

• Confirm idempotency keys are being generated (webhook:{provider}:{eventId}:{hash}).
• Review transaction trail for duplicated provider event IDs.
• If duplicate callbacks still mutate state, disable provider endpoint via feature flag and patch.

Incident: Pending payment too long

• Trigger reconciliation job manually (or wait for cron every 10 minutes).
• Review stale PENDING payments older than 30 minutes.
• Move ambiguous cases to manual queue and notify support team.

Incident: ZNS failure spike

• Inspect notification_logs with status FAILED and error code.
• Verify ZNS_ACCESS_TOKEN, ZNS_TEMPLATE_ID, and template schema fields.
• Retry is automatic; for repeated failures, switch to fallback channel and open provider ticket.

Escalation

• P0 incidents: on-call backend + payment owner, response in <= 15 minutes.
• P1 incidents: service owner response in <= 60 minutes.