数据与分析
backup
Backup and disaster recovery. backup strategy, disaster recovery, RPO/RTO, data integrity, durability, runbook.
arbazkhan971 · 已有 AI 审核记录
安装与使用
运行命令前先阅读来源与依赖说明。复制命令不会自动安装或执行。
npx skills add arbazkhan971/godmode --skill backup技能说明
name: backup description: Backup and disaster recovery. backup strategy, disaster recovery, RPO/RTO, data integrity, durability, runbook.
Backup — Backup & Disaster Recovery
Activate When
- User invokes
/godmode:backup - User says "backup strategy," "disaster recovery," "what happens if we lose the database?"
- User is designing data infrastructure that needs durability guarantees
- After
/godmode:infraprovisions stateful services (databases, storage, queues) - Incident post-mortem reveals backup or recovery gaps
- Compliance requirements mandate backup and recovery documentation
Workflow
Step 1: Inventory Data Assets
Identify all data that needs protection:
DATA ASSET INVENTORY:
| Asset | Type | Size | Growth | Critical |
Step 2: Define RPO/RTO Targets
Set recovery objectives for each data tier:
RECOVERY OBJECTIVES:
| Data Tier | RPO | RTO | Justification |
Step 3: Backup Strategy Design
Design backup approach for each data tier:
Tier 1: Continuous Protection
TIER 1 BACKUP STRATEGY:
Primary database:
Tier 2: Periodic Snapshots
TIER 2 BACKUP STRATEGY:
File uploads (S3/GCS):
Tier 3: Daily Backups
TIER 3 BACKUP STRATEGY:
Application logs:
IF backup age >24h: trigger immediate backup. IF restore test fails: alert and re-run.
Step 4: Backup Verification
Automated checks that backups are actually working:
BACKUP VERIFICATION SCHEDULE:
| Check | Frequency | Method | Alert |
### Step 5: Data Integrity Verification
Verify backed-up data is consistent and usable:
DATA INTEGRITY CHECKS:
| Check | Method | Frequency |
|---|---|---|
| Row count consistency | Compare source vs | After each |
| restored backup | restore test | |
| Checksum verification | SHA-256 of backup | Every backup |
| file | ||
| Foreign key integrity | Run FK constraint | Weekly |
| check on restored DB | restore | |
| Application smoke test | Run app against | Monthly |
| restored DB | restore | |
| Point-in-time accuracy | Restore to specific | Quarterly |
| time, verify records | ||
| Cross-region consistency | Compare checksums | Weekly |
| across regions |
Integrity verification queries:
-- Row count comparison (run against source and restored)
SELECT table_name, n_live_tup
FROM pg_stat_user_tables
Step 6: Recovery Procedures
Document step-by-step recovery for each failure scenario:
RECOVERY: Primary Database Failure
Severity: CRITICAL
RPO: < 1 minute (via streaming replication)
RTO: < 15 minutes (via automated failover)
AUTOMATED RESPONSE:
1. Health check detects primary is unresponsive (30 seconds)
2. Orchestrator promotes synchronous replica to primary (15 seconds)
3. Application connection pool reconnects to new primary (30 seconds)
4. DNS/service discovery updated (60 seconds)
5. Alert sent to on-call engineer
6. Total automated recovery: < 3 minutes
MANUAL RESPONSE (if automated failover fails):
1. Verify primary is truly down (not a network partition)
$ pg_isready -h <primary-host>
2. Check replica lag
$ psql -h <replica-host> -c "SELECT pg_last_wal_replay_lsn();"
3. Promote replica manually
$ pg_ctl promote -D /var/lib/postgresql/data
4. Update connection string in application config
5. Restart application instances
6. Verify application is serving traffic
POST-RECOVERY:
1. Provision new replica from promoted primary
2. Verify replication is streaming
3. Update monitoring and alerting
4. Write incident report
Scenario 2: Data Corruption
RECOVERY: Data Corruption — Severity: HIGH, RTO: 15min-2h
1. STOP: Identify scope, disable source (bad migration, broken code)
2. RECOVER: Point-in-time restore (recent) | Snapshot restore (widespread) | Selective table restore (targeted)
3. VERIFY: Row counts, smoke tests, no orphaned records
4. POST-MORTEM: Root cause, prevention, detection/recovery speed
Scenario 3: Complete Region Failure
RECOVERY: Region Failure — Severity: CRITICAL, RPO: <1min, RTO: <30min
1. Detect failure (monitoring + cloud status page)
2. Activate DR: promote cross-region replica, verify storage, scale instances, update DNS (TTL 60s)
3. Verify DR serving traffic, communicate to stakeholders
RETURN TO PRIMARY: Wait 24h+, reverse replication, verify consistency, canary traffic shift
Step 7: Disaster Recovery Runbook
Generate a comprehensive runbook document:
DISASTER RECOVERY RUNBOOK
Last tested: <date>
Next test scheduled: <date>
Owner: <team/person>
On-call escalation: <contact chain>
Recovery objectives:
| Tier 1 RPO: < 1 min | RTO: < 15 min |
|--|--|
| Tier 2 RPO: < 1 hour | RTO: < 1 hour |
| Tier 3 RPO: < 24 hour | RTO: < 4 hours |
Scenarios covered:
1. Primary database failure → Page 3
2. Data corruption → Page 5
3. Complete region failure → Page 8
4. Ransomware / security breach → Page 11
5. Accidental data deletion → Page 13
6. Cloud provider outage → Page 15
Backup status:
Database: <HEALTHY | DEGRADED | FAILED>
File storage: <HEALTHY | DEGRADED | FAILED>
Configuration: <HEALTHY | DEGRADED | FAILED>
Secrets: <HEALTHY | DEGRADED | FAILED>
Last successful restore test: <date> (<result>)
Last DR failover test: <date> (<result>)
Step 8: Backup & DR Report
BACKUP & DISASTER RECOVERY REPORT
Data assets inventoried: <N>
Backup strategies defined: <N>
Recovery procedures documented: <N> scenarios
Coverage:
Tier 1 (critical): <PROTECTED | GAPS | UNPROTECTED>
Tier 2 (important): <PROTECTED | GAPS | UNPROTECTED>
Tier 3 (operational): <PROTECTED | GAPS | UNPROTECTED>
Verification:
Automated checks: <CONFIGURED | PARTIAL | MISSING>
Last restore test: <date | NEVER>
Last DR test: <date | NEVER>
Gaps identified: <N>
1. <gap description and remediation>
2. <gap description and remediation>
Verdict: <PROTECTED | PARTIAL | AT RISK>