Platform Reliability Model¶

Executive Summary¶

• Reliability for a data platform is not uptime. It is data freshness, pipeline success, query availability, and recovery speed -- measured against explicit SLOs.
• Every data product must have a defined SLA. The platform must have SLOs that ensure those SLAs are achievable.
• Incident classification must distinguish between platform-wide failures, critical data product issues, and non-critical operational noise.
• Recovery is deterministic when source data is immutable. Bronze-layer preservation makes reprocessing a pattern, not a prayer.
• Post-incident process is not optional. Every P1 and P2 incident gets a blameless post-mortem with root cause classification, action items, and runbook updates.

graph TD
    DET[Detection<br/>Monitoring Alert / Consumer Report] --> CLS{Classify}
    CLS --> |P1: Platform-wide| P1[Immediate Response<br/>15 min SLA]
    CLS --> |P2: Critical product| P2[Urgent Response<br/>30 min SLA]
    CLS --> |P3: Non-critical| P3[Standard Response<br/>2 hour SLA]
    CLS --> |P4: Minor| P4[Planned Response<br/>Next sprint]

    P1 --> RES[Resolve]
    P2 --> RES
    P3 --> RES
    P4 --> RES

    RES --> PM[Post-Mortem<br/>within 48 hours]
    PM --> RB[Update Runbooks]
    RB --> DET

Platform SLOs¶

SLOs define what "reliable" means. Without them, reliability is a feeling, not a measurement.

Key principle: SLOs are not aspirations. They are commitments with error budgets. When the error budget is consumed, the team stops feature work and focuses on reliability. If leadership does not enforce this, SLOs are decorative.

SLO Measurement Rules¶

• Pipeline success rate excludes intentionally disabled pipelines. It includes retries -- a pipeline that fails twice and succeeds on the third attempt counts as one failure and one success.
• Data freshness is measured continuously, not at a point in time. A product that is within SLA for 23 hours and stale for 1 hour has a freshness breach.
• Query availability counts only platform-caused failures. A malformed user query that returns an error is not an availability failure.

Incident Classification¶

Escalation Rules¶

• P1 incidents trigger immediate page to on-call platform engineer and engineering manager. If no acknowledgment in 15 minutes, escalate to platform lead.
• P2 incidents trigger page to on-call platform engineer. If no acknowledgment in 30 minutes, escalate to engineering manager.
• P3 and P4 incidents are handled during business hours via the standard ticket queue.
• Any incident that is not resolved within its resolution window is automatically escalated one priority level.

Recovery Patterns¶

Recovery is not improvised. Each failure mode maps to a known recovery pattern.

Reprocessing¶

Replay from bronze. Source data in the landing zone is immutable, so recovery is deterministic -- the same input produces the same output. This is why bronze-layer immutability is a non-negotiable architectural principle.

Backfill¶

Re-ingest from source for a specific time window. Used when bronze data itself is missing or corrupted, or when a new source field must be historically populated.

Rollback¶

Revert to a previous version of a data product -- both schema and data. This is table-level time travel, not pipeline rollback.

Failover¶

Switch to a disaster recovery region or replica. Used for infrastructure-level failures, not data quality issues.

Dependency Management¶

A data platform does not exist in isolation. It depends on upstream source systems and serves downstream consumers. Both directions must be mapped and managed.

Upstream Dependencies¶

Downstream Dependencies¶

Circuit Breaker Pattern¶

Stop processing when upstream quality drops below an acceptable threshold. This prevents bad data from propagating through the platform.

• Trigger: Upstream data fails quality checks (null rate spike, volume anomaly, schema drift) beyond a configured threshold.
• Action: Halt downstream processing for that source, serve last-known-good data, alert data steward and source team.
• Reset: Manual or automatic after upstream quality is restored and validated. Never auto-reset without quality validation.
• Key principle: It is better to serve stale data with a freshness warning than to serve wrong data silently. The circuit breaker enforces this.

Post-Incident Process¶

Every P1 and P2 incident triggers a structured post-incident process. This is not optional, and it is not blame-assignment. It is organizational learning.

Blameless Post-Mortem¶

Conducted within 48 hours of incident resolution. Attendance includes the responders, the platform lead, and affected consumers.

Root Cause Classification¶

Every incident root cause is classified into one of four categories. This enables trend analysis across incidents.

Action Items¶

Every post-mortem produces action items. Every action item has an owner, a deadline, and a verification method.

• Runbook update: If this failure mode was not in the runbook, add it. If the runbook was wrong, fix it.
• Monitoring gap: If detection was slow, add the missing alert or dashboard.
• Architectural fix: If the failure was structural, schedule the fix with a deadline -- do not leave it as tech debt without a timeline.
• Process change: If the failure was procedural, update the process and communicate the change.

Key principle: A post-mortem without action items is a storytelling session. Action items without deadlines are wishes. Deadlines without owners are fiction.