# Resilience and durability: terms and practitioners

**Scope: evergreen.** These terms describe failure, recovery, ownership, and observable behavior. They do not prescribe a runtime, broker, database, or effect library. The cited practitioners use these terms or explain the underlying distinction; this is not a claim of coinage.

A worker that restarts and a system that recovers its obligations are different things. Ask what state survives, who owns recovery, and which repeated actions remain safe. Keep the answer at the boundary that owns the invariant.

Use one lens when one lens answers the problem. Do not turn this reference into a list every review must recite.

## Keep these concerns separate

| Braided concerns | What to separate | Source |
| --- | --- | --- |
| Execution and recovery evidence | A live worker versus the record needed to reconstruct its work | Gray and Tate, Big, Loud Worker-Bees; Jurić, Other Use Cases for ETS |
| Retrying and safe repetition | Repetition policy versus the observable effect of repeated work | Reis and Housley, Ch. 5 Message Queues; Bellemare, Ch. 7 |
| Failure containment and recovery | What a failure can damage versus who notices and repairs it | Logan, Merritt, and Carlsson, Let It Crash; Gray and Tate, Customize Your Lifecycle |
| Business refusal and broken execution | An expected outcome versus infrastructure failure or defect | Wlaschin, Implementation: Working with Errors |
| Shared capacity and critical capacity | A total limit versus capacity reserved for an independent workload | Nygard, §5.3 Bulkheads |
| Historical input and current conditions | Recorded decisions versus a fresh clock or external answer during replay | Bellemare, Ch. 6 Deterministic Stream Processing |
| Valid data and recoverable data | A readable representation versus a version that future consumers can restore | Bellemare, Ch. 3; Ambler and Sadalage, §3.11 |

## Terms

Every row below has `scope: evergreen`. Runtime-specific restart strategy names, queue APIs, and deployment mechanisms are deliberately absent.

| Term | One-line meaning | Practitioner | Book and chapter or section |
| --- | --- | --- | --- |
| Recovery to known state | Restore a state whose invariants are understood instead of continuing from suspected corruption. | Martin Logan, Eric Merritt, Richard Carlsson | *Erlang and OTP in Action*, The Erlang/OTP Platform: Let It Crash |
| Failure isolation | Limit the damage one failing subsystem can inflict on independently useful work. | James Edward Gray II, Bruce A. Tate | *Designing Elixir Systems with OTP*, Big, Loud Worker-Bees: Isolation |
| Recovery owner | The component responsible for observing worker failure and choosing what happens next. | Logan, Merritt, Carlsson | *Erlang and OTP in Action*, fault tolerance through links and supervisor restart strategy |
| Restart domain | The related workers whose recovery must follow their dependency relationships. | Saša Jurić | *Elixir in Action*, Third Edition, Ch. 9 Supervision Trees and Restart Strategies |
| Restart budget | A bound on repeated recovery attempts before the fault escalates. | Logan, Merritt, Carlsson | *Erlang and OTP in Action*, §4.2.2 The Supervisor Restart Strategy |
| Rehydration | Reconstruct the state needed by a restarted worker from retained data. | Gray, Tate | *Designing Elixir Systems with OTP*, Big, Loud Worker-Bees: timeout and crash-recovery discussion |
| Error kernel | The critical core whose state needs a deliberate preservation and recovery decision. | Jurić | *Elixir in Action*, Third Edition, Other Use Cases for ETS |
| Durable scheduled intent | A recorded deadline that can be recovered when its worker loses local timers. | Gray, Tate | *Designing Elixir Systems with OTP*, Big, Loud Worker-Bees: rehydrating scheduled work |
| Idempotency | Repeating an operation produces the same intended observable outcome as performing it once. | Joe Reis, Matt Housley | *Fundamentals of Data Engineering*, Part II, Ch. 5 Message Queues |
| Effectively-once boundary | A processing guarantee has a defined commit boundary and does not automatically cover external effects. | Adam Bellemare | *Building Event-Driven Microservices*, Ch. 7 Transactions and Effectively Once Processing |
| Deterministic processing | Live processing and catch-up produce the same output from the same relevant input. | Bellemare | *Building Event-Driven Microservices*, Ch. 6 Determinism with Event-Driven Workflows |
| Event-time policy | Domain rules choose event ordering and how late arrivals affect results. | Bellemare | *Building Event-Driven Microservices*, Ch. 6 Event Scheduling and Deterministic Processing |
| Clean replay target | Rebuilding starts from controlled state rather than mixing retained partial state with replay. | Bellemare | *Building Event-Driven Microservices*, Ch. 7 Restoring and Scaling from Input Event Streams |
| Schema compatibility | Consumers can still interpret the versions of retained records they need to process. | Bellemare | *Building Event-Driven Microservices*, Ch. 3 Backward Compatibility |
| Timeout | A finite bound on waiting for an answer that may never arrive. | Michael T. Nygard | *Release It!*, §5.1 Use Timeouts |
| Delayed retry | Defer a failing operation instead of extending the caller’s wait with immediate repetition. | Nygard | *Release It!*, §5.1 Use Timeouts |
| Randomized exponential backoff | Increasing delays with randomness avoid synchronized retry waves. | Mike Ulrich | *Site Reliability Engineering*, Ch. 22 Addressing Cascading Failures |
| Retry budget | A limit on retry load, potentially shared across calls rather than only per request. | Ulrich | *Site Reliability Engineering*, Ch. 22 Addressing Cascading Failures |
| Deadline propagation | Downstream work respects the caller’s remaining time rather than receiving a new full allowance. | Ulrich | *Site Reliability Engineering*, Ch. 22 Addressing Cascading Failures |
| Cancellation propagation | Downstream work learns that its caller no longer needs it. | Ulrich | *Site Reliability Engineering*, Ch. 22 Addressing Cascading Failures |
| Circuit breaker | Stateful policy stops attempts against an unhealthy operation and later probes recovery. | Nygard | *Release It!*, §5.2 Circuit Breaker |
| Bulkhead | Independent capacity protects one workload from exhaustion caused by another. | Nygard | *Release It!*, §5.3 Bulkheads |
| Resource pool | A bounded collection mediates access to scarce reusable resources. | Gray, Tate | *Designing Elixir Systems with OTP*, Build Pools of Common Resources |
| Backpressure | A slower consumer constrains how quickly producers can submit more work. | Gray, Tate | *Designing Elixir Systems with OTP*, Prefer Call Over Cast to Provide Back Pressure |
| Admission control | A service can refuse or defer new work when accepting it would worsen overload. | Nygard | *Release It!*, §5.6 Handshaking |
| Load shedding | Reject new work early during overload to preserve useful service. | Rhandeev Singh, Sebastian Kirsch, Vivek Rau | *Site Reliability Engineering*, Ch. 27 Reliable Product Launches at Scale |
| Steady state | Every accumulating resource has a mechanism that recycles or bounds it. | Nygard | *Release It!*, §5.4 Steady State |
| Result cardinality bound | A query limits how many results it asks a caller to process. | Nygard | *Release It!*, §4.4 Unbounded Result Sets |
| Functional core | Domain data and rules stay independent of external interfaces and process machinery. | Gray, Tate | *Designing Elixir Systems with OTP*, Build a Functional Core |
| Actions and calculations | Time-dependent interactions remain separate from deterministic transformations. | Eric Normand | *Grokking Simplicity*, Chs. 15–17, especially Coordinating Timelines |
| Timeline coupling | Shared resources make concurrent action order matter. | Normand | *Grokking Simplicity*, Ch. 17 Coordinating Timelines |
| Domain error | An expected business-process outcome belongs in the domain model. | Scott Wlaschin | *Domain Modeling Made Functional*, Implementation: Working with Errors |
| Applicative validation | Independent checks collect their failures instead of stopping at the first one. | Wlaschin | *Domain Modeling Made Functional*, Composing in Parallel with Applicatives |
| Idempotent postcondition | An already-satisfied desired state can be success rather than an avoidable error. | John Ousterhout | *A Philosophy of Software Design*, §10.3 Define Errors Out of Existence |
| Exception masking | The layer that can recover hides details its callers do not need. | Ousterhout | *A Philosophy of Software Design*, §10.6 Mask Exceptions; §10.10 Taking It Too Far |
| Exception aggregation | Failures with the same response policy reach one owning handler. | Ousterhout | *A Philosophy of Software Design*, §10.7 Exception Aggregation |
| Error promotion | Reuse a larger recovery mechanism when its cost and failure frequency make that simpler. | Ousterhout | *A Philosophy of Software Design*, §10.7 Exception Aggregation |
| Deep failure boundary | A small interface hides recovery mechanics without hiding caller-relevant failure information. | Ousterhout | *A Philosophy of Software Design*, §4.4 Deep Modules; §§10.6, 10.10 |
| Model-based testing | Generated command histories compare real outcomes with an explicit state model. | Fred Hebert | *Property-Based Testing with PropEr, Erlang, and Elixir*, Laying Out Stateful Properties |
| Precondition | A model rule identifies when a generated operation is valid. | Hebert | Same book, Laying Out Stateful Properties: The Validation |
| Postcondition | An observable result must satisfy the model after an operation. | Hebert | Same book, Laying Out Stateful Properties: The Validation |
| Shrinking | Reduce a failing input to a smaller reproducible counterexample while preserving validity. | Hebert | Same book, Ch. 7 Shrinking; State and Data |
| Fault injection | Introduce a known defect to check that the test detects the promised failure. | Hebert | Same book, stateful property failure-injection example |
| Hermetic test | Isolate a test from mutable external conditions to improve determinism and reproduction. | Titus Winters, Tom Manshreck, Hyrum Wright | *Software Engineering at Google*, CI Is Alerting |
| Four golden signals | Observe latency, traffic, errors, and saturation together. | Rob Ewaschuk | *Monitoring Distributed Systems*, The Four Golden Signals |
| Symptom-based paging | User-visible ongoing symptoms justify paging; internal causes support diagnosis. | Ewaschuk | *Monitoring Distributed Systems*, Black-Box Versus White-Box |
| Backfill | Recover missed source data after an outage using an explicit recovery plan. | Reis, Housley | *Fundamentals of Data Engineering*, Part II, Ch. 5 Incident Response |
| Transition period | Old and new schema forms coexist while consumers migrate. | Scott W. Ambler, Pramod J. Sadalage | *Refactoring Databases*, §3.11 What You Have Learned |
| Backsliding prevention | A migration prevents new dependencies on the interface being retired. | Winters, Manshreck, Wright | *Software Engineering at Google*, Ch. 15 Preventing Backsliding |

The generic labels recovery owner, restart domain, durable scheduled intent, clean replay target, and deep failure boundary are our glossary labels. The book and section identify the underlying principle, not a claim that the author coined that label.

## Practitioner references

- **Nygard: capacity under failure.** Use *Release It!* for timeouts, breakers, bulkheads, bounded results, and steady state. Ask which failing dependency can consume shared capacity.
- **Gray and Tate: core, boundary, worker, lifecycle.** Use *Designing Elixir Systems with OTP* to separate domain rules from process machinery. Ask what a worker restart actually reconstructs.
- **Logan, Merritt, Carlsson, and Jurić: failure ownership.** Use *Erlang and OTP in Action* and *Elixir in Action* for dependency-aware recovery. Keep the general principle separate from their runtime’s mechanisms.
- **Bellemare: replay and its commit boundary.** Use *Building Event-Driven Microservices* for stable replay, historical schemas, and processing guarantees. Ask which effects lie outside the guarantee.
- **Reis and Housley: repeated delivery and recovery gaps.** Use *Fundamentals of Data Engineering* for idempotency, source conformance, and backfill. Ask what repeated delivery or an outage leaves incomplete.
- **Ousterhout: remove caller burden.** Use *A Philosophy of Software Design*, Chapter 10, to define avoidable errors away and locate recovery once. Keep information the caller needs for correctness visible.
- **Wlaschin: failure as domain data.** Use *Domain Modeling Made Functional* to distinguish expected business failure, infrastructure failure, and panic. Ask whether independent validations should collect errors.
- **Normand: reduce interaction before coordinating it.** Use *Grokking Simplicity*, Chapters 15–17, to reduce shared resources and concurrent timelines. Ask what can become a calculation.
- **Hebert: test histories, then shrink them.** Use *Property-Based Testing with PropEr, Erlang, and Elixir* for model state, valid command sequences, postconditions, and useful counterexamples. Plant a fault before trusting a property.
- **Google SRE contributors: aggregate behavior.** Use *Site Reliability Engineering*, Chapters 22, 26, and 27, for retry load, deadlines, independent integrity checks, and overload. Use Ewaschuk for user-facing signals.
- **Ambler and Sadalage: shared schema transitions.** Use *Refactoring Databases* for the overlap between schema versions and consumer rollout. Ask who still needs the old representation.
- **Winters, Manshreck, and Wright: changes that converge.** Use *Software Engineering at Google* for hermetic tests, deprecation discovery, and backsliding prevention. Ask what prevents the old dependency from returning.

## Bibliographic limits

These references come from the named chapters and sections, not table-of-contents guesses. The inspected records do not consistently identify editions. Keep their section names when an edition-specific chapter number is unavailable. The Nygard findings use the 2007 text; do not silently attach second-edition chapter numbers.

DDIA replication, consensus, fencing, and Part III detail were not available as adequate passages. Newman’s detailed compensation argument also remained under-sourced. They are not used to fill this glossary from memory.