# durability

Design or review crash recovery, retries, duplicate delivery, overload, worker ownership, replay, and durable timers in Effect applications. Use when work can outlive a request, a worker loses progress, retries repeat writes, or a queue grows under failure.

- [Raw SKILL.md](/skills/durability/SKILL.md)
- [references/bibliography.md](/skills/durability/references/bibliography) ([raw](/skills/durability/references/bibliography.md))
- [references/effect-4.0.5.md](/skills/durability/references/effect-4.0.5) ([raw](/skills/durability/references/effect-4.0.5.md))
- [references/request-budget.md](/skills/durability/references/request-budget) ([raw](/skills/durability/references/request-budget.md))

## Install

```sh
mkdir -p ~/.claude/skills/durability
curl -fsSL https://uncomplect.com/skills/durability/SKILL.md -o ~/.claude/skills/durability/SKILL.md
mkdir -p "$(dirname ~/.claude/skills/durability/references/bibliography.md)"
curl -fsSL https://uncomplect.com/skills/durability/references/bibliography.md -o ~/.claude/skills/durability/references/bibliography.md
mkdir -p "$(dirname ~/.claude/skills/durability/references/effect-4.0.5.md)"
curl -fsSL https://uncomplect.com/skills/durability/references/effect-4.0.5.md -o ~/.claude/skills/durability/references/effect-4.0.5.md
mkdir -p "$(dirname ~/.claude/skills/durability/references/request-budget.md)"
curl -fsSL https://uncomplect.com/skills/durability/references/request-budget.md -o ~/.claude/skills/durability/references/request-budget.md
```

---

# Durability

**Effect is not BEAM.** Durability lives in storage. Fibers are concurrent computations in one JavaScript runtime, not isolated BEAM processes. No OTP restart supervisor was found in the checked Effect 4.0.5 source. Its structured fiber ownership is useful, but does not reconstruct lost state.

> Remember from chapters 8 and 9 that processes lose their state on termination.

Saša Jurić, *Elixir in Action*, Other Use Cases for ETS ([B2], Q02). Even OTP restart needs a separate state-recovery decision.

Use this skill when accepting work, calling an unreliable dependency, starting workers, changing persisted schemas, or reviewing recovery. For ordinary pure functions, use [uncomplect](https://uncomplect.com/skills/uncomplect). For finite modes, use [add a lifecycle machine](https://ratstack.sh/skills/add-a-lifecycle-machine).

## Read the actual boundary

Read the project's instructions, pin, service, storage adapter, acknowledgement path, and restoration tests. Check Effect APIs against that exact source. The mappings here are **our reading of Effect 4.0.5**, not the book authors' recommendations about Effect. They do not authorize a dependency upgrade. The rat-stack template declares Effect 4.0.0; that version was not audited here. Do not generalize 4.0.5 API availability or negative searches to 4.0.0.

For each accepted operation, identify:

- the durable record that remains after the process disappears;
- the authority that decides its next step;
- the operation identity used by every repeated side effect;
- the failure domain and the recovery owner;
- the acknowledgement point and the outcome the caller can truthfully infer.

If no durable record exists, describe the work as best-effort. A running fiber, an in-memory queue, or a machine snapshot held only in memory is not that record.

## Principles, ranked

Rank by risk of lost work, repeated effects, and cascading failure. Numbers are this skill's priorities, not a ranking from the books.

In the table, source paths are relative to `packages/effect/src/` at **effect@4.0.5**. [The source map](/skills/durability/references/effect-4.0.5.md) gives the exact revision, imports, evidence, and limits.

| Rank | Principle and action | Failure prevented | Effect answer and remaining work | Book evidence |
| --- | --- | --- | --- | --- |
| 1 | Persist accepted intent and recovery progress. Name the backend. | A healthy restarted worker silently loses accepted work. | `effect/workflow`: `Workflow`, `Activity`, `WorkflowEngine` (`workflow/*.ts`). Select persistent engine/storage; `WorkflowEngine.layerMemory` is not crash-durable. | [B1], [B2], [B3] |
| 2 | Make repeated external effects safe before enabling retries. | A crash after remote success repeats a write. | `Workflow` identity and `Activity.idempotencyKey` (`workflow/Workflow.ts`, `workflow/Activity.ts`) help name work. The receiver or authoritative database must enforce deduplication. | [B5], [B6] |
| 3 | Couple state and checkpoint where one transaction can own both. Acknowledge only the completed handoff or processing promised by the contract. | A checkpoint advances past missing work, or replay reapplies committed changes. | `SqlClient.withTransaction` (`sql/SqlClient.ts`); broker acknowledgement remains adapter policy. No atomic transaction across arbitrary providers. | [B5], Chs. 7, 9 |
| 4 | Give every task a lifetime owner and every failed worker a recovery owner. | Escaped tasks, unnoticed death, or endless restart churn. | `Effect.forkScoped`, `Scope`, `FiberSet` (`Effect.ts`, `Scope.ts`, `FiberSet.ts`). Add dependency-aware restart policy in XState or a host supervisor. | [B1], [B2], [B3] |
| 5 | Bound waiting and carry the caller's remaining deadline. | Abandoned calls keep consuming scarce capacity. | `Effect.timeout` / `timeoutOption` (`Effect.ts`). A timeout is not proof that a remote write stopped. Carry deadline and cancellation in the adapter protocol. | [B4] §5.1; [B12] Ch. 22 |
| 6 | Put retries at one informed boundary, with jitter and a finite budget. | Nested loops and synchronized retries amplify an outage. | `Effect.retry`; `Schedule.exponential`, `jittered`, `recurs`, `upTo` (`Effect.ts`, `Schedule.ts`). Shared retry accounting is separate policy. | [B4] §5.1; [B9]; [B10] §10.6; [B12] Ch. 22 |
| 7 | Bound admitted work and isolate scarce capacity by workload. | A noisy caller fills every slot and backlog grows without limit. | Bounded `Effect.forEach`, `Semaphore`, `Queue.bounded` (`Effect.ts`, `Semaphore.ts`, `Queue.ts`). Use `Pool` for an actual resource pool. Admission and upstream cooperation remain protocol work. | [B3]; [B4] §§5.3, 5.6 |
| 8 | Stop calling a known-unhealthy dependency and probe recovery explicitly. | Retrying doomed calls exhausts capacity needed for recovery. | Timeout and typed errors are building blocks. No dependency circuit-breaker API was found. Model closed/open/half-open policy in XState. | [B4] §5.2; [B11] Adjusting the Model |
| 9 | Recover from known state, with stable replay inputs and recoverable deadlines. | Restart changes old decisions or forgets scheduled work. | `Activity`, `DurableClock`, `DurableDeferred` (`workflow/*.ts`). Persist decisions, operation names, and timers under the selected engine or storage owner. | [B1]; [B3]; [B5] Ch. 6 |
| 10 | Preserve old payloads' meaning across deployments. | Old accepted work becomes undecodable during recovery. | `Schema.decodeUnknownEffect` (`Schema.ts`) supplies decoding. Define version support and transition tests; the decoder does not invent migrations. | [B5] Ch. 3; [B14] §3.11 |
| 11 | Park exhausted or malformed work durably and provide a return path. | Poison work disappears, retries forever, or leaves an invisible parked workflow. | `PersistedQueue`, `DurableQueue` (`persistence/PersistedQueue.ts`, `workflow/DurableQueue.ts`) have failure records. Redrive, operator visibility, and repair remain application work. | [B5] Ch. 9; [B9] |
| 12 | Keep expected refusals, transient faults, and defects distinct. | A crash becomes permanent refusal, or permanent refusal is retried. | `Schema.TaggedError`, `Effect.catchTag` (`Schema.ts`, `Effect.ts`). Expose information callers need; do not turn every failure into success. | [B7]; [B10] Ch. 10 |
| 13 | Recycle accumulation and bound results at their source. | Retained records or a huge query exhaust storage or memory despite concurrency limits. | `Cache`, `Scope`, `Stream.take`, `PersistedQueue.layerCleanup` (`Cache.ts`, `Scope.ts`, `Stream.ts`, `persistence/PersistedQueue.ts`). Add storage retention and query limits. | [B4] §§4.4, 5.4 |
| 14 | Prove recovery with deterministic histories and bad dependencies. | A design passes happy-path tests but loses work at the commit/ack gap. | `TestClock`, `Deferred`, `Arbitrary.checkEffect` (`testing/TestClock.ts`, `Deferred.ts`, `Arbitrary.ts`). Write the model, inject faults, and test real storage recovery separately. | [B4] §5.7; [B11]; [B15] |
| 15 | Observe useful outcomes, backlog, and saturation. | Fast failures, parked work, or growing queues appear healthy. | `Metric.counter`, `gauge`, `histogram` (`Metric.ts`) provide instruments. Dashboards, integrity checks, recovery alerts, and ownership are outside the constructors. | [B8]; [B12] Ch. 26 |

> Protect critical clients by giving them their own pool to call.

Michael T. Nygard, *Release It!*, §5.3 ([B4], Q07). One global limit does not reserve capacity for critical work.

## Choose what owns recovery

This is house composition, not a claim that the authors recommend these products.

| Need | Use | Do not claim |
| --- | --- | --- |
| Process-local resource and child lifetime | Effect Scope, resource acquisition/release, scoped fibers | Cleanup after hard process death, independent heaps, or automatic child restart |
| Post-response Worker work | `ctx.waitUntil` for bounded best-effort work; persist required work before responding | More than the shared post-response allowance or recovery after cancellation |
| Legal modes, retry/cancel events, restart thresholds | XState 6 with `@xstate/effect`, behind the project's checked integration | A machine alone is a durable store or an OS supervisor |
| Existing per-entity durable state and alarms | A Cloudflare Durable Object adapter behind an Effect service | An in-memory object field or local timer survives restart; verify storage/alarm recovery |
| Authoritative relational state, dedupe ledger, checkpoint, or outbox | A Postgres adapter with explicit transaction and uniqueness contracts | A local transaction includes a remote provider's side effect |
| Durable multi-step orchestration without an existing recovery owner | Effect workflow engine plus its persistent storage/cluster configuration | `layerMemory`, a detached fiber, or an API named Durable proves production durability |
| Heap, CPU, or fatal-runtime isolation | A separate process/worker/service and a host recovery policy | A Scope, FiberSet, or state machine provides BEAM isolation |

Keep one recovery authority for each operation. When an existing Durable Object or database already owns progress, adapt it; do not add a second workflow ledger by default. The source map distinguishes verified primitives from these design choices.

`Effect.forkDetach` is the 4.0.5 detached-fiber name. `Effect.forkDaemon` is not a public export in this pin. Detaching changes lifetime within a live runtime; it does not persist work. On HTTP-triggered Cloudflare Workers, `ctx.waitUntil` extends execution for up to 30 seconds after the response or disconnect. All calls in one request share that allowance. Unsettled work can be canceled; this is not durable execution. See the [worked request-budget example](/skills/durability/references/request-budget.md).

## Check the uncomfortable boundaries

Use the contract's promised guarantee, not a generic claim of exactly-once execution.

| Boundary | Evidence to collect |
| --- | --- |
| Accepted, not yet started | Kill the worker; accepted identity remains discoverable in the durable store. |
| Remote effect succeeded, local result missing | Retry the same identity; show receiver dedupe or reconciliation. If neither exists, record the outcome as uncertain. |
| State committed, acknowledgement missing | Redeliver; prove state does not apply twice. |
| Retry exhausted | Show a durable failure record, visible owner, and tested redrive. With DurableQueue, show how the parked waiter resumes. |
| Timer pending at crash | Recover the persisted deadline; test wakeup and cancellation semantics. |
| Old payload after deploy | Decode retained fixtures and prove restoration across versions. |
| Dependency slow, absent, or malformed | Check deadline, admission, typed failure, breaker transition, and capacity release. |

These are our proposed verification cases derived from the cited failure modes. Passing an in-memory test does not prove the backend's crash contract.

## Finish

Report the persistence owner, recovery owner, operation identity, acknowledgement boundary, remaining uncertainty, and checks run. Name what the design deletes. Leave a missing guarantee explicit.

Read [the durability index](https://uncomplect.com/principles/durability) for individual patterns, [the bibliography](/skills/durability/references/bibliography.md) for books and verified quotations, and [the Effect map](/skills/durability/references/effect-4.0.5.md) for the pin's actual APIs. Use [investigate a failure](https://ratstack.sh/skills/investigate-a-failure) for a live fault.

[B1]: /skills/durability/references/bibliography.md#b1-erlang-and-otp-in-action
[B2]: /skills/durability/references/bibliography.md#b2-elixir-in-action
[B3]: /skills/durability/references/bibliography.md#b3-designing-elixir-systems-with-otp
[B4]: /skills/durability/references/bibliography.md#b4-release-it
[B5]: /skills/durability/references/bibliography.md#b5-building-event-driven-microservices
[B6]: /skills/durability/references/bibliography.md#b6-fundamentals-of-data-engineering
[B7]: /skills/durability/references/bibliography.md#b7-domain-modeling-made-functional
[B8]: /skills/durability/references/bibliography.md#b8-monitoring-distributed-systems
[B9]: /skills/durability/references/bibliography.md#b9-serverless-architectures-on-aws
[B10]: /skills/durability/references/bibliography.md#b10-a-philosophy-of-software-design
[B11]: /skills/durability/references/bibliography.md#b11-property-based-testing
[B12]: /skills/durability/references/bibliography.md#b12-site-reliability-engineering
[B14]: /skills/durability/references/bibliography.md#b14-refactoring-databases
[B15]: /skills/durability/references/bibliography.md#b15-software-engineering-at-google