Skip to main content

​
Distributed Systems

Reference and design-time guide for the core guarantees, failure patterns, and coordination strategies every distributed system must handle. Each section pairs the concept with the tradeoff an agent or engineer must resolve when choosing it.

​
CAP Theorem

In a distributed system you can guarantee at most two of three properties simultaneously: Network partitions are a physical reality, so P is non-negotiable. The real choice is CP vs AP.

​
Practical implications

CP systems (e.g., HBase, Zookeeper, traditional RDBMS with sync replication): sacrifice availability when a partition occurs — requests block or return errors rather than stale data. Use when: banking, inventory, anything requiring atomic reads/writes. AP systems (e.g., Cassandra, DynamoDB, CouchDB): return possibly stale data during partitions, reconciling later. Use when: social feeds, DNS, shopping carts — any domain that tolerates temporary inconsistency. The CAP theorem describes a binary boundary; real systems operate on a spectrum. Latency is the hidden third variable — strongly consistent systems pay in latency under load even without a partition.

​
Consistency Patterns

​
Conflict resolution strategies for eventual consistency

Eventual consistency pushes conflict resolution to the application layer. Common strategies:
  • Last-write-wins (LWW): timestamp decides winner — simple but loses data on concurrent writes
  • Vector clocks: track causal history per node; detect concurrent writes explicitly (Amazon Dynamo approach)
  • CRDTs (Conflict-free Replicated Data Types): data structures that merge deterministically — counters, sets, sequences
  • Application-level merge: caller defines merge logic (e.g., union of sets, max of counters)
Choose LWW only if losing concurrent writes is acceptable. CRDTs when the data type maps naturally. Vector clocks when you need to expose conflicts to the user.

​
Idempotency

An operation is idempotent if applying it multiple times produces the same result as applying it once. Why it matters: networks fail, clients retry, message queues deliver at-least-once. Without idempotency, retries cause double-charges, duplicate records, or inconsistent state.

​
Implementation patterns

  • Idempotency keys: client sends a unique request ID; server stores (key → result) and returns cached result on duplicate. Used in Stripe and payment APIs.
  • Upsert semantics: INSERT ... ON CONFLICT DO UPDATE instead of blind insert
  • Conditional writes: PUT /resource with If-None-Match or optimistic locking (UPDATE WHERE version = $expected)
  • At-least-once with deduplication: consumers check a seen-ids store before processing
HTTP method semantics: GET, PUT, DELETE are idempotent by spec. POST is not — wrap POST operations in idempotency keys when retries are possible.

​
Circuit Breaker Pattern

Prevents cascading failures when a downstream service is degraded.
When to use: any synchronous service-to-service call where the callee can become slow or unavailable (databases, third-party APIs, downstream microservices). Failure mode without it: thread pools exhaust waiting for slow downstream; entire call chain degrades. Libraries: Resilience4j (Java), Polly (.NET), pybreaker (Python).

​
Backpressure

When a producer outpaces a consumer, queue size grows unboundedly — eventually consuming all memory and degrading the entire system. Backpressure is the mechanism by which a consumer signals capacity limits upstream, causing the producer to slow or shed load.

​
Strategies

See Scalability and Reliability for rate limiting algorithm details. Signal to caller: return HTTP 503 with Retry-After header, or use exponential backoff with jitter on the client side. Never silently queue indefinitely.

​
Saga Pattern

Manages distributed transactions across multiple services without two-phase commit. A saga is a sequence of local transactions. Each step publishes an event or message triggering the next. On failure, compensating transactions undo completed steps in reverse order.

​
Choreography vs Orchestration

Example (order processing): Reserve inventory → Charge payment → Ship. If payment fails, emit “release inventory” compensating event.

​
Two-Phase Commit (2PC) — and Why to Avoid It

2PC achieves distributed atomicity via a prepare phase and a commit phase coordinated across all participants. Problems:
  • Blocking: if coordinator fails after prepare but before commit, participants hold locks indefinitely
  • Single point of failure: coordinator is critical path
  • Latency: two network round trips for every transaction
  • Partition intolerance: choosing CP at significant availability cost
When it appears legitimately: XA transactions within a single datacenter, legacy system integration with no alternative. Prefer instead: Saga pattern for cross-service consistency. Design services to be idempotent and eventually consistent. Use a message queue (Kafka, SQS) as the coordination mechanism.

​
Distributed Locks

Used when multiple nodes must not concurrently access a shared resource (job scheduling, inventory reservation, leader election). Always set a TTL to prevent lock starvation if holder crashes. Always handle lock expiry: the critical section must be shorter than the TTL, or implement lock heartbeat renewal. Fencing tokens: use a monotonic counter issued with each lock grant. Pass the token to the resource being protected. Resource rejects operations from stale token holders — guards against process pauses (GC, OS) that expire the lock while the holder still runs.

​
Cross-references