Backend System Design Interview Guide

This guide adapts the provided standalone HTML system design guide into the repo's existing markdown-driven study flow. It is meant to complement the ML-first design framework with a broader backend interview checklist.

Table of Contents

1. At a Glance
2. Top 10 to Memorize First
3. How to Use This Guide
4. Reference Backend Architecture
5. How to Explain Architecture in an Interview
6. Interview Q&A (Scalability and Databases)
   • Scalability
   • Databases
7. Data Modeling, DDD, and Data-Driven Development
8. Interview Q&A (Caching, Distribution, and Reliability)
   • Caching
   • Distributed Systems
   • Messaging and Events
   • Networking and APIs
   • Storage
   • Reliability and Fault Tolerance
   • Search and Real-Time Systems
9. Worked Architecture Examples
10. Operational Review Checklist
11. Key Trade-Off Cheat Sheet
12. Suggested Interview Walkthrough
13. Common Mistakes in Backend Interviews
14. References
15. Recommended Follow-Ups in This Repo

At a Glance

• Scope: 32 high-yield backend system design concepts
• Priority set: 10 concepts worth memorizing first
• Coverage: scalability, databases, caching, distributed systems, messaging, networking, storage, reliability, and search
• Best use: review this page before system design interviews, then go deeper into the case-study files in this track

Top 10 to Memorize First

1. Load balancing
2. Caching strategies
3. SQL vs NoSQL
4. Database sharding
5. CAP theorem
6. Consistent hashing
7. Message queues and Kafka
8. Horizontal vs vertical scaling
9. Circuit breaker pattern
10. REST vs GraphQL vs gRPC

Why these first:

• They show up repeatedly in product, platform, and infrastructure interviews.
• They give you enough vocabulary to explain most trade-offs in larger system design answers.
• They form the backbone of common discussions around scale, reliability, and data architecture.

How to Use This Guide

For each concept, focus on five things:

1. Definition: what the concept is in plain language
2. Why it matters: why interviewers care
3. Example: where it appears in real systems
4. Common prompts: how it shows up in interview questions
5. Trade-off: when it helps and what it costs

Reference Backend Architecture

Most interview systems can be explained as a variation of this baseline:

Client
  |
  v
DNS / CDN / Edge
  |
  v
Load Balancer / API Gateway
  |
  v
Stateless Application Services
  |                 \
  |                  \--> Cache (Redis / Memcached)
  |
  +--> Primary Database / Search Index / Object Store
  |
  +--> Message Queue / Event Stream
          |
          v
       Async Workers / Consumers
          |
          v
   Secondary Stores / Notifications / Analytics / ML Pipelines

What each layer does

• Client: browser, mobile app, internal service, or batch producer making requests
• DNS / CDN / Edge: routes the request and offloads static assets or geo-distributed traffic
• Load Balancer / API Gateway: terminates TLS, routes traffic, applies auth, rate limits, and observability hooks
• Stateless application services: enforce business logic and stay horizontally scalable
• Cache: reduces read latency and protects the database from repeated hot queries
• Primary datastore: holds the source of truth for transactional or user-facing state
• Queue / event bus: moves slow, high-volume, or decoupled work off the synchronous path
• Workers: process async jobs such as notifications, indexing, retries, or media processing
• Secondary systems: analytics stores, search indexes, object stores, ML pipelines, or third-party integrations

Hot path versus async path

A strong answer usually distinguishes:

• the latency-sensitive path: what must happen before the user gets a response
• the deferred path: what can happen after the response

Examples:

• In checkout, charging the card may be on the hot path, but sending an email is async.
• In a feed system, writing the post may be synchronous, but fan-out and ranking can be async.
• In media upload, generating thumbnails and indexing metadata are usually async.

A simple request lifecycle

For a read-heavy endpoint:

1. Client sends request
2. Load balancer routes to an app instance
3. App checks cache
4. On cache hit, return immediately
5. On cache miss, query database
6. Store result in cache
7. Return response

For a write-heavy endpoint:

1. Client sends mutation request
2. App validates auth and business rules
3. App writes durable state to source-of-truth store
4. App emits event to queue or stream
5. Background workers update search, notifications, analytics, or downstream materialized views

How to Explain Architecture in an Interview

Interviewers are not looking for a giant diagram first. They want a sequence of defensible decisions.

Good explanation order

1. Clarify requirements
2. State system APIs or core entities
3. Draw the happy path
4. Choose storage
5. Add scale controls
6. Add failure handling
7. Add observability and operations

Questions you should ask first

• What is the scale in users, QPS, write rate, and storage growth?
• What latency matters: p50 only, or p95/p99?
• Is the system read-heavy or write-heavy?
• Is correctness strict or is eventual consistency acceptable?
• Do we need multi-region or is one region enough?
• Are there compliance, privacy, or audit constraints?

What strong answers sound like

Instead of saying:

• "I'll use microservices and Kafka and Redis."

Say:

• "The API tier stays stateless behind a load balancer so we can scale horizontally."
• "The primary write goes to a transactional store because correctness matters."
• "Search indexing and notifications are moved to Kafka consumers so the user-facing path stays under 100 ms."
• "We'll place Redis in front of the hot read endpoints because traffic is skewed and repeated."

Capacity estimation you should be able to do quickly

Even rough math earns points if it is directionally correct.

Examples:

• 10M daily active users with 5 requests per day is about 50M requests/day.
• 50M / 86400 is roughly 580 requests/sec average.
• If peak is 5x average, design for about 3K requests/sec.
• If each write is 2 KB, and there are 20M writes/day, raw write volume is about 40 GB/day before replication and indexing overhead.

You do not need exact precision. You need enough math to justify architecture choices.

Interview Q&A (Scalability and Databases)

The 32 core concepts are written as interview questions. Each answer covers the definition, why it matters, an example, common prompts, and the trade-off.

1. Scalability

Q1: What is the difference between horizontal and vertical scaling?

• Definition: Vertical scaling adds more CPU, RAM, or disk to one machine. Horizontal scaling adds more machines and distributes load across them.
• Why it matters: Almost every "design for millions of users" problem starts with a scaling decision.
• Example: Netflix scales service fleets horizontally across many instances instead of relying on a single oversized server.
• Common prompts: "How would you scale this API by 10x?" "Design YouTube, Twitter, or Instagram."
• Trade-off: Vertical scaling is simpler but capped. Horizontal scaling removes the single-box limit but introduces distributed systems complexity.

Q2: What is load balancing and which strategies are common?

• Definition: A load balancer distributes incoming requests across multiple backends. Common strategies include round robin, least connections, and layer-7 routing.
• Why it matters: It is usually the first scaling component interviewers expect you to mention.
• Example: Ride requests can be routed through regional load balancers before reaching matching services.
• Common prompts: URL shortener, API gateway, "How do you handle traffic spikes?"
• Trade-off: Stateless backends are easier to balance; sticky sessions reduce rehydration costs but can create hot spots.

Q3: What is rate limiting and which algorithms implement it?

• Definition: Rate limiting constrains how many requests a client can issue in a time window. Common algorithms include token bucket, leaky bucket, fixed window, and sliding window.
• Why it matters: Protects against abuse, DDoS-style bursts, and noisy neighbors.
• Example: Public APIs often enforce per-user or per-key quotas to keep traffic fair.
• Common prompts: "Design a rate limiter." "How would you protect a shared API platform?"
• Trade-off: Tight limits protect infrastructure but can punish legitimate bursty traffic. Token bucket is a common compromise.

Q4: What is auto-scaling and what metrics should drive it?

• Definition: Auto-scaling changes capacity dynamically based on metrics like CPU, request rate, queue depth, or latency.
• Why it matters: Shows you understand cost control as well as performance.
• Example: Order-processing workers can scale with queue backlog during peak traffic.
• Common prompts: "How would the system respond to seasonal spikes?" "How do you size workers for bursty workloads?"
• Trade-off: Aggressive scaling reduces latency but can increase cost and thrash if policies are noisy.

2. Databases

Q5: How do you choose between SQL and NoSQL?

• Definition: SQL databases favor relational structure, transactions, and rich querying. NoSQL systems favor flexible schemas and scale-oriented access patterns.
• Why it matters: Database choice is foundational in almost every design.
• Example: Payment ledgers fit relational databases; social activity feeds often lean toward NoSQL stores.
• Common prompts: "Which database would you choose and why?" "How do you store user content at scale?"
• Trade-off: SQL is strong on joins, consistency, and transactions. NoSQL is strong on horizontal scale and schema flexibility.

Q6: How does database indexing work and what does it cost?

• Definition: Indexes create fast access paths for lookups, sorting, and filtering without scanning the full table.
• Why it matters: Interviewers often check whether you understand how queries stay fast as data grows.
• Example: A composite index on (user_id, created_at) can speed timeline or order-history queries.
• Common prompts: "Why is the query slow?" "How do you optimize lookup latency?"
• Trade-off: Indexes improve reads but increase storage and write amplification.

Q7: What is replication and what trade-offs does it introduce?

• Definition: Replication copies data across multiple nodes for availability, durability, and read scaling.
• Why it matters: It is one of the first tools for high availability.
• Example: Primary-replica databases send writes to the primary and spread reads across replicas.
• Common prompts: "How do you improve read throughput?" "How do you survive node failure?"
• Trade-off: Replication improves resilience but introduces lag, failover complexity, and consistency questions.

Q8: What is sharding and when do you need it?

• Definition: Sharding partitions data across multiple databases so one machine does not hold all data or traffic.
• Why it matters: Eventually many fast-growing systems outgrow a single database instance.
• Example: User data can be distributed by hashed user ID across many shards.
• Common prompts: "One database is too large. What now?" "How do you scale writes?"
• Trade-off: Sharding removes single-node limits but complicates joins, resharding, and hot key management.

Data Modeling, DDD, and Data-Driven Development

These topics sit between "database choice" and "service design." Strong candidates do not just pick Postgres or Kafka; they define the domain, model the data correctly, and design feedback loops so the product can improve with evidence.

Data Modeling Fundamentals

Good backend design starts with the right data model.

Core questions:

• What are the main entities?
• What is the system-of-record for each entity?
• What is the write path and source of truth?
• What is the read pattern and access pattern?
• What is the grain of the data?

Example entities for an e-commerce system:

• users
• products
• orders
• order_items
• payments
• shipments

Typical relational model:

users (user_id)
orders (order_id, user_id, status, created_at)
order_items (order_item_id, order_id, product_id, quantity, unit_price)
payments (payment_id, order_id, provider, amount, status)
shipments (shipment_id, order_id, carrier, status, tracking_number)

Why this matters:

• It makes ownership and consistency boundaries explicit.
• It helps you reason about transactions and failure recovery.
• It prevents mixing user-level, order-level, and item-level facts incorrectly.

Normalized versus Read-Optimized Models

Operational systems usually begin with normalized data because:

• writes are simpler to keep correct
• duplication is reduced
• transactional guarantees are easier to reason about

Read-heavy systems often add denormalized views because:

• user-facing pages need fewer joins
• analytics or dashboard queries become cheaper
• cache keys and search documents often want pre-joined shapes

A strong answer distinguishes:

• write model: optimized for correctness
• read model: optimized for latency and query simplicity

This is one reason patterns like CQRS, materialized views, and search indexes appear so often.

Data Modeling Pitfalls to Call Out

• using one table for multiple unrelated entity types
• unclear primary key or business key
• storing mutable aggregate state without an event trail where audit matters
• mixing transactional tables and analytics tables as if they serve the same purpose
• ignoring timestamps, versioning, or status transitions

Domain-Driven Design (DDD)

DDD is useful when the domain is complex and the hardest problems are not technical primitives but business rules and model boundaries.

Key ideas:

• ubiquitous language: use the same terms in code, docs, and conversations
• bounded context: each domain area owns its own model and vocabulary
• aggregate: a consistency boundary for related entities
• entity versus value object: persistent identity versus descriptive structure
• domain service: logic that spans multiple entities but belongs to the domain

Bounded Contexts

A common backend mistake is to pretend the whole company shares one global model.

Example:

• Ordering context: cart, checkout, order lifecycle
• Payment context: authorization, capture, refund, chargeback
• Fulfillment context: warehouse picking, packing, shipment, delivery
• Recommendation context: candidate generation, ranking, experimentation

Order status in the ordering context does not need to mean the same thing as payment status in the payments context.

That separation gives you:

• cleaner service ownership
• fewer accidental coupling points
• more explicit integration contracts

Aggregates and Consistency Boundaries

An aggregate defines what must be changed together atomically.

Example:

• Order plus its order_items may be one aggregate for checkout integrity
• Payment is often a separate aggregate because it follows a different lifecycle and interacts with external systems

Strong interview point:

• not everything that relates conceptually should be in one transaction
• choose small consistency boundaries, then coordinate bigger workflows asynchronously when possible

Context Mapping

When domains talk to each other, define the contract clearly.

Common choices:

• synchronous API call
• asynchronous event publication
• replicated read model
• anti-corruption layer when integrating with legacy systems

Example:

• Payments emits payment_authorized
• Ordering consumes it and moves the order to the next state
• Fulfillment only starts after the ordering context publishes order_confirmed

This is where DDD and event-driven architecture connect naturally.

Data-Driven Development

Data-driven development means product and engineering decisions are shaped by instrumentation, experiments, and observed behavior rather than intuition alone.

In backend interviews, this usually shows up as:

• event schema design
• metric definitions
• experiment support
• observability and product analytics
• feedback loops into ranking, recommendations, or abuse detection

What to Instrument

At minimum, track:

• request received
• request succeeded or failed
• key domain actions completed
• downstream retries or fallback paths triggered
• user-visible outcomes

Example for checkout:

• checkout_started
• payment_attempted
• payment_authorized
• order_confirmed
• notification_sent

Why this matters:

• you can debug conversion drops
• you can distinguish product issues from infrastructure issues
• you can support A/B tests and funnel analysis

Event Schema Design

A good event usually includes:

• event name
• event timestamp
• entity identifiers
• user identifier when appropriate
• correlation or trace identifier
• version number
• context metadata such as device, region, provider, experiment variant

Example:

{
  "event_name": "payment_authorized",
  "event_time": "2026-04-10T10:15:00Z",
  "order_id": "ord_123",
  "payment_id": "pay_456",
  "user_id": "usr_789",
  "provider": "stripe",
  "experiment_variant": "checkout_v2",
  "schema_version": 3
}

Strong interview point:

• event schemas need versioning just like APIs do

Metrics and Product Feedback Loops

Data-driven systems usually define:

• north-star metrics
• guardrail metrics
• operational metrics

Example for search:

• north-star: successful search sessions
• guardrail: latency and zero-result rate
• operational: index freshness lag, query errors, cache hit rate

Example for recommendations:

• north-star: watch time or conversion
• guardrail: diversity, complaint rate, latency
• operational: feature freshness, model timeout rate, serving cost

Data-Driven Development in Practice

A mature backend answer may include:

• feature flags to roll out safely
• A/B testing support in request or event metadata
• warehouse or stream pipelines for analytics
• dashboards for funnel health and regressions
• batch jobs or online learning loops informed by observed outcomes

Example: Checkout Domain with DDD and Data-Driven Design

Ordering Context
  - owns cart, checkout, order lifecycle

Payments Context
  - owns authorization, capture, refund

Fulfillment Context
  - owns shipping and delivery workflow

Shared Data / Events
  - checkout_started
  - payment_authorized
  - order_confirmed
  - shipment_dispatched

How this helps:

• DDD gives clean ownership and consistency boundaries
• the data model gives clear tables and keys
• instrumentation gives funnel visibility and experiment support

Interview Questions These Topics Help With

• "How would you model the data for this system?"
• "What should be one service and what should be multiple services?"
• "What would you store in the transactional database versus the warehouse?"
• "How would you support experiments and product iteration?"
• "How do you keep your event schema maintainable over time?"

Interview Q&A (Caching, Distribution, and Reliability)

3. Caching

Q9: What is the cache-aside pattern?

• Definition: The application checks the cache first, falls back to the database on a miss, then writes the result into cache.
• Why it matters: It is the most common caching pattern in production systems.
• Example: Product detail pages are often cached in Redis after the first database read.
• Common prompts: "How do you reduce read load?" "How would you speed up a feed or profile page?"
• Trade-off: Simple to implement, but cache invalidation and staleness management become your responsibility.

Q10: What are write-through and write-behind caching?

• Definition: Write-through updates cache and backing store together. Write-behind writes to cache first and persists asynchronously later.
• Why it matters: Interviewers look for whether you understand caching on the write path, not only reads.
• Example: Session state may use write-through; high-volume counters may use write-behind buffering.
• Common prompts: "How do you cache hot writes?" "How would you handle very frequent updates?"
• Trade-off: Write-through improves consistency but increases write latency. Write-behind boosts throughput but risks data loss on failure.

Q11: How do CDN and edge caching work?

• Definition: A CDN caches content near users at edge locations to reduce latency and origin load.
• Why it matters: Global systems nearly always rely on edge caching for static and semi-static assets.
• Example: Images, JavaScript bundles, and video chunks are served from edge POPs instead of origin servers.
• Common prompts: "How do you improve global latency?" "How would you scale an image or video platform?"
• Trade-off: Excellent for read-heavy content, but invalidation and personalization can be difficult.

Q12: What is consistent hashing and why use it?

• Definition: Consistent hashing maps keys to cache or storage nodes so that adding or removing nodes remaps only a small fraction of keys.
• Why it matters: It is a standard answer for distributed caches and partitioned key-value systems.
• Example: Memcached clusters commonly use consistent hashing to spread keys.
• Common prompts: "How do you distribute keys across many cache nodes?" "What happens when one node is added or removed?"
• Trade-off: Great for minimizing churn, but poor virtual-node strategy can still create uneven distribution.

4. Distributed Systems

Q13: What is the CAP theorem?

• Definition: In the presence of a network partition, a distributed system can prioritize consistency or availability, but not fully guarantee both.
• Why it matters: It gives a framework for discussing consistency trade-offs clearly.
• Example: A banking ledger leans toward consistency; a social feed often leans toward availability.
• Common prompts: "Would you choose CP or AP here?" "What happens during a partition?"
• Trade-off: The theorem is not a design recipe, but it forces you to decide what the system sacrifices under failure.

Q14: What is the difference between strong and eventual consistency?

• Definition: Strong consistency means every read reflects the latest successful write. Eventual consistency means replicas converge over time.
• Why it matters: Many interview questions hinge on whether stale reads are acceptable.
• Example: Inventory counts often need strong consistency; like counts can usually tolerate eventual consistency.
• Common prompts: "Can users see stale data?" "How much inconsistency can the product tolerate?"
• Trade-off: Strong consistency improves correctness but usually costs latency and availability.

Q15: What is leader election and where is it used?

• Definition: A distributed system may elect one node as leader to coordinate writes, scheduling, or ownership.
• Why it matters: It appears in consensus systems, schedulers, and service coordination.
• Example: Kubernetes control-plane components rely on leader election for safe coordination.
• Common prompts: "How do you prevent two coordinators from acting at once?" "How does failover happen?"
• Trade-off: Central leadership simplifies coordination but creates failover and split-brain considerations.

Q16: What is consensus, and how does Raft achieve it?

• Definition: Consensus protocols let distributed nodes agree on a sequence of operations even when some nodes fail. Raft is a commonly discussed protocol because it is easier to reason about than Paxos.
• Why it matters: It comes up when discussing metadata stores, configuration management, and distributed locks.
• Example: etcd uses Raft for replicated metadata and leader election.
• Common prompts: "How do you keep replicas in sync?" "How does a distributed lock service work?"
• Trade-off: Consensus gives stronger guarantees but requires quorum and increases write-path coordination cost.

Q17: What is two-phase commit (2PC)?

• Definition: Two-phase commit coordinates atomic changes across multiple participants by asking each to prepare, then commit or abort.
• Why it matters: Interviewers may ask how to preserve consistency across systems.
• Example: Legacy financial workflows sometimes used 2PC for cross-database updates.
• Common prompts: "How do you commit changes across services atomically?" "How do you avoid partial updates?"
• Trade-off: Strong but blocking, coordinator-dependent, and often avoided in large microservice architectures.

Q18: What is the saga pattern?

• Definition: A saga models a distributed transaction as a sequence of local transactions with compensating actions if a later step fails.
• Why it matters: It is the modern answer when 2PC is too rigid for microservices.
• Example: Travel booking can reserve a room, authorize payment, and compensate prior steps if later steps fail.
• Common prompts: "How do you handle partial failure in an order flow?" "How do services coordinate without a global transaction?"
• Trade-off: More resilient and scalable than 2PC, but harder to reason about and debug end to end.

5. Messaging and Events

Q19: What are message queues and event streaming, and how do they differ?

• Definition: Message queues decouple producers and consumers for task processing. Event streams provide durable logs (ordered within a partition, in Kafka) that multiple consumers can replay.
• Why it matters: Async communication is core to modern large systems.
• Example: Kafka powers activity feeds, audit streams, and analytics pipelines in many companies.
• Common prompts: Notification service, analytics pipeline, "How do you decouple services with different speeds?"
• Trade-off: Improves resilience and throughput, but increases operational complexity and requires idempotent consumers.

Q20: What is event sourcing?

• Definition: Instead of storing only current state, the system stores the sequence of events that produced that state.
• Why it matters: Useful for auditability, replay, and debugging in domains with strict history requirements.
• Example: Ledger-like systems can reconstruct balances by replaying transaction events.
• Common prompts: "How do you support audit trails?" "How would you implement time travel or rollback?"
• Trade-off: Great for history and recovery, but current-state queries usually require snapshots or derived views.

Q21: What is CQRS?

• Definition: Command Query Responsibility Segregation separates the write model from the read model so each can be optimized independently.
• Why it matters: Often paired with event-driven architectures when reads and writes have very different shapes.
• Example: Feed systems may precompute read-optimized timelines while writes flow through a separate command path.
• Common prompts: "How do you optimize for heavy reads and different write semantics?" "How do you support many query patterns?"
• Trade-off: Strong read optimization, but eventual consistency and extra infrastructure become part of the design.

6. Networking and APIs

Q22: How do you choose between REST, GraphQL, and gRPC?

• Definition: REST is resource-oriented over HTTP. GraphQL lets clients request exactly the fields they need. gRPC uses strongly typed schemas and efficient binary transport, usually over HTTP/2.
• Why it matters: API protocol choice affects latency, evolution, client ergonomics, and caching.
• Example: Internal service-to-service calls may use gRPC; public developer APIs may use REST or GraphQL.
• Common prompts: "How do you design the API?" "What protocol should mobile clients use?"
• Trade-off: REST is simple and cache-friendly, GraphQL reduces over-fetching, and gRPC is fast and typed but less browser-native.

Q23: When do you use WebSockets versus long polling?

• Definition: WebSockets keep a persistent bidirectional connection open. Long polling keeps an HTTP request open until data is available and then reconnects.
• Why it matters: Real-time systems need a push mechanism.
• Example: Chat apps use persistent connections to deliver messages and presence updates.
• Common prompts: Chat system, live sports, collaborative editing, "How do you push updates in real time?"
• Trade-off: WebSockets are efficient at scale for bidirectional traffic but require connection state management.

Q24: How do DNS and anycast routing work in large systems?

• Definition: DNS maps names to addresses. Anycast advertises the same IP from many locations so traffic naturally reaches the nearest healthy edge.
• Why it matters: Global systems start with traffic steering before application logic even runs.
• Example: Edge providers route users to the closest data center to reduce latency and absorb attacks.
• Common prompts: "How does a request reach your system?" "How do you design a global service?"
• Trade-off: Powerful for geo-routing and resilience, but routing behavior can be difficult to debug.

7. Storage

Q25: What is the difference between object, block, and file storage?

• Definition: Object storage handles blobs at large scale, block storage acts like attached disks, and file storage exposes shared hierarchical file systems.
• Why it matters: Choosing the wrong storage type can hurt latency, cost, and operational simplicity.
• Example: Media files often live in object storage while transactional metadata lives in a database.
• Common prompts: "How would you store uploads?" "Where should user-generated media live?"
• Trade-off: Object storage is cheap and durable but higher latency; block storage is fast but less elastic; file storage is convenient but not always the most scalable.

Q26: What data partitioning strategies exist and how do you choose one?

• Definition: Range, hash, and list partitioning distribute data based on different keying schemes.
• Why it matters: Partitioning strategy strongly affects balance, query efficiency, and hot spot risk.
• Example: Time-series systems often partition by time range, while general key-value systems often partition by hashed key.
• Common prompts: "Why is one shard 10x hotter?" "How do you support efficient range scans?"
• Trade-off: Hashing balances load but hurts range locality; range partitioning helps range queries but risks hot partitions.

8. Reliability and Fault Tolerance

Q27: What is the circuit breaker pattern?

• Definition: A circuit breaker stops repeated calls to a failing dependency after an error threshold and retries only after a cooldown window.
• Why it matters: Prevents cascading failures in service meshes and microservice graphs.
• Example: A checkout service can fast-fail calls to a degraded recommendation service instead of exhausting threads.
• Common prompts: "What happens if a downstream service is failing?" "How do you contain blast radius?"
• Trade-off: Improves system stability, but threshold tuning matters or you risk false opens or slow protection.

Q28: What is idempotency and why does it matter?

• Definition: An operation is idempotent if repeating it does not change the final result beyond the first successful application.
• Why it matters: Safe retries depend on idempotency.
• Example: Payment APIs use idempotency keys so retrying a timed-out request does not double charge the user.
• Common prompts: "How do you handle duplicate requests?" "How do retries avoid creating duplicate orders?"
• Trade-off: Usually requires request tracking, deduplication state, and careful API contract design.

Q29: How should you design health checks, timeouts, and retries?

• Definition: Health checks remove unhealthy instances, timeouts bound waiting time, and retries recover from transient faults.
• Why it matters: These are baseline reliability controls for any production system.
• Example: A load balancer stops routing to a slow or unhealthy instance while clients retry with backoff.
• Common prompts: "How do you handle slow dependencies?" "What do you do when a service becomes flaky?"
• Trade-off: Retries improve transient-failure handling but can create retry storms without backoff, jitter, and circuit breakers.

Q30: What are replication factor and quorum?

• Definition: Replication factor decides how many copies of data exist. Quorum reads and writes require a majority or another chosen threshold.
• Why it matters: It lets you tune durability, availability, and consistency explicitly.
• Example: Distributed databases can offer consistency levels like ONE, QUORUM, or ALL.
• Common prompts: "How do you guarantee durability?" "How do you balance consistency and latency?"
• Trade-off: Higher replication improves durability but increases storage cost and often write latency.

9. Search and Real-Time Systems

Q31: How do search indexes work?

• Definition: Inverted indexes map terms to documents so full-text search can run efficiently at scale.
• Why it matters: Traditional database indexes are not enough for large search products.
• Example: Restaurant or marketplace search often uses Elasticsearch for text, filters, and ranking.
• Common prompts: "How would you build search?" "How do you implement autocomplete or faceted discovery?"
• Trade-off: Search engines are powerful but create sync and consistency work between the primary database and the index.

Q32: How do you store and query time-series data?

• Definition: Time-series systems are optimized for append-heavy writes, retention policies, and range queries over timestamps.
• Why it matters: Monitoring, IoT, analytics, and location streams all produce time-based data.
• Example: Metrics platforms store CPU, memory, latency, and service health as time-series records.
• Common prompts: "Design a metrics platform." "How do you store and query high-volume telemetry?"
• Trade-off: Time-series stores excel on time-based access patterns, but they are not general-purpose transactional databases.

Worked Architecture Examples

These are compact examples of how the concepts in this guide fit together in actual interview answers.

Example 1: URL Shortener

Requirements

• Create short links quickly
• Redirect with low latency
• Handle read traffic much higher than write traffic
• Avoid collisions in short-code generation

High-level design

Client -> LB/API -> Shortener Service -> SQL/NoSQL Store
                               |
                               +-> Cache for hot short codes
                               +-> Async analytics event stream

Key decisions

• Use a durable primary store for mapping short_code -> long_url
• Cache the hottest short codes because redirect traffic is skewed
• Keep analytics off the hot redirect path by sending click events asynchronously
• Pre-generate IDs or use a central ID service to avoid short-code collisions

Trade-offs to discuss

• SQL is fine early because the entity is simple and correctness matters
• NoSQL becomes more attractive if partitioned key lookups dominate at very large scale
• Redirect latency should take priority over perfect real-time analytics

Example 2: Notification System

Requirements

• Accept notification requests from many upstream services
• Support email, push, SMS, and in-app notifications
• Handle spikes without losing requests
• Support retries and provider fallback

High-level design

Producer Services
    |
    v
Notification API -> Queue / Topic -> Channel Workers -> External Providers
                         |
                         +-> Status Store / Retry DLQ / Audit Log

Key decisions

• Use a queue because notifications are naturally asynchronous
• Separate workers by channel to isolate provider issues
• Store delivery attempts and status transitions for debugging and audit
• Use retry with backoff and dead-letter queues for poison messages

Failure cases to discuss

• external SMS provider times out
• push provider rate-limits requests
• duplicate requests arrive from upstream callers
• user preference service is temporarily unavailable

Strong interview point:

• mention idempotency early because notification systems are retry-heavy and duplicate sends are a common production failure

Example 3: Chat or Real-Time Messaging

Requirements

• Low-latency send and receive
• Message persistence
• Online presence
• Ordering within a conversation

High-level design

Client <-> WebSocket Gateway <-> Chat Service
                               |      \
                               |       \-> Presence Store (Redis)
                               |
                               +-> Message Store
                               +-> Stream / Queue for fan-out, push, analytics

Key decisions

• Use WebSockets for bidirectional real-time communication
• Store message history durably in a database
• Use Redis or similar for ephemeral presence state
• Use async fan-out to notify other connected users or devices

Trade-offs to discuss

• strict global ordering is expensive; per-conversation ordering is often enough
• presence is ephemeral and can tolerate some inaccuracy
• multi-device sync may require sequence numbers or message offsets

Example 4: Search and Discovery Service

Requirements

• Full-text search with filters
• Relevance ranking
• Near-real-time updates
• Support typo tolerance or stemming

High-level design

Writers -> Primary DB -> Change Events -> Indexer -> Search Engine
Clients -> API -> Search Service -> Search Engine

Key decisions

• Keep source-of-truth writes in the primary database
• Build the search index asynchronously
• Accept eventual consistency between DB and index
• Use specialized search infrastructure instead of stretching an OLTP database

Common follow-up

Interviewers often ask how stale the index can be. Your answer should tie staleness budget to product expectations.

Operational Review Checklist

Before finishing a backend design answer, quickly review these dimensions.

Functional shape

• What are the core APIs?
• What are the main entities and their relationships?
• What is the read path?
• What is the write path?

Scale

• expected QPS and peak QPS
• storage growth over time
• read/write ratio
• request and payload size

Reliability

• timeouts
• retries with backoff and jitter
• circuit breakers
• dead-letter queues
• graceful degradation path

Data correctness

• consistency model
• idempotency strategy
• deduplication strategy
• backup and restore plan
• replication and failover

Operations

• metrics: QPS, latency, error rate, queue depth
• logs: structured request and failure logs
• tracing: cross-service call visibility
• alerts: SLO-based rather than noise-based

Security

• auth and authz model
• rate limiting and abuse controls
• encryption in transit and at rest
• audit logging where needed

Key Trade-Off Cheat Sheet

Suggested Interview Walkthrough

When answering a system design problem, move in this order:

1. Clarify users, traffic, latency, and correctness requirements.
2. Draw the happy path end to end.
3. Choose storage and explain read and write patterns.
4. Add caches, queues, and background workers where they remove pressure from the hot path.
5. Explain scaling: load balancers, sharding, replication, and auto-scaling.
6. Explain failure handling: retries, timeouts, circuit breakers, idempotency, and fallback behavior.
7. Close with observability, cost, and the key trade-offs you deliberately chose.

Common Mistakes in Backend Interviews

• jumping into microservices before establishing the core data model and traffic shape
• naming tools without explaining why they are needed
• forgetting the difference between the synchronous user path and the background path
• ignoring failure modes and only describing the happy path
• choosing strong consistency everywhere without discussing latency and availability cost
• choosing eventual consistency everywhere without checking whether correctness permits it
• omitting idempotency in retry-heavy workflows
• skipping observability entirely
• never mentioning backpressure, queue depth, or hot keys in high-scale designs

References

• Designing Data-Intensive Applications, Martin Kleppmann
• System Design Interview, Alex Xu
• Web Scalability for Startup Engineers, Artur Ejsmont
• Release It!, Michael T. Nygard
• Site Reliability Engineering, Google
• The Architecture of Open Source Applications
• Kafka, Redis, PostgreSQL, and Elasticsearch official documentation

Recommended Follow-Ups in This Repo

• ML System Design Framework
• Backend and System Design for AI
• Recommendation System
• Fraud Detection
• ML System Design Patterns