Design S3 Storage

Designing an S3-like object storage system is a classic system design interview problem. It tests whether you can handle huge scale, strict durability requirements, and practical API/product constraints at the same time.

This walkthrough follows the Interview Framework and focuses on what you would present in a 45-60 minute interview.

Phase 1: Requirements

Functional Requirements

1. Clients should be able to create buckets and manage bucket-level settings (versioning, lifecycle, ACL/policy).
2. Clients should be able to upload, download, and delete objects by key.
3. Clients should be able to upload large files with multipart upload and resume failed uploads.
4. Clients should be able to list objects by prefix for folder-like browsing.
5. Clients should be able to generate time-limited pre-signed URLs for delegated uploads/downloads.

Non-Functional Requirements

1. Durability: 99.999999999% ("11 nines") for stored objects.
2. Availability: 99.99%+ for read/write APIs.
3. Scalability: Billions of objects and millions of requests per second globally.
4. Latency: Low p99 for metadata operations, with streaming throughput for large objects.
5. Consistency: Strong read-after-write and strong list consistency inside a region; cross-region replication can be asynchronous.
6. Security: Strong IAM/policy enforcement, encryption in transit and at rest, and auditability.

Capacity Estimation

Assume we are designing for a large cloud provider:

Estimated annual new logical storage:

• 25 PB/day x 365 ~= 9.1 EB/year logical data
• With erasure coding/replication overhead (say 1.4x effective): ~12.7 EB/year physical footprint

Phase 2: Data Model

Core Entities

Bucket(
  bucket_id UUID PK,
  tenant_id UUID,
  name VARCHAR UNIQUE,
  region VARCHAR,
  versioning_status ENUM('enabled','suspended'),
  default_encryption JSONB,
  created_at TIMESTAMP
)

Object(
  object_id UUID PK,
  bucket_id UUID FK,
  object_key TEXT,                -- e.g. "images/2026/hero.png"
  latest_version_id UUID,
  object_state ENUM('active','delete_marker'),
  created_at TIMESTAMP,
  updated_at TIMESTAMP,
  UNIQUE(bucket_id, object_key)
)

ObjectVersion(
  version_id UUID PK,
  object_id UUID FK,
  content_hash CHAR(64),
  size_bytes BIGINT,
  storage_class ENUM('standard','infrequent','archive'),
  etag VARCHAR,
  metadata JSONB,
  chunk_manifest JSONB,           -- ordered chunk IDs + checksums
  retention_until TIMESTAMP NULL,
  created_at TIMESTAMP
)

MultipartUpload(
  upload_id UUID PK,
  bucket_id UUID FK,
  object_key TEXT,
  initiated_by UUID,
  status ENUM('initiated','completed','aborted'),
  created_at TIMESTAMP,
  expires_at TIMESTAMP
)

MultipartPart(
  upload_id UUID FK,
  part_number INT,
  chunk_ref TEXT,
  size_bytes BIGINT,
  part_etag VARCHAR,
  PRIMARY KEY(upload_id, part_number)
)

Relationships

• One Bucket has many objects.
• One Object has one current pointer plus many ObjectVersion rows.
• One MultipartUpload has many MultipartPart rows before completion.
• ObjectVersion points to immutable chunk manifests stored in data nodes.

Phase 3: API Design

Protocol Choice

• External API: REST over HTTPS (matches S3-style semantics and broad client compatibility).
• Internal service-to-service: gRPC for low-latency, typed contracts between metadata, placement, and storage-node services.

Core Endpoints

PUT /{bucket}
  Create bucket

PUT /{bucket}/{key}
  Upload object (single-part)
  Headers: Content-Length, Content-MD5, x-amz-meta-*

GET /{bucket}/{key}
  Download object (supports Range)

HEAD /{bucket}/{key}
  Fetch metadata only (size, etag, version)

DELETE /{bucket}/{key}
  Delete object (or create delete marker when versioning enabled)

GET /{bucket}?list-type=2&prefix=...&continuation-token=...
  List objects by prefix with pagination

Multipart Upload APIs

POST /{bucket}/{key}?uploads
  Initiate multipart upload
  Response: { uploadId }

PUT /{bucket}/{key}?partNumber={n}&uploadId={id}
  Upload one part
  Response header: ETag

POST /{bucket}/{key}?uploadId={id}
  Complete upload with ordered part list + ETags

DELETE /{bucket}/{key}?uploadId={id}
  Abort multipart upload

Access Control APIs

PUT /{bucket}?policy
  Set bucket policy JSON

POST /v1/presign-url
  Generate pre-signed URL
  Request: { bucket, key, method, expires_in_seconds }

Phase 4: High-Level Design

Architecture Overview

Write Flow (Single-Part Upload)

1. Client sends PUT /bucket/key with auth signature and checksum.
2. API gateway authenticates and authorizes against bucket policy.
3. Metadata service resolves bucket and asks placement service for chunk placement (multi-AZ).
4. Data proxy streams object bytes to selected storage nodes.
5. Storage nodes persist chunks and acknowledge durability threshold (for example, erasure set completion).
6. Metadata service commits new ObjectVersion atomically and emits an event.
7. Client receives success with ETag and versionId.

Read Flow

1. Client calls GET /bucket/key (optionally with Range).
2. Metadata service resolves the latest version and chunk manifest.
3. Data proxy reads chunks from nearest healthy nodes.
4. Response is streamed to client with integrity headers.

Multipart Upload Flow

Key Component Decisions

• Cell-based architecture: isolate blast radius; each cell owns a shard of buckets/objects.
• Metadata-data separation: strongly consistent metadata store + distributed immutable chunk store.
• Erasure coding for warm/cold data: lower storage cost than full replication while preserving durability.
• Background pipelines: lifecycle transitions, replication, inventory scans, and repair are asynchronous.

Phase 5: Scaling & Trade-offs

How the Design Meets NFRs

1. Durability
   • Multi-AZ placement for each object version.
   • Checksums per chunk + periodic scrubbing and repair.
   • Immutable versions simplify recovery and corruption rollback.
2. Availability
   • Cell isolation limits outage impact.
   • Stateless API/data-proxy layer scales horizontally.
   • Fast failover to healthy nodes on read path.
3. Low latency
   • Metadata cached at router and metadata-service layer.
   • Direct streaming from data proxy to storage nodes.
   • Range GET support avoids reading full large objects.
4. Scale
   • Partition metadata by (bucket_id, key_prefix_hash).
   • Auto-splitting hot partitions and adaptive load balancing.
5. Security
   • Signed requests validated at API gateway and policy engine.
   • Encryption in transit (TLS) and at rest (KMS-managed keys).
   • Access/audit logs for all control-plane and data-plane operations.

Bottlenecks and Mitigations

Trade-offs

Deep Dive: Metadata Partitioning

• Start with partition key: bucket_id + key_hash_prefix.
• Track per-partition QPS and storage growth.
• Split partitions when thresholds exceed SLO.
• Use consistent hashing for placement but keep logical mapping in metadata control plane.

Deep Dive: Data Integrity and Repair

• Store content hash and per-chunk checksums.
• Run continuous scrubbing jobs to detect silent corruption.
• If corruption is detected, reconstruct via replicas/parity shards and rewrite healthy chunks.
• Keep repair traffic rate-limited so it does not starve customer reads/writes.

Common Pitfalls

Interview Checklist

Before concluding, verify you covered:

• Functional requirements: bucket/object CRUD, list, multipart, auth
• Durability target and how architecture achieves it (multi-AZ, checksums, repair)
• Data model separation: metadata vs object bytes
• API surface including multipart flow and idempotency
• End-to-end write and read data flows
• Scaling plan for metadata hotspots and small-object overhead
• Trade-offs around consistency, latency, and cost
• Failure handling (node loss, AZ loss, replication lag)

Summary

If you present this structure clearly in an interview, you demonstrate both practical architecture skills and strong trade-off reasoning expected at senior levels.