Design a News Aggregator

A news aggregator like Google News collects articles from thousands of publishers and serves personalized feeds to millions of users. With 50,000+ publishers producing 5 million articles daily and 10 million monthly active users, this system design question tests your understanding of data ingestion pipelines, caching strategies, and personalization at scale.

This walkthrough follows the Interview Framework and focuses on what you'd actually present in a 45-60 minute interview.

Phase 1: Requirements

Functional Requirements

1. Aggregate news from publishers - Collect articles from 50,000+ news sources via APIs or web crawling
2. Personalized feed - Show relevant articles based on user's subscribed publishers and category preferences
3. Trending/global feed - Show trending articles for users without preferences or logged-out users
4. Category subscription - Users can subscribe to categories (Sports, Tech, Politics) and publishers

Non-Functional Requirements

• Scale: 50,000 publishers (100 articles/day each), 10M monthly active users, 100K concurrent at peak
• Availability: 99.9% uptime - users expect news to always be accessible
• Latency: Feed loads within ~200ms; users expect instant access to news
• Freshness: New articles appear in feeds within 5-15 minutes of publication
• Consistency: Eventual consistency is acceptable - a new article doesn't need to be instantly visible to all users

Capacity Estimation

Let's establish the scale we're designing for:

Traffic (Read):

• 10M monthly active users, ~3M daily active users
• Each user checks feed ~5 times/day = 15M feed requests/day
• 15M requests / 86,400 seconds = ~175 feed requests/second
• Steady peak: 100K concurrent users, ~3x average = ~500 requests/second
• Breaking news burst: if 100K users refresh within 2 seconds = ~50K requests/second

Ingestion (Write):

• 50,000 publishers × 100 articles/day = 5M articles/day
• 5M articles / 86,400 seconds = ~58 articles/second

Storage:

• Article metadata: ~5KB per article (title, description, URL, thumbnail, etc.)
• 5M articles/day × 5KB = ~25 GB/day of new article metadata
• Retain 30 days = 750 GB total (manageable in a single database)

Phase 2: Data Model

Core Entities

Article
├── article_id (PK)
├── publisher_id (FK)
├── title
├── description (snippet)
├── url (original source)
├── thumbnail_url
├── published_at
├── ingested_at
├── category (enum: tech, sports, politics, business, etc.)
├── language
├── popularity_score (computed)
└── dedup_hash (for duplicate detection)

Publisher
├── publisher_id (PK)
├── name
├── domain
├── feed_url (RSS/API endpoint)
├── crawl_frequency (minutes)
├── last_crawled_at
├── is_active
└── quality_score (editorial ranking)

User
├── user_id (PK)
├── email
├── name
├── region (geo: US, EU, APAC - inferred from IP or explicit)
├── preferred_language
├── preferred_categories[] (array of categories)
└── created_at

Subscription
├── subscription_id (PK)
├── user_id (FK)
├── type (enum: publisher, category)
├── target_id (publisher_id or category)
├── subscribed_at
└── notification_enabled

Database Selection

Article ID Generation

Use monotonically increasing IDs for efficient cursor-based pagination:

• Snowflake-style ID: Timestamp (41 bits) + machine ID (10 bits) + sequence (12 bits)
• IDs are globally unique, time-sortable, and monotonically increasing
• Cursor pagination uses the last (score, published_at, article_id) tuple: WHERE (popularity_score, published_at, article_id) < ({last_score}, {last_published_at}, {last_id}) ORDER BY popularity_score DESC, published_at DESC, article_id DESC LIMIT 20
• Deduplication hash: MD5/SHA256 of (publisher_id + title + published_at) to detect duplicates before insertion

Phase 3: API Design

Get Personalized Feed

GET /api/v1/feed
Headers: Authorization: Bearer <token>

Query Parameters:
- cursor: (optional) pagination token
- limit: 20 (default)
- category: (optional) filter by category

Response: 200 OK
{
  "articles": [
    {
      "article_id": "1234567890",
      "title": "Tech Giants Report Earnings",
      "description": "Quarterly results exceed expectations...",
      "url": "https://publisher.com/article/123",
      "thumbnail_url": "https://cdn.example.com/thumb/123.jpg",
      "publisher": {
        "publisher_id": "pub_123",
        "name": "TechNews"
      },
      "category": "tech",
      "published_at": "2024-01-15T10:30:00Z"
    }
  ],
  "next_cursor": "eyJsYXN0X3Njb3JlIjo5OC41LCJsYXN0X3B1Ymxpc2hlZF9hdCI6IjIwMjQtMDEtMTVUMTA6MzA6MDBaIiwibGFzdF9pZCI6IjEyMzQ1Njc4OTAiLCJ2IjoyfQ=="
}

Get Trending Feed (Guest Users)

GET /api/v1/feed/trending
Query Parameters:
- country: US (optional, for regional trending)
- category: (optional)
- limit: 20

Response: 200 OK
{
  "articles": [...],
  "next_cursor": "..."
}

Subscribe to Publisher/Category

POST /api/v1/subscriptions
Headers: Authorization: Bearer <token>

Request Body:
{
  "type": "publisher" | "category",
  "target_id": "pub_123" | "tech"
}

Response: 201 Created
{
  "subscription_id": "sub_456",
  "subscribed_at": "2024-01-15T10:30:00Z"
}

Search Articles

GET /api/v1/search?q={query}&cursor={token}&limit={limit}

Response: 200 OK
{
  "results": [
    {
      "article_id": "abc123",
      "title": "Matching Article",
      "snippet": "...context around match...",
      "publisher_name": "TechNews",
      "published_at": "2024-01-15"
    }
  ],
  "next_cursor": "..."
}

Phase 4: High-Level Design

Architecture Diagram

News Ingestion Flow

1. Scheduler triggers crawl: Cron-based scheduler determines which publishers need crawling based on crawl_frequency and last_crawled_at
2. Crawler fetches content: Workers call publisher APIs or parse RSS feeds
3. Parser normalizes data: Extract title, description, thumbnail, category from varying formats
4. Deduplication check: Hash (publisher + title + date), skip if already exists
5. Store article: Insert into PostgreSQL, enqueue for further processing
6. Index for search: Async job updates Elasticsearch with new articles
7. Update trending: Initialize popularity score from recency + source quality; engagement events update it continuously
8. Invalidate cached feeds: Publish event to update personalized feed caches

Feed Generation Strategies

There are two main approaches for generating personalized feeds:

Option 1: Fan-out on Write (Precomputed Feeds)

• When article is ingested, push to all subscribers' feed caches
• Pros: Fast reads (just fetch from cache), simple read path
• Cons: Expensive for publishers with millions of subscribers, wasted work if user doesn't check feed

Option 2: Fan-out on Read (Compute on Demand)

• When user requests feed, query articles from subscribed publishers
• Pros: No wasted computation, always fresh
• Cons: Slower reads, complex query, harder to rank across publishers

Hybrid Approach (Recommended):

• For users with few subscriptions (fewer than 50 publishers): Fan-out on read with caching
• For trending/global feed: Precompute and cache for all users
• Cache personalized feeds in Redis with 5-minute TTL

Feed generation query (simplified):
SELECT * FROM articles
WHERE publisher_id IN (user's subscribed publishers)
   OR category IN (user's preferred categories)
ORDER BY popularity_score DESC, published_at DESC, article_id DESC
LIMIT 50

Trending Feed Calculation

Trending articles are calculated using a time-decayed popularity score:

base_score = quality_score / (hours_since_published + 2)^1.5
engagement_score = (clicks + 10*shares) / (hours_since_published + 2)^1.5
score = base_score + engagement_score

Store in Redis Sorted Set:

• Key: trending:global or trending:category:tech
• Members: article IDs with scores
• Update every 1-5 minutes via background job (or use Apache Flink for real-time streaming aggregation)
• Use ZREVRANGE to fetch top articles

Phase 5: Deep Dives

News Ingestion Pipeline Design

Scheduling Strategy:

Implementation Options:

• Cron + Message Queue: Scheduler publishes crawl jobs to Kafka/RabbitMQ, workers consume
• Distributed scheduler: Use Apache Airflow or Temporal for complex workflows
• Serverless: AWS Lambda with CloudWatch Events for simpler setups
• Stream processing: For real-time trending/analytics, consider Apache Flink or Spark Streaming

Error Handling:

• Retry with exponential backoff (3 attempts, then mark publisher as failed)
• Alert on repeated failures (publisher might have changed their API)
• Dead letter queue for manual investigation

Caching Strategy

Cache Layers:

Cache Invalidation:

• Trending cache: Time-based expiration (recalculated every 1-5 minutes)
• Personalized feed: Invalidate on new subscription OR TTL expiration
• Article cache: Rarely invalidated (articles don't change often)

Redis Data Structures:

# Trending articles (sorted set)
ZADD trending:global {score} {article_id}
ZREVRANGE trending:global 0 19  # Top 20

# User's feed (list)
LPUSH feed:user:{user_id} {article_id}
LTRIM feed:user:{user_id} 0 99  # Keep last 100
LRANGE feed:user:{user_id} 0 19  # Get 20

# User's subscriptions (set)
SADD subscriptions:publisher:user:{user_id} {publisher_id}
SADD subscriptions:category:user:{user_id} {category}
SMEMBERS subscriptions:publisher:user:{user_id}
SMEMBERS subscriptions:category:user:{user_id}

Personalization & Recommendation

Lightweight Personalization (Recommended for interview):

1. Explicit preferences: Categories and publishers user subscribed to
2. Implicit signals: Click history → infer category preferences
3. User vector: Store lightweight interest vector in user table

{
  "user_id": "user_123",
  "interests": {
    "tech": 0.8,
    "sports": 0.5,
    "politics": 0.2
  }
}

Feed Scoring:

article_score = base_popularity 
              + (category_interest * 0.3) 
              + (subscribed_publisher ? 0.5 : 0)
              + recency_boost

Advanced (mention if asked):

• Collaborative filtering: "Users like you also read..."
• Content-based: Article embeddings using BERT/sentence transformers
• Real-time personalization: Feature store + ML model inference

Duplicate Article Detection

Publishers often syndicate the same story. Detect duplicates to avoid cluttered feeds:

Approaches:

1. Exact match: Hash of (title + publisher) - catches same article from same publisher
2. Near-duplicate: MinHash/SimHash on article content - catches syndicated content
3. Semantic similarity: Embedding similarity - catches rewritten versions

In practice:

• Stage 1: Exact title + URL deduplication (fast, blocking)
• Stage 2: Near-duplicate clustering (async, post-ingestion)
• Keep the highest-quality source when duplicates found

Handling Breaking News

Breaking news creates traffic spikes and requires special handling:

1. Priority ingestion: Tier 1 publishers trigger immediate crawl on webhook/push
2. Cache stampede prevention: Use "stale-while-revalidate" pattern
3. CDN surge: Push breaking stories to CDN edge immediately
4. Rate limiting: Protect origin servers from thundering herd

Thumbnail Storage & Delivery

Why not just link to publisher thumbnail URLs? Publisher images can be slow, change URLs, or go down entirely. For a consistent, fast feed experience:

Solution: S3 + CloudFront CDN with Multiple Sizes

1. Download & store: When ingesting articles, download the thumbnail and store in S3
2. Generate multiple sizes: Create responsive variants (100px mobile, 300px tablet, 600px desktop)
3. Serve via CDN: CloudFront delivers thumbnails from edge locations globally
4. URL structure: https://cdn.example.com/thumbnails/{article_id}/{size}.webp

# Thumbnail processing on ingestion
1. Download original image from publisher
2. Validate image (format, size, content moderation)
3. Generate sizes: thumb_small.webp (100px), thumb_medium.webp (300px), thumb_large.webp (600px)
4. Upload to S3 bucket with article_id prefix
5. Store CDN URLs in article metadata

This approach adds minimal storage cost (~50KB per article × 5M articles/day = ~250GB/day) but dramatically improves feed load times.

Redis Scaling with Read Replicas

With 100K concurrent users at peak, breaking news bursts can push 20-50K RPS. A single Redis instance might handle raw throughput, but you'll want read replicas for headroom and HA:

Architecture:

• Master: Handles all writes (new articles, cache updates)
• Read Replicas: Handle all reads (feed requests), load-balanced with round-robin
• Redis Sentinel: Automatic failover if master fails
• Per Region: Each region (US, EU, Asia) has its own master + replicas

Capacity math:

• 100K concurrent users, each refreshing feed ~5 times during session
• Breaking news spike: if 100K refresh within 2 seconds = ~50K RPS
• Single Redis instance: ~100K RPS, but keep utilization < 50% for P99 latency
• Needed: 1-2 read replicas per region for normal load, auto-scale to 5-10 during breaking news spikes

Category-Based Feeds

Users expect to browse by category (Sports, Tech, Politics). Pre-compute category feeds:

Implementation:

• Store category feeds in Redis: feed:{region}:{category} (e.g., feed:US:sports)
• Each category feed contains top 100 article IDs sorted by relevance/recency
• Update on new article ingestion (O(log N) sorted set insertion)
• Client requests specific category: merge with personalization at request time

In-Memory Filtering (Great Solution): For complex category combinations (e.g., "Tech AND Politics"), use in-memory filtering on the Feed Service:

1. Fetch base regional feed from Redis (fast, cached)
2. Filter in-memory by requested categories
3. Apply personalization scoring
4. Return top 20 results

This avoids pre-computing every category combination while maintaining sub-200ms latency.

Common Pitfalls

Interview Checklist

Before concluding, verify you've covered:

• News ingestion pipeline with tiered crawl scheduling and webhooks
• Deduplication strategy for syndicated content
• Personalized feed generation (fan-out on read vs write trade-off)
• Trending feed with time-decayed scoring
• Redis caching strategy with read replicas for scaling
• Regional architecture for geographic isolation
• Thumbnail storage in S3 + CDN (not linking to publisher)
• Category-based feeds with in-memory filtering
• Database choice justification (SQL vs NoSQL)
• Monotonic IDs for cursor-based pagination
• Handling breaking news / traffic spikes
• Capacity estimation showing system is read-heavy

Summary