Notification Service

Difficulty: Medium Topics: Pub-Sub, Message Queues, Third-party Integration, Rate Limiting Time: 45 minutes Companies: Amazon (SNS), Uber, LinkedIn, Airbnb

Problem Statement

Design a scalable notification service that sends notifications across multiple channels (Email, SMS, Push Notifications) to millions of users.

Requirements:

• Functional:

  • Send email, SMS, and push notifications.

  • Support bulk notifications (e.g., "Marketing blast to 1M users").

  • Support transactional notifications (e.g., "OTP", "Order Confirmed").

  • User preferences (Opt-out/Opt-in channels).

• Non-Functional:

  • High Availability: 99.99%

  • Durability: No notification lost.

  • Throughput: 1M+ notifications/min.

  • Latency: Real-time for OTP (<5s), eventual for marketing (<10m).

  • Deduplication: Don't send duplicate alerts.

Architecture

Client Services
(Order Service, Marketing)
       ↓
   Load Balancer
       ↓
┌───────────────────┐
│ Notification API  │ (Rate limiting, Validation)
└────────┬──────────┘
         │
         ▼
┌───────────────────┐        ┌──────────────────┐
│  Service Router   │ ──────▶│ User Preferences │
└────────┬──────────┘        │ DB (MySQL/NoSQL) │
         │                   └──────────────────┘
         ▼
    Kafka / RabbitMQ (Decoupling & Buffering)
    topics: [priority-otp], [email], [sms], [push]
         │
         ▼
┌───────────────────┐
│     Workers       │ (Stateless Consumers)
└────────┬──────────┘
         │
    ┌────┴────┬─────────────┐
    ▼         ▼             ▼
  Email      SMS          Push
  Handler    Handler      Handler 
(SendGrid)  (Twilio)     (FCM/APNS)

Data Schema

User Preferences

CREATE TABLE user_preferences (
    user_id BIGINT PRIMARY KEY,
    email_enabled BOOLEAN DEFAULT TRUE,
    sms_enabled BOOLEAN DEFAULT FALSE,
    push_enabled BOOLEAN DEFAULT TRUE,
    timezone VARCHAR(50),
    dnd_start_time TIME,
    dnd_end_time TIME
);

Notification Log (Cassandra/DynamoDB)

CREATE TABLE notification_logs (
    notification_id UUID PRIMARY KEY,
    user_id BIGINT,
    type VARCHAR(20), -- EMAIL/SMS
    status VARCHAR(20), -- SENT/FAILED
    created_at TIMESTAMP,
    third_party_id VARCHAR(100)
);

Key Components

1. API & Validation

• Rate Limit: Prevent abusive clients (e.g., 100 req/sec per service).

• Validation: Check email format, phone number validity.

2. Priority Handling (Message Queues)

• Problem: Marketing blast (1M emails) blocks critical OTPs.

• Solution: Separate Queues!

  • High_Priority_Queue: OTPs, Security Alerts (Dedicated Workers).

  • Low_Priority_Queue: Marketing, Monthly Statements.

3. Deduplication and Exactly-Once Semantics (SDE-3 Concept)

• Problem: Client retries or network blips create duplicate notifications.

• Solution: Idempotency Keys.

  • The calling service generates a unique idempotency_key (UUID).

  • Check a fast distributed cache (Redis) for the key. TTL = 10 mins to 24 hours.

  • If exists -> Return 200 OK (already processed).

  • If not exists -> Process message, add key to Redis, return 200 OK.

  • Note on Kafka: Kafka provides "At-least-once" delivery by default. To prevent workers from sending duplicates when they crash before committing offsets, we must also check idempotency at the worker level before making the 3rd party API call.

4. Failure Handling with Dead Letter Queues (DLQ)

• Problem: What if an email address is permanently invalid or a message always crashes the worker (poison pill)?

• Solution:

  • If a message fails processing N times (e.g., 3 retries), move it to a Dead Letter Queue (DLQ).

  • Setup alerts on the DLQ size.

  • Engineers can inspect the DLQ, fix the bug or data issue, and replay the messages back into the main queue.

4. Third-Party Integration (The "Hard" Part)

• Challenges:

  • Vendor Downtime (Twilio down).

  • Rate Limits (Vendor blocks IP).

  • Latency (Vendor slow).

• Mitigation:

  • Retry with Backoff: Exponential backoff for 5xx errors.

  • Circuit Breaker: Stop calling if 50% errors, failover to secondary provider (e.g., SendGrid -> AWS SES).

  • Rate Limiter (Outbound): Throttle requests to match Vendor's TPS limit.

Scale Estimation

• Traffic: 10M notifications/day.

• Peak: 1M within 10 mins (Marketing).

• Storage:

  • Log retention: 1 year.

  • 1KB per log * 10M/day * 365 = ~3.6 TB/year.

  • Database: Cassandra (Write-heavy, TTL support).

Interview Tips

Q: "How do you handle users in Do Not Disturb (DND) mode?"

• A: "Workers check User Preferences DB before sending. If DND is active, park message in a Delayed_Queue with a timestamp to process later."

Q: "How to prevent spamming users?"

• A: "Implement a Frequency Cap in Redis (e.g., 'User X received 3 marketing emails today -> Drop 4th'). Critical alerts bypass this."

Q: "What if the worker crashes after sending to Twilio but before updating DB?"

• A: "Idempotency! Store third_party_id in DB. If worker restarts, check if we already have a success ID for this notification_id before retrying Twilio. If Twilio's API supports idempotency keys, pass our notification_id to them."