Notification Ranking System Design

"Design a system that decides which push notifications and emails to send, to whom, and when" looks like feed ranking with a different surface. It is not. A feed is pulled: the user opens the app and asks for content, and the ranker only decides the order. A notification is pushed: the system interrupts someone who did not ask, and it can always choose to send nothing. A bad feed item costs a scroll. A bad notification costs attention, and if it happens often enough the user turns notifications off at the OS level or unsubscribes, and that channel is gone for good. Most of the design follows from that asymmetry: the objective is incremental value minus long-term damage, volume is a budget, and the evaluation needs holdouts that receive nothing at all.

For generic ranking machinery (retrieval, multi-task rankers, calibration) see News Feed Ranking System Design and Recommendation System Design. This guide focuses on what is specific to notifications.

Table of Contents

1. Clarify the Problem First
2. Why Notifications Differ from Feeds
3. Candidate Generation
4. Defining the Objective
5. Uplift: Would the User Have Come Back Anyway?
6. Models and Features
7. Selection Under a Budget
8. Volume Control
9. Send-Time Optimisation
10. Channel Selection
11. Dedup and Aggregation
12. Evaluation
13. Delivery Pipeline
14. Cold Start
15. Feedback Loops
16. Consent, Privacy and Regulation
17. Interview Q&A
18. Common Pitfalls
19. Related Topics

Clarify the Problem First

"Notifications" covers several products with different rules. Separate them before designing.

Which notification types?

• Transactional: password resets, payment receipts, security alerts, order shipped. The user expects them. They are not ranked or throttled by the ML system, and they usually bypass marketing opt-outs (but not all legal rules).
• Social / activity: "Alex commented on your photo", "3 new followers". Triggered by events involving the user. High relevance, high volume for active users.
• Content / recommendation: "A post you might like", "New video from a creator you follow", "Trending near you". Generated by the platform, no direct event. This is where most of the ML value and most of the fatigue risk sit.
• Re-engagement / lifecycle: "You haven't posted in a while", digests, onboarding nudges. Aimed at dormant or new users.

Which channels? Mobile push, email, SMS, in-app inbox (the bell icon), web push. Each has different cost, reach, and opt-out behaviour.

What does success mean? Opens are easy to measure and easy to game. Ask whether the goal is daily active users, sessions, retention at 28 days, or a specific action (purchases, posts). Also ask what counts as harm: notification disables, unsubscribes, spam reports, uninstalls.

Scale numbers to ask for:

• Users reachable per channel (push-enabled devices, email-subscribed addresses)
• Candidate events per day (every like, comment, follow, new post from a followed account)
• Latency expectations: a direct message notification should arrive in seconds; a "you might like" push can wait hours

For the rest of this guide, assume: a social app with a few hundred million monthly users, billions of candidate events per day, mobile push plus email plus an in-app inbox, and a goal of long-term retention without raising the disable rate. These are working assumptions for the exercise, not industry figures.

Non-Functional Requirements

Why Notifications Differ from Feeds

Three points deserve emphasis in an interview:

The "send nothing" option. A feed ranker picks the best k items from a candidate pool. A notification system decides whether the best candidate is worth an interruption at all. The threshold is per user: for a user who opens the app ten times a day, most notifications add nothing; for a user drifting away, one well-chosen push may matter.

Interruption cost. Every send spends a little of the user's tolerance. The cost is not visible in the send's own metrics: a push can get opened and still move the user closer to turning notifications off.

Opt-outs are permanent in practice. Once a user disables push in iOS or Android settings, the app cannot re-enable it; it can only ask the user to go to settings, which few do. Email unsubscribes must be honoured by law in many regions. So the loss from an opt-out is the value of every future notification to that user. That makes disable risk the dominant negative term in the objective.

Candidate Generation

Candidates come from two very different sources, and they need different pipelines.

 Event-triggered                              Scheduled / generated
 ---------------                              ---------------------
 like, comment, follow, DM, mention           daily/weekly digest jobs
 new post from followed account               "you might like" recommendations
 friend joined, live stream started           re-engagement for dormant users
 price drop, back in stock                    onboarding sequences
        |                                              |
        v                                              v
 Kafka event stream --> fan-out to              Batch job over user segments
 recipients (followers, thread members)         --> retrieval (recsys candidates)
        |                                              |
        +------------------+---------------------------+
                           v
              Per-user candidate queue (pending notifications)
                           |
                           v
              Eligibility filters --> Scoring --> Selection --> Scheduling

Event-triggered candidates arrive in a stream. The design question is how long to hold them. Sending instantly gives the best freshness but makes aggregation impossible ("Alex liked your post" followed a minute later by "Sam liked your post"). Holding for a short window allows batching and better decisions. Most systems use per-type hold windows: zero for DMs, minutes for likes, longer for "new post from followed account".

Fan-out. A post by an account with millions of followers creates millions of candidates at once. Treat it like feed fan-out: do not generate a candidate for every follower immediately. Generate for followers with a high predicted affinity first, rate-limit the rest, and let the ranker drop most of them.

Scheduled candidates come from batch jobs: pick users (by segment, timezone, days since last visit), retrieve content for each (using the same retrieval models as the feed), and write candidates into the per-user queue with an expiry time.

Every candidate carries: type, source event, recipient, content IDs, creation time, expiry time (a live stream notification is worthless after the stream ends), priority class, and allowed channels.

Eligibility filters run before any model: opted out of this type or channel, quiet hours, account state (suspended, deleted), content removed or blocked, the actor is muted or blocked by the recipient, already seen in-app (the user already read the comment, so do not notify).

Defining the Objective

The naive objective is P(open). It fails for three reasons.

1. It rewards clickbait. Vague or alarming copy gets opens and erodes trust.
2. It ignores harm. A push that gets opened but pushes the user toward disabling notifications is scored as a success.
3. It ignores what would have happened anyway. A daily user opens the app at 8am regardless. A push at 7:58 gets "credit" for an open that was going to happen.

A better per-candidate value:

value(u, n) =   w_1 * delta P(visit | send)                       # incremental visit, not raw open
              + w_2 * delta E[downstream actions]                 # posts, purchases, replies after open
              + w_3 * delta long-term retention proxy             # e.g. change in P(active in 28 days)
              - w_4 * P(disable or unsubscribe | send)
              - w_5 * P(spam report / negative feedback | send)
              - c_interrupt(u)                                    # per-user cost of an interruption

The delta terms are uplifts: the difference between sending and not sending. The disable term is multiplied by a large weight because a disable removes all future value on that channel. A rough way to size it: w_4 ~ expected future value of the channel for this user, which is larger for users who currently get a lot from notifications.

Long-term proxies. Retention takes weeks to observe. Teams often train a model that predicts long-term outcomes (such as 28-day activity) from short-term signals, then use it as a surrogate target, validating it periodically against long-running holdouts.

Weights come from experiments, not intuition. Tune w by running policy variants against each other and measuring the long-term outcome, then pick the weights whose short-term proxies correlate with it.

Uplift: Would the User Have Come Back Anyway?

This is the question that separates a strong answer. The value of a notification is the causal effect of sending it:

uplift(u, n) = P(visit within T | send n) - P(visit within T | no send)

A user with a 95% daily visit probability has at most 5 points of headroom. A user with 20% has much more. P(open) models are highest for the first user; uplift is often highest for the second.

Estimating uplift

You need data where sending was randomised. Logs from the production policy are confounded: the policy sent to users it predicted would open, so "sent" and "would have visited anyway" are correlated.

Randomised exploration. Hold out a small random share of eligible candidates (do not send them), and randomly send a small share of candidates the policy would have dropped. Log the propensity of every decision.

Model families:

import numpy as np
import lightgbm as lgb

def fit_t_learner(X, sent, visited):
    """X: features; sent: 0/1 randomised treatment; visited: 0/1 outcome within T."""
    params = dict(objective="binary", n_estimators=500, learning_rate=0.05,
                  num_leaves=63, min_child_samples=500)
    m_treat = lgb.LGBMClassifier(**params).fit(X[sent == 1], visited[sent == 1])
    m_ctrl = lgb.LGBMClassifier(**params).fit(X[sent == 0], visited[sent == 0])
    return m_treat, m_ctrl

def predict_uplift(m_treat, m_ctrl, X):
    return m_treat.predict_proba(X)[:, 1] - m_ctrl.predict_proba(X)[:, 1]

Evaluating uplift models uses the Qini curve or uplift-at-k on randomised data: sort by predicted uplift, and check that the observed treatment-minus-control difference is highest in the top buckets. Ordinary AUC on opens says nothing about uplift.

Practical compromise. Pure uplift models are noisy because individual effects are small and labels are binary. Many systems keep a multi-task model for P(open), P(disable), and downstream actions, and add a "baseline visit probability" feature or a separate uplift correction, so the ranker learns to discount users who were going to come back anyway.

Models and Features

Per-candidate scoring

A multi-task model scores every eligible candidate:

Disable and unsubscribe events are rare, so that head needs care: aggregate over a window after the send, use class weighting, and check calibration, since its output is multiplied by a large weight.

GBDTs work well on the tabular user-history features. A deep multi-task model helps when there are many notification types and content embeddings to share across heads. Calibration matters for every head because the outputs are combined in a weighted sum and compared against a threshold.

Features

The last group matters: the value of a candidate depends on what else the user will receive today.

Selection Under a Budget

Scoring each candidate is only half the job. The system must choose a set of sends for each user over a time window under constraints. That is a knapsack-style problem.

maximise    sum_i  x_i * value_i
subject to  sum_i  x_i * cost_i        <= budget_u     (per-user daily budget)
            x_i in {0, 1}
            per-type caps, min spacing between sends, quiet hours
            value_i > threshold_u    (never send something worse than nothing)

With unit costs, this reduces to "take the top candidates by value until the budget or the threshold is hit". Costs vary when different types have different interruption costs (a push costs more than an in-app inbox entry), which makes it a real knapsack; a greedy by value / cost ratio is the usual practical solution.

Online versus batch selection. Candidates arrive over the day, so the system cannot see the whole day's pool at the start. Two common approaches:

• Periodic batch: every N minutes, score the pending queue and send the best if it clears a threshold that reflects the remaining budget.
• Threshold policy: learn a per-user threshold from history (the value of the k-th best candidate on a typical day, where k is the budget). Send anything above it immediately, subject to caps. This is the online knapsack / secretary-problem framing.

Global constraints with Lagrangian relaxation. If there is a platform-wide goal (for example, total sends must not increase, or disable rate must stay under a target), attach a price lambda to the constrained quantity and subtract it from every candidate's value. Tune lambda so the constraint holds. This turns one global problem into independent per-user decisions.

def select_sends(candidates, budget, threshold, lam=0.0):
    """candidates: list of dicts with 'value', 'cost', 'p_disable'.
    lam: shadow price on disable risk from a global constraint."""
    scored = []
    for c in candidates:
        adj = c["value"] - lam * c["p_disable"]
        if adj > threshold:
            scored.append((adj / c["cost"], adj, c))
    scored.sort(key=lambda t: t[0], reverse=True)
    chosen, spent = [], 0.0
    for ratio, adj, c in scored:
        if spent + c["cost"] <= budget:
            chosen.append(c)
            spent += c["cost"]
    return chosen

Learned budgets. The budget itself can be personalised: estimate the marginal value of the k-th notification per day for each user (from randomised volume experiments) and set the budget where marginal value drops below marginal harm.

Volume Control

Volume control is the layer that keeps the system safe regardless of what the model says.

Priority classes. Transactional and security messages sit above all caps. Direct messages come next. Ranked content comes last and is the first to be dropped when budgets are tight.

Global throttles shape load. A digest sent to every user at 9am local time still produces a spike per timezone. Spread sends across a window and smooth by region, both to protect the delivery providers and because a synchronised burst of app opens can overload the backend.

Send-Time Optimisation

The same notification has different value at different times. Send-time optimisation (STO) picks when to deliver deferrable notifications.

Signals. The user's historical active hours, open latency by hour, timezone, and day of week. A simple baseline is the user's most common session start hour, smoothed toward the population profile for their region.

Model. Predict P(open within 1 hour | send at hour h) for each candidate hour, and pick the best hour within the notification's allowed window (before it expires, outside quiet hours).

import numpy as np

def best_send_hour(p_open_by_hour, allowed_hours, prior_by_hour, n_user_sends, k=20):
    """Shrink the user's hourly open-rate estimate toward a population prior,
    then choose the best allowed hour."""
    w = n_user_sends / (n_user_sends + k)
    blended = w * np.asarray(p_open_by_hour) + (1 - w) * np.asarray(prior_by_hour)
    mask = np.zeros(24, dtype=bool)
    mask[list(allowed_hours)] = True
    blended[~mask] = -np.inf
    return int(np.argmax(blended))

Pitfalls in STO:

• Selection bias. You only observe opens at hours the old policy sent. Add randomised send-time exploration.
• Open-time is not uplift. The hour when the user is most likely to open may be when they are already in the app. The incremental effect can be higher at a time they would not otherwise visit.
• Freshness trade-off. Delaying a social notification to the "best" hour makes it stale. Only defer types whose value decays slowly.
• Bandits are a good fit. Treat hours as arms with a contextual bandit (Thompson sampling or UCB) to balance exploration and exploitation per user.

Channel Selection

Model channel choice as part of selection: each (candidate, channel) pair is an option with its own value, cost, and harm terms, and at most one channel per candidate is chosen. Rules on top: if a push was sent and opened, suppress the email version; if a user has push disabled, fall back to email for high-value items; and anything dropped from the push budget can still go to the in-app inbox, which costs nothing.

Cross-channel coordination. Keep one per-user ledger of sends across channels so budgets and dedup work across them. Separate teams owning email and push with separate caps is a common source of over-messaging.

Dedup and Aggregation

Dedup. The same underlying event can produce several candidates (a comment that is also a mention, a post that is both "from a followed account" and "trending"). Key candidates by (recipient, content_id, action) and keep only the highest-value version. Also suppress anything the user has already seen in-app since the candidate was created.

Aggregation. Many small events are better sent as one: "Alex, Sam and 3 others liked your post" instead of five pushes.

 t=0   Alex liked post P    --+
 t=1m  Sam liked post P       |   hold window for (user, post P, "like")
 t=3m  Kim liked post P       +--> window closes or count threshold hit
 t=4m  Lee liked post P       |
 t=6m  Ana liked post P     --+
                                   v
                  "Alex, Sam and 3 others liked your post"
                  (actors ordered by recipient's affinity)

Design choices:

• Aggregation key: usually (recipient, target object, action type).
• Window: fixed hold, or send the first event immediately and aggregate subsequent ones into a later update. The second gives freshness for the first event but risks two pushes.
• Which names to show: rank actors by the recipient's affinity; the value of an aggregate depends mainly on the best actor in it.
• Score the aggregate, not the parts. The ranker scores the combined notification with count and top-actor features.
• Update in place. Some platforms support replacing a delivered push (using a collapse or thread identifier), so a later aggregate can replace the earlier one rather than add a new alert.

Evaluation

Offline

Offline results are weak evidence here, because the target is long-term and causal. Treat them as a filter before online tests.

Online: holdouts and incrementality

Global holdout. Keep a small random share of users who receive no ranked notifications at all (transactional still go out), for months. Comparing them with everyone else measures the total incremental value of the notification system, and it is the only clean way to answer "are we just taking credit for organic visits?"

Policy A/B tests. Randomise users between ranking policies and compare:

Novelty and fatigue effects. More notifications almost always look good in week one: opens and visits go up. Disables accumulate slowly and retention effects take weeks to show. So:

• Run volume experiments for weeks, not days, and plot the treatment effect over time. A rising-then-falling curve is the fatigue signature.
• Watch the cumulative disable rate per arm, not the daily rate.
• Keep some long-running holdouts at the old volume level to measure effects that only appear after months.

Measuring the value of a lost channel. Compare users who disabled push with similar users who did not (matched on pre-disable activity) to estimate the long-term cost of a disable. That estimate is what sets w_4.

import numpy as np

def incrementality(treat_active, holdout_active):
    """Daily-active indicators for the treatment population and the no-notification holdout.
    Returns absolute lift and a normal-approximation 95% CI."""
    p_t, p_h = np.mean(treat_active), np.mean(holdout_active)
    se = np.sqrt(p_t * (1 - p_t) / len(treat_active) + p_h * (1 - p_h) / len(holdout_active))
    lift = p_t - p_h
    return lift, (lift - 1.96 * se, lift + 1.96 * se)

Network interference. Social notifications involve two users: a treatment policy that notifies more about comments may lead to more replies, which generate notifications to control users. Use cluster-randomised designs (by graph community) when the change affects what users send each other.

Delivery Pipeline

                           NOTIFICATION SYSTEM
===============================================================================

 Product events --> Kafka --> Candidate generator --> Pending store
 (likes, posts,               (fan-out, dedup key,      (per-user queue,
  DMs, follows)                expiry, priority)         keyed by user_id)
                                                              |
 Batch jobs --> digests, recs, re-engagement -----------------+
                                                              v
                                     +---- Eligibility (consent, quiet hours,
                                     |     blocks, already seen)
                                     v
 Feature store --------------> Scoring service (multi-task + uplift)
                                     |
                                     v
                              Selection (budget, caps, channel, send time)
                                     |
                                     v
                              Scheduler (delay queue by send time, local TZ)
                                     |
                                     v
                              Delivery workers --> rate limiter --> APNs / FCM
                              (idempotency key,                     Email provider
                               retries, backoff)                    In-app inbox
                                     |
                                     v
 Delivery receipts, opens, disables, unsubscribes --> Kafka --> Logs + labels
                                                                   |
                                  Retrain --> Offline eval --> A/B --> Rollout

===============================================================================

Queueing. Separate queues by priority class so a large digest campaign cannot delay security alerts or DMs. Deferred sends go into a delay queue (a sorted set by send time, or a scheduler with time-bucketed partitions).

Rate limiting. Push providers (APNs for iOS, FCM for Android) and email providers throttle senders and can penalise bursts. Use token buckets per provider and per region. Email reputation also depends on bounce and complaint rates, so suppress addresses that hard-bounce.

Timezone handling. Store each user's timezone (from the device, updated on change) and compute quiet hours and send times in local time. Travel makes this stale; prefer the latest device-reported timezone. Watch daylight-saving transitions: schedule in UTC but compute from local rules at scheduling time, using a timezone library, not fixed offsets.

Idempotency. Workers retry, and at-least-once queues redeliver. Assign every send a deterministic idempotency key, such as a hash of (recipient, dedup key, channel, send window), and check-and-set it in a fast store before calling the provider. A duplicate push is a visible bug to the user.

Retries. Retry transient provider errors with exponential backoff and jitter, but give up after the notification's expiry. Handle permanent errors by updating state: an invalid device token means the app was uninstalled or the token rotated, so remove it; a hard email bounce means suppress that address.

Re-check at send time. Between selection and delivery, the user may have opened the app, seen the content, or opted out. Re-check consent and "already seen" right before sending.

Transactional path. Keep it separate from the ranking path, with its own capacity, so a ranking outage never blocks password resets.

Logging. Log every decision, including suppressed candidates with the reason (below threshold, over budget, quiet hours) and their propensity. Suppressed candidates are the control data for uplift models.

Cold Start

New users. No history, and the first days decide whether they stay. Use onboarding-specific rules (a small number of welcome and social-connection notifications), population priors by acquisition channel, region, and platform, and conservative budgets. Asking for push permission is itself a design decision: on iOS the OS prompt can be shown only once, so many apps first show an in-app explanation and ask at a moment when the value is clear (for example, after the user follows someone).

New notification types. No training labels for the new type. Start with a rule-based policy and a small exposure share, randomise sends to collect clean labels, share model heads with similar existing types, and scale up once disable rates look acceptable.

Dormant users. Their recent features are empty, and they are the most at risk of unsubscribing. Re-engagement campaigns should use tight caps and stop quickly if there is no response ("sunsetting": stop emailing after a long streak of no opens, which also protects sender reputation).

Shrink per-user estimates toward segment priors as data accumulates, for example (n x user_rate + k x prior_rate) / (n + k).

Feedback Loops

Volume creep is the one to mention unprompted. Each new notification type is usually tested alone, wins its short A/B test, and ships. The sum of all of them is never tested. A central budget and a global holdout are the structural fixes.

Consent, Privacy and Regulation

These are hard constraints, enforced in the eligibility layer and again at send time, not model features.

• Consent. Push requires OS permission. Marketing email and SMS require consent in many jurisdictions (for example, opt-in rules under GDPR and ePrivacy in the EU; CAN-SPAM in the US requires a working unsubscribe; SMS rules such as the TCPA in the US are strict). Store consent per channel and type, with timestamp and source.
• Unsubscribe handling. Email needs a visible unsubscribe link, and major mailbox providers now expect one-click unsubscribe headers for bulk senders. Process unsubscribes promptly and propagate them to every sending system.
• Granular preferences. Let users turn off individual types. A user who can mute "trending" pushes is less likely to disable all push.
• Quiet hours. Default no-send windows at night in local time, plus user-configured windows. Some jurisdictions restrict marketing messages at certain hours. Only critical transactional messages bypass them.
• Content privacy. Push previews appear on lock screens. Do not include sensitive content (health, finance, private messages if the user chose hidden previews) in the visible text.
• Data minimisation. Notification logs contain behavioural data; apply retention limits and honour deletion requests across logs and training sets.
• Minors. Stricter defaults and limits for younger users, as required by platform policy and local law.

Interview Q&A

Why is P(open) not a good objective?

It rewards the wrong things. Clickbait copy raises opens and erodes trust. It counts opens from users who would have visited anyway, so it favours the most active users, where the incremental effect is smallest. And it ignores harm: a push that gets opened but increases the chance of the user disabling notifications is scored as a win. The objective should be the incremental effect on visits and downstream actions, minus a heavily weighted disable and unsubscribe risk, minus an interruption cost.

How do you know notifications are adding value rather than taking credit for organic visits?

A long-running global holdout: a small random group that gets no ranked notifications. The difference in activity and retention between them and everyone else is the true incremental value. Per-decision, randomised exploration (randomly suppressing some sends the policy chose, and sending some it dropped) gives the data to train uplift models that estimate P(visit | send) - P(visit | no send) per user and candidate.

How would you decide how many notifications a user should get?

Treat it as a budgeted selection. Score candidates, then pick the best under a per-user budget and a threshold below which sending is worse than nothing. The budget should be personalised: run randomised volume experiments to estimate the marginal value and marginal disable risk of the k-th notification per day for different segments, and set the budget where they cross. Layer hard caps and per-type limits on top as safety rails.

Your new policy sends 20% more pushes and raises DAU in a one-week test. Do you ship?

Not yet. More pushes almost always lift short-term activity. The costs (disables, unsubscribes, lower open rates as users learn to ignore pushes) accumulate slowly. I would run it for several weeks, plot the treatment effect over time, and check the cumulative disable rate per arm. If the lift decays while disables rise, the long-term effect may be negative, because a disabled user is lost for every future notification.

How do you choose between push, email, and in-app?

Score each (candidate, channel) pair with its own value and harm, then pick at most one channel per candidate during selection. Push is for time-sensitive, high-relevance items because it has the highest interruption cost. Email suits digests. The in-app inbox has no interruption cost and no opt-out risk, so it receives anything that did not make the push budget. Coordinate through one cross-channel ledger so a push that was opened suppresses the matching email.

Five people liked a user's post in ten minutes. What gets sent?

One aggregated notification. Hold like-events for that (recipient, post, action) key for a short window, then send "Alex, Sam and 3 others liked your post", naming the actors the recipient has the highest affinity with. The ranker scores the aggregate using count and top-actor features. If the platform supports replacing a delivered notification, later updates replace the earlier one instead of alerting again.

How do you guarantee no duplicate pushes?

Queues deliver at least once and workers retry, so duplicates are prevented at the send step with a deterministic idempotency key derived from recipient, dedup key, channel, and send window, set atomically in a fast store before calling the provider. Candidate-level dedup upstream removes the same event arriving through two paths. Monitor the duplicate rate as a system metric.

How do you train when all your data comes from the current policy?

The logs only contain sends the current policy chose, so the model never sees outcomes for dropped candidates, and "sent" is correlated with "likely to visit anyway". Add a small amount of randomised exploration, log the propensity of every decision, including suppressions, and use inverse propensity weighting or doubly robust estimators for training and for off-policy evaluation. The global holdout gives the clean no-send baseline.

Common Pitfalls

Related Topics

• ML System Design Framework
• News Feed Ranking System Design
• Recommendation System Design
• Ads CTR Prediction and Ranking
• ETA Prediction System Design
• ML System Design Patterns
• Causal Inference and Uplift
• Recommender Systems
• Model Evaluation and Metrics
• Reinforcement Learning
• Apache Kafka
• A/B Testing
• Feature Store
• Model Serving
• Model Monitoring
• Responsible AI and Fairness