This is billed as a system design round, but it does not play like one. There is no whiteboard, and the interviewer is not looking for the usual parade of boxes: load balancer, API gateway, cache, database. The prompt is a single monitoring service that must collect metrics from 1000 other servers every 10 minutes, and almost the entire conversation is about one thing: how you design the workers that do the collecting. It ends with you writing the multithreaded code for that piece.
So this page focuses where the interview does. You will not get much credit for a sprawling architecture diagram. You get credit for reasoning clearly about concurrency, backing it with a bounded, fault-tolerant worker design, and then writing code that actually holds up.
The whole round rests on one observation: collecting a metric from a server is an I/O-bound operation, not a CPU-bound one. A worker spends nearly all of its time waiting on a network round trip, not computing. Say this out loud early. It is the fact that justifies running far more workers than you have cores, and it drives every decision that follows. [Source: darkinterview.com]
This walkthrough follows the Interview Framework loosely and is sized for a 45 to 60 minute round, but weighted heavily toward the worker design and the code, because that is where the time goes.
Pin down what "collect metrics from 1000 servers every 10 minutes" actually asks for:
Do the arithmetic out loud, because it is what motivates everything else. You have 1000 collections to finish inside 600 seconds. A single network round trip can take anywhere from a few milliseconds to, for a hung server, several seconds until it times out. Run them one at a time and the worst case is 1000 multiplied by your timeout, which blows straight past the 10-minute window. Serial collection does not fit, so concurrency is not an optimization here. It is the design. [Source: darkinterview.com]
Resist the urge to start drawing databases and dashboards. The interviewer has deliberately handed you a prompt where the storage and the UI are not the interesting part. The interesting part is: 1000 independent, I/O-bound tasks, a fixed time budget, and unreliable targets. Steer straight at that.
This is the core of the round. You are designing the workers that fan out the 1000 collections and bring the results back within the window. Five decisions carry the design.
The tempting first idea is one thread per server: 1000 threads, each collecting from one target. Reject it. A thousand threads burn memory on stacks, thrash the scheduler, and give you no knob to turn when the host or the network is the bottleneck. Instead use a bounded pool of workers, say 50, pulling targets from a shared queue. Resource use is predictable and tunable, and because the work is I/O-bound the pool can be far larger than the core count without wasting CPU. [Source: darkinterview.com]
Each cycle, the 1000 server targets go into a queue. Workers pull from it, collect, and come back for the next target. This gives you load balancing for free: a worker that finishes a fast server immediately grabs another, so the fast workers naturally absorb more targets while a few workers sit blocked on slow ones. You never hand-partition the 1000 servers across workers; the queue does it dynamically.
This is the decision the interviewer will push on hardest, so lead with it. Every single collection must have its own timeout. Without one, a single hung server holds its worker forever, and enough dead servers drain the pool until the cycle stalls indefinitely. With a per-server timeout, the worst a bad server costs you is that one bounded wait, after which you record it as unreachable and move on.
The per-server timeout is not a detail you add at the end. It is what makes the 10-minute budget achievable at all. Together with the pool size it gives you a provable worst case: with a pool of P workers, N servers, and a timeout of T seconds, even if every server times out the cycle finishes in about ceil(N / P) * T. With 1000 servers, 50 workers, and a 5-second timeout that is 20 * 5 = 100 seconds, comfortably inside 600. State that bound. It turns "I used threads" into "here is why it fits." [Source: darkinterview.com]
One server erroring out, a refused connection, a timeout, a malformed payload, must not kill the worker or abort the cycle. Each task catches its own failures and turns them into a recorded result. A failure is a data point the monitor wants: "server X was unreachable at 14:30" is exactly the kind of thing a system monitor is for. The rule is that no single target can take down the collection.
A scheduler kicks off a fresh cycle on the cadence. The subtle question, and the interviewer will ask it, is what happens when a cycle has not finished by the time the next one is due. The safe default is to not launch an overlapping cycle: waiting for the current cycle to finish before starting the next avoids piling a second wave of load onto servers that are already struggling. If cycles routinely overrun, that overrun is itself a signal worth alerting on.
This is the only diagram worth drawing, and even this is really about the workers. The scheduler feeds a queue, a bounded pool drains it against the fleet of servers, and every result, success or failure, lands in the sink. [Source: darkinterview.com]
This is the part you are actually asked to write. A thread pool fits cleanly because the work is I/O-bound, and a language's pool-plus-futures API expresses it directly. The Python below uses concurrent.futures.ThreadPoolExecutor; the shape is the same with a Java ExecutorService and Future, or a Go worker pool over a channel.
Each task is self-contained: make the request under a timeout, and return a result whether it succeeds or fails. It never raises past its own boundary.
import concurrent.futures
import time
from dataclasses import dataclass
from typing import Optional
PER_SERVER_TIMEOUT = 5.0 # seconds; one slow server must not stall the cycle
POOL_SIZE = 50 # I/O-bound work, so far more workers than CPU cores
@dataclass
class MetricResult:
server: str
ok: bool
metrics: Optional[dict] = None
error: Optional[str] = None
def collect_one(server: str) -> MetricResult:
try:
# fetch_metrics is a blocking network call with its own timeout.
payload = fetch_metrics(server, timeout=PER_SERVER_TIMEOUT)
return MetricResult(server, ok=True, metrics=payload)
except Exception as exc: # timeout, refused connection, bad payload
return MetricResult(server, ok=False, error=str(exc))
Catching a broad Exception here is deliberate. A worker must survive any single target so it can go back for the next one, and an unreachable server is a result to record, not a crash to propagate. [Source: darkinterview.com]
Submit all 1000 targets to the pool and drain the results as they complete. Because collect_one already swallows its own errors and every request is time-bounded, the whole cycle is bounded too.
def run_cycle(servers: list[str], sink) -> None:
with concurrent.futures.ThreadPoolExecutor(max_workers=POOL_SIZE) as pool:
futures = [pool.submit(collect_one, s) for s in servers]
for fut in concurrent.futures.as_completed(futures):
result = fut.result() # never raises: collect_one caught everything
sink.write(result) # failures are written too; they are data
Using as_completed means you hand each result to the sink the moment it is ready, rather than waiting for the slowest server before writing any of them. When the with block exits it waits for the pool to drain, so the cycle is provably done, and provably bounded by ceil(1000 / POOL_SIZE) * PER_SERVER_TIMEOUT even in the all-timeout worst case.
One caveat worth saying aloud, because a sharp interviewer will reach for it: a thread pool cannot forcibly cancel a task that has already started running. That bound holds only because the timeout lives inside fetch_metrics. If you leaned on future.result(timeout=...) instead, the timeout would fire on the waiter while the hung request kept its worker occupied, and the with block would still block on shutdown. Pushing the timeout down into the network call, not the future, is what actually caps the cycle. [Source: darkinterview.com]
The scheduler runs a cycle, then sleeps only for the remainder of the 10-minute window, which keeps the cadence steady without ever launching a second cycle on top of a still-running one.
CYCLE_SECONDS = 600 # every 10 minutes
def scheduler(servers: list[str], sink) -> None:
while True:
start = time.monotonic()
run_cycle(servers, sink) # blocks until this cycle finishes
elapsed = time.monotonic() - start
remaining = CYCLE_SECONDS - elapsed
if remaining > 0:
time.sleep(remaining) # align to the 10-minute boundary
else:
log_warning(f"cycle overran by {-remaining:.1f}s") # do not stack cycles
Python's GIL scares people off threads, but it is a non-issue here. The GIL is released while a thread waits on a blocking network call, so the collections genuinely overlap. Threads are the right tool precisely because the work is I/O-bound. If a collection were CPU-heavy, say parsing enormous payloads, you would move that part to a process pool, but the fan-out itself stays on threads.
Having designed and coded the workers, expect the interviewer to keep pressing on them. These are the questions that come next, and good answers to them are where the round is won. [Source: darkinterview.com]
Size the pool from the budget, not by feel. Worst-case cycle time is roughly ceil(N / P) * T. With 1000 servers and a 5-second timeout, a pool of 50 caps the worst case near 100 seconds, leaving huge headroom under 600. You could shrink the pool and still fit, but a little headroom absorbs a slow tail. The ceiling is real, though: too many threads waste memory, and a larger pool means more requests in flight at once, which can saturate the monitor host's network link or file-descriptor budget and hit the targets with more simultaneous load than they want. Tune P to the slow tail and the timeout, not to the average collection time.
Do not start overlapping cycles; a second wave doubles the load on servers that are already the reason you are behind. Either wait for the current cycle and align to the next boundary, as the scheduler above does, or explicitly skip the missed window and alert. A persistently overrunning cycle is a capacity signal: raise the pool size, lengthen the timeout budget, or split the fleet.
Handled by the two mechanisms already in place: the bounded per-server timeout caps the cost of a hung target, and per-task error isolation turns a failure into a recorded MetricResult rather than a lost worker. The monitor should treat repeated failures for a server as a first-class metric, not swallow them silently. [Source: darkinterview.com]
If writing or forwarding results is slower than collecting them, workers should not block indefinitely on the sink. Bound the sink or batch writes so a slow downstream does not stall the cycle and eat into the window.
One collector process comfortably handles 1000 servers, so keep this brief. If the fleet grew by orders of magnitude, you would shard the server list across several collector instances, each running this same worker-pool design over its slice. Scale is not what this round is about, and saying so is the right instinct.
Spawning one thread per server. A thousand threads waste memory, thrash the scheduler, and give you no tuning knob. A bounded pool draining a queue is both cheaper and adjustable. [Source: darkinterview.com]
Omitting the per-collection timeout. Without it, one hung server holds a worker forever, and enough dead servers stall the whole cycle. The timeout is what makes the 10-minute budget provable, not optional.
Letting one task's exception abort the cycle. A refused connection or bad payload must become a recorded failure, not a crash. Catch per task so no single target can take down the collection.
Launching overlapping cycles. Starting a new cycle while the last is still running piles load onto already-struggling servers. Wait for the cycle to finish, or skip and alert. [Source: darkinterview.com]
Treating it as CPU-bound. Sizing the pool to the core count starves an I/O-bound workload. The workers spend their lives waiting on the network, so scale with concurrency, not cores.
Over-drawing the architecture. Spending the round on a full component diagram misreads the prompt. There is no whiteboard and no traditional component design here; the interviewer wants the worker model and the code.
Framing [Source: darkinterview.com]
Worker Design
The Code
ceil(N / P) * TTrade-offs [Source: darkinterview.com]
| Concern | Decision |
|---|---|
| Round shape | A "system design" slot with no whiteboard; almost entirely worker design plus writing the multithreaded code |
| Workload nature | I/O-bound network fan-out, which justifies many more workers than cores |
| Concurrency model | Bounded thread pool draining a shared queue of 1000 targets, not a thread per server |
| Timeout | A bounded per-server timeout, the decision that makes the 10-minute budget provable |
| Failure handling | Each task catches its own errors; a failure is a recorded result, not a crash |
| Worst-case cycle time | About ceil(N / P) * T; 1000 servers, 50 workers, 5s timeout gives ~100s, well under 600s |
| Scheduling | Run every 10 minutes, wait for the cycle to finish, and never launch overlapping cycles |
| Scale | One collector handles 1000 servers; shard across instances only if the fleet grows far larger |