LLM Inference Timeout and Restart Strategy

Problem Statement

You are serving an LLM. The inference latency for a single request is a nonnegative random variable T, and you are told only:

E[T] = 1 minute

The interviewer asks a sequence of...