No Room for Agents at 10ms, a Billion Times a Day

A few months ago Spotify published an engineering post about their new multi-agent architecture for media planning. Specialised agents parsing advertiser intent in parallel, an orchestrator merging the results, a heuristic planner ranking ad sets against historical campaign data, plan generation in 5–10 seconds instead of 15–30 minutes of clicking through forms. It is a good post. It is also, for me, a strange one to read — because two years earlier a team I led at a Tier-1 European telco built the layer that sits underneath an architecture like that, and almost nothing about that layer ever made it into the agentic conversation.

This is not a post about who got there first. Adtech is too old and too crowded for that to be an interesting argument. It is a post about an architectural assumption the agentic conversation tends to leave implicit: that there is a deterministic, measurable, low-latency execution layer sitting underneath the agents, doing the things the agents are too slow to do. We built that layer. It is called Intelliserver. This is a post about what it looks like, why it has to be deterministic, and what changes when you treat that constraint as a feature rather than a limitation.

The shape both posts share

Strip away the vocabulary and Intelliserver and Spotify’s agentic planner share one architectural primitive: specialised units running in parallel, results merged into a single decision.

Spotify calls these units agents. Ours were modules. Their orchestrator is a Google ADK runtime; ours was an executor pool. Their merge step is an LLM finalising a media plan; ours was a ChainMap over per-module result dicts. The shape is the same. In Intelliserver the primitive was concrete enough to be the public surface of the abstract base class every module inherited from:

class Module(ABC):
    @abstractmethod
    def name(self) -> str: ...

    @abstractmethod
    def execute(self, request: RequestDTO): ...

    @classmethod
    def run_parallel(cls, executor, request, modules):
        tasks = [executor.submit(_timed(m, request)) for m in modules]
        return dict(ChainMap(*[t.result() for t in as_completed(tasks)]))

    @classmethod
    def run_sequential(cls, request, modules):
        return dict(ChainMap(*[_timed(m, request) for m in modules]))

run_parallel and run_sequential were first-class siblings. The framework did not care which one a deployment used. What it cared about was that every module had a name, every module had an execute(request), and every result was timed and tagged so the orchestrator could merge them into a single response with per-module latency breakdowns. The signal sources — page context, behavioural cohort, optional identity — were independent on purpose. They could be added, removed, reordered, or run in either mode without the framework knowing.

If you take Spotify’s post and squint, you see the same primitive: a RouterAgent decides which specialised agents to invoke; the specialised agents run in parallel; a final agent merges the results into a media plan. Spotify’s version is more flexible because the LLM can decide which agents to run. Ours was more strict because the deployment decided at startup. But the architectural commitment — parallel, specialised, merged — is the same on both sides. That convergence is not a coincidence. It is what happens when two teams encounter the same problem shape: a decision that depends on multiple independent signal sources, where adding a new source has to be cheap and where the merge logic has to be reasoned about separately from the individual sources. Spotify’s post is worth reading in full.

So the shape is shared. The interesting argument is about what changes when you push that shape into different latency budgets.

The shape that is different

Spotify’s agentic planner has 5 to 10 seconds. Intelliserver had 10 milliseconds at the 95th percentile.

That is not a 1000x difference in the architecture’s surface area. It is a 1000x difference in what is allowed to live on the hot path. At seconds-per-plan, an LLM call per agent is the reason the architecture exists — the reasoning is what the user is paying for, both in time and in money, and one plan generation is a handful of LLM calls billed once. At milliseconds-per-bid, an LLM call per module is not just slow, it is unaffordable: a header-bidding deployment makes billions of decisions a day, and a model call per impression for inventory that monetises at fractions of a cent means the economics do not work even before you account for latency. There is no version of “send the request to a model and wait” that fits inside the budget — neither the latency budget nor the cost budget — of an inline header-bidding decision. The auction is already happening. Prebid has a timeout. Publishers do not care that your reasoning is sophisticated; they care that you answered before the deadline or you did not, and the inventory cleared without you.

This forces a different architectural commitment underneath the shared shape: everything on the hot path has to be deterministic, bounded, and measurable. Each module’s execute either returns a result within a known time budget or it returns nothing and the framework moves on. There is no retry. There is no escalation. There is no fallback to “ask a smarter model.” The fallback is built into the module itself, because by the time you would notice you needed a fallback, the auction is over.

That commitment ages well, and it ages well for a reason that has very little to do with adtech. It ages well because the agentic systems of 2025 and 2026 are also running into this exact constraint at their edges. Anywhere an agent has to interact with a real-time system — fraud scoring, payment authorisation, route selection at packet level, anything inline with a user-visible latency budget — the same architectural truth holds: agents can plan, but execution has to be deterministic. The agentic planning layer and the deterministic execution layer are not interchangeable. They are a stack.

What “deterministic substrate” actually looks like

The cleanest example in Intelliserver was the contextual module. Its job: given a publisher’s page URL and the page’s cleaned content, return the IAB categories most relevant to that page. The honest implementation is an embedding lookup — encode the page with a multilingual sentence-transformer, do a nearest-neighbour search against a pre-built index of IAB-category embeddings, return the top-k. That implementation is correct, but it is also too slow for the hot path. A sentence-transformer encode is tens of milliseconds on CPU, and tens of milliseconds is the entire request budget for the whole request, not just one module.

The shape we settled on was a layered cache with a deterministic miss path:

def execute(self, request):
    content_hash = sha256(request.context.pageContent)
    url_hash = sha256(request.context.url)

    # One round-trip: both lookups pipelined, results returned together.
    pipe = redis.pipeline()
    pipe.hget(content_key, content_hash)
    pipe.hget(url_key, url_hash)
    content_cats, url_cats = pipe.execute()

    if content_cats is not None:
        return content_cats[:limit]
    if url_cats is not None:
        return url_cats[:limit]

    # Cold miss: enqueue the page for offline embedding on the Ray
    # cluster, and return a deterministic URL-token fallback so the
    # caller still gets a real answer this request.
    enqueue_for_offline_embedding(url_hash, content_hash, request.context)
    return url_token_bucket(request.context.url)[:limit]

Three things to notice about this.

1. The model is not on the hot path. A request that has never been seen before does not wait for model.encode. It gets a deterministic answer derived from URL tokens — a coarser answer, yes, but a bounded one — and the real embedding work happens asynchronously on a Ray cluster that drains a bounded Redis queue in batches. By the second impression on any given page, the cache is warm and the answer is the real one. By the thousandth, the model is irrelevant to the request path entirely.

2. The cache key is computed client-side too. The supply-side adapter we shipped into participating publishers’ Prebid wrappers extracted page content in the browser, hashed it, and put both the content hash and the URL hash in the bid request before it ever reached Intelliserver. That meant two impressions on the same page from two different publishers in our network would land on the same cache slot by construction. The cache wasn’t a Bloom filter or a probabilistic fast-path; it was an exact agreement between the browser and the server on what “the same page” means.

3. The fallback is a feature, not a graceful degradation. A naive design would return an empty list on a cold miss and rely on a downstream system to handle it. That is not deterministic — it pushes the indeterminacy onto the caller. The URL-token bucket is a real answer with a real semantic: “we haven’t embedded this page yet, but here’s what its URL structure says about it.” Downstream systems treat it the same way they treat a cache hit, because as far as the architecture is concerned, it is a cache hit — against a coarser key.

The behavioural module followed the same pattern. Cohorts were built upstream in a Data Clean Room — fuzzy-c-means clustering over first-party telco data with a k-anonymity floor, soft membership across cohorts so the signal stayed probabilistic enough to be useful. Crucially, the data was completely and irreversibly anonymised inside the DCR before it ever crossed into Intelliserver. A stack of privacy-enhancing technologies — k-anonymity thresholds, differential-privacy noise injection on sensitive aggregates, secure-aggregation primitives, deterministic salted hashing of any identifiers that survived — ran inside the DCR boundary, and what came out the other side was cohort statistics with no path back to a person. By the time Intelliserver saw the data, re-identification was not a privacy concern that needed mitigating; it was mathematically not on the table. The hot path saw the cohorts only as a Redis lookup keyed on (device_brand, day_of_week, hour_of_day) with a deterministic wildcard fallback (*:dow:hod) for long-tail devices. Same shape as the contextual module: heavy work offline, cheap deterministic lookup online, fallback that is a real answer rather than an absence.

The optional identity layer worked the same way. With an ID (ID5 and a few others), retargeting signals were available; without one, the request flowed through context and cohort alone. The framework didn’t branch on whether identity was present — every module knew how to produce a useful answer with the signal it had, including the identity module, which produced a useful “no-ID” answer when there was no ID. The whole thing was DSP-agnostic by construction; integration testing covered three major DSPs and the integration surface was the same in each case.

The pattern in all three modules is: the expensive, probabilistic, model-driven work happens off the hot path, asynchronously, with its own budget; the hot path is a Redis lookup with a deterministic fallback that is itself a real answer. That is what “deterministic substrate” means concretely. It is not “no ML.” It is “no ML on the request path.” The ML is upstream of the cache, not in front of it.

Why this matters for the agentic layer above

Here is the part that the agentic posts mostly skip.

An agent that generates a media plan for a campaign is, in production, going to be asked to produce plans that execute. The plan is going to be turned into line items, those line items are going to enter an auction pipeline, and the auction pipeline is going to make billions of decisions a day inside a 10-millisecond window each. The agent does not make those decisions. The agent decided what the line items should look like; something else has to decide, in real time and at scale, whether this specific impression on this specific page for this specific user matches the line item well enough to bid, and at what price.

That something else is the substrate.

• If the substrate is brittle, the agent’s plan is theoretical.
• If the substrate is non-deterministic, the agent’s plan cannot be measured — every plan will produce different outcomes for reasons that have nothing to do with the plan.
• If the substrate is slow, the agent’s plan does not execute at the scale the agent assumed when it was generating the plan.

The reason this needs to be said explicitly is that the agentic conversation tends to treat “the rest of the system” as a solved problem. The interesting work is the agents, the prompts, the orchestration. The boring work — the cache layout, the deterministic fallback, the supply-side hashing contract, the per-module timing instrumentation, the observability pipeline that records “what did the system decide and why” for every decision — is treated as plumbing. But the plumbing is what makes the agents safe to deploy, because the plumbing is where you can audit, measure, and bound.

Spotify’s post mentions, almost in passing, that their agentic planner sits on top of a “mostly consolidated backend.” That backend is the substrate. Their post is not about it. Most agentic posts are not about it. Mine is.

Where agentic ideas fit on top

The substrate-vs-agents framing is not a defence of determinism against AI. It is a claim about where in the stack each one earns its keep. Several of the ideas in the recent agentic posts slot neatly onto layers above the hot path without disturbing it.

The demand-side surface is the obvious one. Resolving a natural-language campaign brief into targeting parameters is classically a mix of regex, taxonomy lookup, and brittle prompt engineering. That is exactly what a modern agent does well — parsing intent, asking clarifying questions, grounding answers in tools that return real taxonomies and real inventory. Seconds-per-plan, never touching the hot path. Strict upgrade.

Campaign optimisation is the second. A hand-rolled feedback loop reacts slowly and deterministically; an agent that proposes targeting changes in response to performance signals — with a human in the loop — sits naturally above the substrate. The agent reasons about what to change; the substrate executes at auction speed.

Publisher-facing insight is the third. Turning structured bid signals into a narrative about which inventory underperforms and why is a translation problem, and translation is what agents are good at.

The pattern is the same in all three. The agentic layer earns its keep where being slow and articulate is fine. The substrate earns its keep where being wrong, slow, or unmeasurable costs the business. The job is to know which layer you are in.

Adtech decisioning is the layer where there is no room for an agent at 10ms. That is not a limit. That is the design.