Latency Requirements by Workload: Where Data Center Proximity Matters

Latency is the time a signal takes to travel from one point to another and back. For site selection, the part that matters is propagation delay over fiber, because it scales with distance and no amount of hardware can remove it. NetApp’s MetroCluster documentation gives the working rule: latency usually increases by about 1 ms of round-trip delay per 100 km.⁷ That is the floor. Switches, routers, optical equipment and the application itself add more on top.

Two details make the floor higher in practice. First, fiber follows roads, railroads and utility rights of way, so the route between two points is longer than the straight-line distance. A site 80 km from a hub as the crow flies may be 110 km away by cable. Second, traffic may not take the shortest path at all if the carrier’s network routes it through a distant switching point. Both are reasons to ask carriers for route maps and measured round-trip times rather than estimating from a map. Our guide to fiber route diversity and latency covers how to evaluate paths.

New fiber types can shave the floor. Hollow core fiber carries light through an air-filled channel; Ciena has said it is expected to offer 33 percent lower latency than standard single-mode fiber.¹⁰ Prysmian describes it as transmitting data nearly 50 percent faster than glass-core fiber because light moves faster in air.¹¹ Deployments are early and concentrated on short, high-value links, so for screening purposes conventional fiber is the right assumption.

Every workload has a latency budget: the round-trip time it can absorb before performance, user experience or data integrity suffers. The budget, not the workload’s label, is what decides how far from users, partners or a sister site a data center can sit. The table summarizes published figures for the main categories.

These figures are design targets and examples, not universal rules. A chatbot that streams text can hide more delay than a voice assistant, and a batch analytics job may not care about latency at all. Start with the specific application, then work back to distance.

AI changed the latency question because its two main workloads sit at opposite ends of the scale. Analysys Mason describes training as power hungry, relatively latency insensitive and suitable for remote, centralized data centers placed wherever land, water and low-cost energy are available.⁵ McKinsey makes the same point with a number: training workloads are insensitive to latency and can tolerate delays of up to 100 milliseconds between adjacent regions, which lets hyperscalers site them in remote, power-rich areas.⁶

Inference is different. It powers real-time applications such as search, chatbots and recommendation engines, and Analysys Mason notes that it is much more sensitive to latency and needs high availability, which pushes it toward regional facilities or edge deployments.⁵⁶ McKinsey reports that inference is driving build-outs in metro and near-metro sites chosen for low latency, strong network connectivity and energy efficiency.⁸ It projects that inference will surpass training by 2030 to become more than half of all AI compute, while cautioning that the growth path is uncertain.⁶

Data Center Dynamics frames the result as training having built the remote campuses while inference will choose the markets, because inference revenue is recurring and operators are sensitive to performance variability.¹³ BGO’s comparison sizes training sites at roughly 100–500 MW and inference sites at 1–50 MW, with an inference latency target under 50 ms.¹² Treat those as one firm’s planning view. Our guide to AI training vs. inference site needs covers the power and cooling side of the split.

Cloud architecture sets a quieter but firm latency constraint. AWS describes the availability zones in a region as meaningfully distant from each other, up to about 60 miles (about 100 km), so that power loss, fiber cuts, floods or earthquakes do not hit them at once, yet close enough for synchronous replication with single-digit millisecond latency.³ The zones are linked by redundant, dedicated metro fiber.³

Enterprise storage follows similar limits. Dell states that PowerStore metro volumes support a maximum of 5 ms latency and about 100 km (80 miles).² NetApp’s MetroCluster IP guidance requires round-trip latency between sites at or below 7 ms, with distance capped at 700 km, whichever limit is reached first.⁷ In practice, both point to sister sites in the same metro or an adjacent one.

For land, this explains why hyperscale operators buy clusters of parcels around an existing region rather than one isolated site. A new campus 40 km from an existing one can join the same region; one 400 km away usually cannot. When a cloud operator says it wants land “in the region,” it is describing a latency radius as much as a market. Our guide on how hyperscalers choose data center sites explains the rest of their criteria.

User-facing applications are bound by human perception. Nielsen Norman Group’s long-standing guidance holds that a response within 0.1 second feels instantaneous and one within 1 second keeps the user’s flow of thought, even if the delay is noticed.¹⁴ Network round trips are only part of that budget, so services with heavy processing per request leave little room for distance.

Some applications need far less than 0.1 second. AWS built Local Zones to run applications that need single-digit millisecond latency by placing infrastructure closer to end users and business centers.⁴ Its first, in Los Angeles, was aimed at real-time gaming, film production and graphics-intensive virtual workstations, where users needed single-digit millisecond latencies.⁹ AWS later announced more than 30 additional Local Zones worldwide on the same premise.¹⁵

These sites are small and urban, and their siting looks nothing like a gigawatt campus. Power in the low megawatts, zoning that allows a compact building, and proximity to metro fiber matter more than acreage. See edge data center site selection and the edge data center glossary entry for details.

Electronic trading is the clearest example of latency deciding location. Firms place servers in the same data centers as exchange matching engines, and specialized networks compete on fractions of a millisecond between those buildings. In 2016, Quincy Data reported one-way latency of 3.982 ms between CME’s Aurora, Illinois facility and Nasdaq’s data center in Carteret, New Jersey, measured rack to rack over a network provided by McKay Brothers.¹ The same update listed 3.986 ms to the NYSE campus in Mahwah and 4.015 ms to Equinix NY2 in Secaucus.¹

Routes like these often rely on wireless links rather than buried fiber, because signals travel faster through air than through glass, the same physics behind hollow core fiber.¹¹ For site selection, trading is a niche: the relevant locations are fixed by where exchanges already operate, and a new site rarely changes that. It is useful mainly as a reminder that latency budgets can be strict enough to make every other site factor secondary.

The practical method is to work backward from the application to a map. Start with the latency budget, subtract time the network equipment and application consume, convert what remains into route distance, then discount for indirect fiber paths.

A few checks keep the exercise honest:

• Ask which endpoints matter: end users, an existing cloud region, a partner exchange or a sister campus. Each gives a different radius.
• Get measured round-trip times from at least two carriers, not just distance. See fiber connectivity for data center sites.
• Check whether the site reaches an internet exchange point or carrier hotel on diverse routes.
• Do not pay a latency premium the workload does not need. A training campus that insists on metro proximity gives up the power and land advantages of remote sites.⁶

Latency is one input among many, and the site needs by data center type shift with it. If you want a parcel checked against a specific workload’s radius, power path and fiber options, you can get a site reviewed.