With Rust, Cloudflare is Trying to Tackle the Industry’s Inference Bottleneck

Cloudflare has introduced Infire, a new LLM inference engine built in Rust, to run AI workloads on its distributed network.  Unlike hyperscalers that rely upon large centralised data centres packed with expensive GPUs, Cloudflare operates a lean global network that sits within 50 milliseconds of 95% of internet users. That unique architecture demands a more efficient way to serve inference. Mari Galicer, group product manager at Cloudflare, in an interaction with AIM, explained how inference is a different challenge for them compared to hyperscalers. “Most hyperscalers operate large, centralised data centres with nodes dedicated to AI compute, whereas Cloudflare operates a lean, distributed network, with each compute node needing to serve different types of traffic.” “T