Why AI Infrastructure Is Moving to Rust

AI teams are not dropping Python. They are replacing what sits underneath it. Over the past two years, the tokenizers, model file formats, inference servers, data tools and coding agents that AI products run on have been rebuilt in Rust, a systems language that gives C-level speed without C’s memory bugs.

Cloudflare wrote its own LLM inference engine in Rust. OpenAI rewrote its Codex coding agent in Rust and then bought the company behind uv and Ruff, two Rust-built Python tools. Hugging Face’s tokenizers and the safetensors model format both have Rust cores.

The reasons are practical: AI infrastructure is expensive to run, it handles untrusted files and prompts, and governments now tell software makers to use memory-safe languages. Rust addresses all three while letting data scientists keep writing Python on top. Below is what is moving, why, what is not moving, and what it means if you are hiring the engineers who build this layer.

What “moving to Rust” actually means

Almost nobody is retraining data scientists to write Rust. Model code, notebooks and experiments stay in Python. What changes is the layer that does the heavy lifting: the code that splits text into tokens, loads model weights, batches requests onto GPUs, crunches data frames, searches vectors and installs packages.

That layer used to be C or C++ wrapped in Python. Increasingly it is Rust wrapped in Python. A data scientist who types pip install tokenizers, polars or uv is running Rust without writing any. The bindings come from a library called PyO3, which lets Rust code show up as a normal Python package.

The other big change is at the edges. Coding agents, command line tools and inference servers that used to be written in Node.js or Python are being rewritten as single Rust binaries.

Reason 1: Speed without a garbage collector

AI infrastructure is mostly a cost problem. GPUs are the expensive part, and anything that wastes CPU time or keeps a GPU waiting costs real money. Python is an interpreted language with a global interpreter lock, which makes it slow for the tight loops that sit around model calls. Languages like Go and Java avoid that but add garbage collection pauses. Rust has neither: it compiles to machine code and frees memory at known points, so its performance is fast and predictable.

Cloudflare’s Infire is the clearest public example. Cloudflare had been serving most of its models with vLLM, the popular open-source Python inference server, but found it could not get full use out of GPUs spread across its edge network. It also had to run vLLM inside a gVisor sandbox for security, which added more overhead. So it built its own engine in Rust, and says it is writing much of its new infrastructure in the language.

In Cloudflare’s benchmark, Infire was up to 7% faster than vLLM on an idle machine. The bigger gain was CPU: Infire needed a quarter of one core, while vLLM used 1.4 cores on bare metal and 2.5 cores inside the sandbox. Cloudflare reports GPU utilization above 80%, and because Infire runs as a trusted process, it no longer steals CPU from the other services on the same server. Infire also uses Hugging Face’s Rust tokenizers library for the step that turns text into tokens.

OpenAI’s tiktoken tokenizer, used to count and encode tokens for GPT models, follows the same pattern: a Rust core with a Python interface. OpenAI says it runs 3 to 6 times faster than a comparable open-source tokenizer.

Reason 2: Memory safety is now a security requirement

Memory bugs, such as buffer overflows and use-after-free errors, have been the largest single source of serious security holes in software written in C and C++. Rust’s compiler blocks most of them before the code ever runs.

Google has the longest public record. After it started writing new Android code in memory-safe languages around 2019, memory safety bugs fell from 76% of Android’s vulnerabilities to 24% in 2024, and below 20% in 2025. Google says its Rust code has roughly 1,000 times fewer memory safety bugs per line than its C and C++ code, and that Rust changes need fewer revisions.

Governments have taken notice. In June 2025 the NSA and CISA published joint guidance telling software makers, especially those serving national security and critical infrastructure, to plan for memory-safe languages. In December 2025, Linux kernel maintainers declared the Rust experiment over: Rust is now a permanent part of the kernel, Android 16 devices already ship a kernel memory subsystem written in it, and the graphics driver maintainers said they were about a year from requiring Rust for new drivers.

AI adds its own risks. Model weights were long shared as Python pickle files, which can run arbitrary code the moment they are loaded. Hugging Face, EleutherAI and Stability AI backed safetensors, a Rust-built format that stores only data, and had it audited by Trail of Bits, which found no flaw allowing code execution. It has since become the default way Hugging Face’s libraries save models. For an AI service that loads files and prompts from strangers, a smaller attack surface matters.

Reason 3: Python developers keep Python

Earlier attempts to replace Python in machine learning asked researchers to learn a new language. Rust adoption has worked because it does not. Each of the Rust projects below ships as a Python (or JavaScript) package, so the person using it never sees Rust. The team that maintains the package gets speed and safety, and everyone else just gets a faster library.

Reason 4: Agents and AI tools want single, sandboxed binaries

When OpenAI open-sourced its Codex CLI coding agent in April 2025, it was written in TypeScript on Node.js. By June it had announced a rewrite in Rust. Its stated reasons map neatly onto what AI agents need: no Node.js install for users, native access to operating system sandboxing (Landlock on Linux) so the agent cannot touch what it should not, lower memory use with no garbage collector, and an existing Rust implementation of the Model Context Protocol for connecting to tools. The npm package is now a thin wrapper that downloads the Rust binary.

In March 2026, OpenAI agreed to acquire Astral, maker of uv, Ruff and ty, and moved the team into Codex. All three are written in Rust and run many times faster than the older Python tools they replace. In the 2025 Stack Overflow survey, uv was the most admired technology tag. An AI agent that installs packages, lints and type-checks code thousands of times a day benefits directly from tools that finish in milliseconds.

Developers want to write it

Rust has topped Stack Overflow’s admired list for years, and the 2026 survey keeps it there among widely used languages. Its admired score has eased from 83% in 2024 and 72% in 2025 to 67% in 2026 as more people use it for everyday work, while its usage moved up to 12th place overall.

Inside companies, the 2025 State of Rust Survey of 7,156 developers found 48.8% of represented organizations make non-trivial use of Rust, up from 38.7% two years earlier. 55.1% of respondents use it daily, and 84.8% of those using it at work say it helped their company meet its goals.

What is slowing the move down

Rust is not taking over AI wholesale, and anyone planning a team around it should know the limits:

• Training stays in Python, C++ and CUDA. PyTorch, GPU kernels and most training code are not moving. Rust’s gains are in serving, data handling and tooling.
• vLLM and other Python servers still lead. Cloudflare built Infire for an unusual setup, a global edge network with tight security rules. Most teams running a few large GPU servers still use vLLM or similar engines.
• The learning curve is real. Rust’s ownership rules take time to learn, and slow compile times are the top frustration in the State of Rust Survey.
• The talent pool is small and senior. Most Rust developers are experienced engineers who learned it on top of C++, Go or Python. Survey respondents report it is much harder for junior developers to land Rust jobs, which also means fewer people coming up through the pipeline.

Interview questions for Rust infrastructure engineers

1. Explain Rust’s ownership and borrowing rules to someone who writes Python. What bugs do they prevent?
2. Tell me about a time you profiled a slow service. What did you measure, what did you change, and what did it save?
3. Have you exposed Rust code to Python or another language? How did you handle errors and data passing across the boundary?
4. When would you use async Rust, and when would you avoid it?
5. Where have you used unsafe code, and how did you keep it contained and tested?
6. How would you batch requests to a GPU-backed model so the GPU stays busy without making users wait?

Frequently asked questions