Beyond tokens per watt – using Ubuntu 26.04 LTS for AI

Tokens per watt (TpW) – the measure of useful AI work produced per watt of energy consumed – is the metric at top of mind for CEOs, heads of AI, and infrastructure teams alike. With the tremendous cost of GPU clusters, extracting as much value as possible from the expense is critical.

But in the pursuit of tokens, it’s important to remember that hardware efficiency isn’t the only factor influencing data center operating costs, or the output of useful, revenue-generating AI work. While TpW is crucial, we also need to consider time-to-value and the impact of human productivity, which are largely determined at the software level.

We’re shaping Ubuntu to be the software foundation for efficient AI, and in this article, I’ll share some examples of what we mean when we say that we are optimizing Ubuntu for AI. With Ubuntu 26.04 LTS, we’re not just helping organizations get more from their hardware, we’re also making life easier and more productive for teams that rely on and support the AI stack. 

An OS that’s optimized for silicon

How do you squeeze more tokens from your hardware? The prevailing wisdom is to prioritize model optimization, GPU utilization, time to first token, and tokens per second. However, it’s also essential to have a software layer that enables you to make the most of your silicon.

The host operating system plays a central role in the AI infrastructure stack. That’s “central” not just in the sense that it’s important, but also in the sense that it sits in the center of the stack, acting as the bridge between the hardware and software. The OS manages the underlying compute, so it’s responsible for ensuring you can take full advantage of your GPUs and other AI accelerators.

With that in mind, Canonical partners with silicon vendors (such as NVIDIA, AMD, Intel, Arm, and Qualcomm, as well as RISC-V platforms) to optimize Ubuntu across all major architectures. This optimization helps to ensure that the maximum watts are spent on AI workloads rather than OS overhead. 

We also work with partners to certify hardware. By providing standardized, pre-integrated secure boot enablement and firmware delivery, Canonical enables organizations to avoid having to do custom OS engineering for every new piece of hardware they add to their stack. Enterprises can get to value faster, and save on engineering resources.

Single command toolkit integrations

Let’s continue on that theme of accelerating time-to-value and enhancing human productivity. Even in the age of AI, Ubuntu remains a Linux for human beings, and a core pillar of our philosophy is minimizing the friction involved in deploying and operating AI infrastructure for our users.

To that end, we’re collaborating with NVIDIA and AMD to integrate and distribute key AI solutions with Ubuntu. Starting with Ubuntu 26.04 LTS, users can get NVIDIA CUDA, AMD ROCm, and NVIDIA DOCA-OFED each with a single command. 

GPGPU frameworks

NVIDIA CUDA and AMD ROCm are frameworks for general-purpose computing on graphics processing units (GPGPU). They are the critical software layers that enable developers to harness the massive throughput of NVIDIA and AMD GPUs for AI workloads.

Historically, installing these frameworks required multi-step processes, and navigating dependency and compatibility issues could often prove challenging, especially for inexperienced users. But with Ubuntu 26.04 LTS, NVIDIA CUDA or AMD ROCm can each be installed with just one apt install command. 

The new distribution model can save teams hours or even days on GPGPU framework setup, so organizations can start gaining value from GPUs faster. Canonical also ensures that users have smooth upgrade paths, so they can be confident when updating, and get the benefits of the latest features of these platforms.

Have questions about AMD ROCm on Ubuntu? We’ve just published a deep dive.

High-performance networking

For organizations with large-scale AI factories and HPC clusters, NVIDIA DOCA-OFED is among the go-to high-performance networking stacks. However, traditionally, the tradeoff for enabling ultra-low latency and high-throughput data transfers was the complexity of setup and maintenance. System administrators had to manage networking drivers through external installers or complex manual builds, potentially leading to version conflicts or kernel mismatch issues during OS updates.

Now that NVIDIA DOCA-OFED can be installed seamlessly, the entire lifecycle management is simplified. Alongside rapid installation, the new workflow solves common operational pain points like kernel drift, driver incompatibility, and CI breakage following kernel or OS upgrades. Infrastructure teams can deliver speed and stability, while saving resources.

Optimized for hardware and humans

Jon Seager, Canonical’s VP of Engineering, has written recently about the future of AI in Ubuntu. He signs off by stating that “Ubuntu is not becoming an AI product.” But what we are committed to is making Ubuntu an enabler for AI. Whether it’s at the silicon level with deep optimization for every architecture, or at the user level with streamlined toolkit adoption and lifecycle management, Ubuntu is the software layer that underpins an effective AI infrastructure strategy. It can help you get more tokens per watt, and beyond that, it can help you get to value faster and help bring down the operating costs for your stack.

If you’d like to learn more about AI infrastructure best practices, and how Ubuntu can fit into your AI strategy, read the enterprise guide to private AI infrastructure.