GPUStack

Racks that mix NVIDIA cards with Ascend, Hygon or other accelerators rarely share one management console, and GPUStack was built to give them one. It is an open-source GPU cluster manager released under the Apache 2.0 licence, with its source code on GitHub, and the company behind it also sells an enterprise platform that covers model serving and GPU rental under a single control plane. The site announces version 2.2 as available.

Mixed accelerators under one control plane

The hardware list is long. NVIDIA data-centre cards, AMD Instinct and Radeon parts, and Ascend, T-head, Hygon, MetaX, Moore Threads, Cambricon and Iluvatar accelerators are all named, on x86_64 or ARM64 machines running Ubuntu, Red Hat, openEuler, Kylin or other Linux. Deployment can be on premises, including air-gapped sites, on Kubernetes, or across public clouds, with AWS, Azure, Google Cloud and Alibaba Cloud listed. GPUStack picks an inference engine to match the hardware from vLLM, SGLang, llama.cpp, TensorRT-LLM and MindIE, and can spread a large model over several nodes with tensor or pipeline parallelism.

Model serving and GPU instances

The platform is split in two. The model-serving half manages deployment, traffic weighting, retries and fallback, load balancing, performance modes for throughput or latency, and virtual model routing for upgrades without downtime, and it exposes models through two widely used hosted-model API formats. The GPU-rental half provisions instances with persistent S3 or NFS storage, SSH key injection and Jupyter access, GPU slicing with overcommit, and custom instance templates. Models can be pulled from public model hubs or local files and cover language, multimodal, embedding, reranker, image and speech types.

Quotas, SSO and metering

Enterprise features listed include role-based access and multi-tenancy, sign-in through OIDC, SAML and LDAP, scoped API keys, IP allowlists, per-user token quotas and rate limits, usage analytics, high availability, and billing and metering by token, request or GPU time. Observability covers latency, token rate, queue depth and a topology view of nodes and model instances. The site also publishes its own benchmark results, such as a 135 percent throughput gain on eight H200 GPUs against an untuned baseline, which are company figures rather than independent tests.

Key facts

Product: GPUStack, enterprise AI infrastructure platform Licence: Apache 2.0 for the open-source project Latest version announced: 2.2 Engines: vLLM, SGLang, llama.cpp, TensorRT-LLM, MindIE Deployment: on premises, Kubernetes, multi-cloud Company details: copyright notice reads GPUStack.ai; headquarters and founders are not stated on the pages read