Azure GPU Virtual Machines

Choosing a size for self-hosting

Match total GPU memory to the model's footprint, not just parameter count — a 70B model in fp16 needs roughly 140GB of VRAM before accounting for KV cache and batch size, which is why most dense 70B-class models target a 2-4x A100/H100 configuration while a quantized version can fit a single 24GB card. MoE models only need enough VRAM for their active parameters to run at reasonable speed, though the full weight set still has to fit in memory (or be offloaded) somewhere. See the Open Weight Models directory for parameter counts and architecture per model.

NC-series VMs use PCIe GPUs without NVLink between cards in the same VM — fine for independent parallel jobs, but not for training a single model that needs to span GPUs efficiently. ND-series VMs are built for exactly that: every size has NVLink within the VM and InfiniBand between VMs, which is why there's no single-GPU ND option.