InfiniBand™ networking for Compute virtual machines with GPUs

You can group your virtual machines with GPUs into a GPU cluster. The cluster accelerates high-performance computing (HPC) tasks such as training and inference. These tasks require a lot of processing power that a single VM cannot provide.

The GPU clusters are built with InfiniBand secure high-speed networking. Each GPU in a VM is connected through a network interface card (NIC) that provides 400 Gbps. As a compute VM for GPU clusters consists of 8 GPUs, the total bandwidth for a node is 3.2 Tbps.

Nebius AI Cloud uses GPUDirect RDMA, an NVIDIA technology of remote direct memory access (RDMA) that allows data to flow directly between each GPU and its NIC, avoiding CPU, thus boosting the data exchange speed.

InfiniBand fabrics

Each GPU cluster is created in one of physical InfiniBand fabrics. This is where GPUs interconnected over InfiniBand are located. Each fabric has limited GPU capacity.

When creating a GPU cluster, select an InfiniBand fabric for it. Take into account the type of GPUs you are going to use. For example, if you select fabric-7, you can only add NVIDIA® H200 NVLink with Intel Sapphire Rapids GPUs to this cluster.

Available fabrics and corresponding regions (private regions are marked with *):

Fabric	GPU platform	Region
fabric-2	NVIDIA® H100 NVLink with Intel Sapphire Rapids (gpu-h100-sxm)	eu-north1
fabric-3	NVIDIA® H100 NVLink with Intel Sapphire Rapids (gpu-h100-sxm)	eu-north1
fabric-4	NVIDIA® H100 NVLink with Intel Sapphire Rapids (gpu-h100-sxm)	eu-north1
fabric-5	NVIDIA® H200 NVLink with Intel Sapphire Rapids (gpu-h200-sxm)	eu-west1
fabric-6	NVIDIA® H100 NVLink with Intel Sapphire Rapids (gpu-h100-sxm)	eu-north1
fabric-7	NVIDIA® H200 NVLink with Intel Sapphire Rapids (gpu-h200-sxm)	eu-north1
eu-north2-a	NVIDIA® H200 NVLink with Intel Sapphire Rapids (gpu-h200-sxm)	eu-north2 *
me-west1-a	NVIDIA® B200 NVLink with Intel Emerald Rapids (gpu-b200-sxm-a)	me-west1
uk-south1-a	NVIDIA® B300 NVLink with Intel Granite Rapids (gpu-b300-sxm)	uk-south1*
us-central1-a	NVIDIA® H200 NVLink with Intel Sapphire Rapids (gpu-h200-sxm)	us-central1
us-central1-b	NVIDIA® B200 NVLink with Intel Emerald Rapids (gpu-b200-sxm)	us-central1

In most cases, you do not need to change the preselected fabric. We recommend that you create a GPU cluster in another fabric only if it is better suited for a different platform or if you experience capacity issues with an existing GPU cluster.

Isolation and security of InfiniBand traffic

Nebius AI Cloud isolates InfiniBand traffic between GPU clusters by using InfiniBand partition keys (P-Keys).

Each GPU cluster is assigned a unique P-Key to create isolation inside shared physical InfiniBand fabrics. This way, nodes in different GPU clusters cannot communicate over InfiniBand even if they use the same fabric infrastructure. This creates isolation between tenants without requiring a dedicated physical fabric for each cluster.

How to enable InfiniBand for VMs with GPUs

Web console

1. Create a GPU cluster:

   1. In the sidebar, go to  Compute → GPU clusters.
   2. Click  Create GPU cluster.
   3. On the page that opens, specify the cluster name. It should contain from 3 to 63 characters: lowercase letters, numbers and hyphens.
   4. Select the InfiniBand fabric.
   5. Click Create GPU cluster.

2. Add VMs to the cluster. You can do it only when creating the VMs:

   All virtual machines added to the GPU cluster, including Managed Service for Kubernetes® nodes, must be in the same project.

   1. In the sidebar, go to  Compute → Virtual machines.
   2. Click  Create virtual machine.
   3. On the page that opens, specify the VM’s details and select the GPU cluster name in the GPU cluster list.

You can also create a GPU cluster while creating the first VM in it:

1. In the Computing resources section of the VM creation form:

   1. Select With GPU.
   2. Select a platform and a preset compatible with GPU clusters. For more information, see Types of virtual machines and GPUs in Nebius AI Cloud.

2. In the Boot disk section of the VM creation form, select the boot disk for NVIDIA GPUs. For details, see Boot disk images for Compute virtual machines.

CLI

1. Check that your project ID is saved in the Nebius AI Cloud CLI profile configuration:

   cat ~/.nebius/config.yaml
   

2. If you have not set your project ID as parent-id, or you want to create resources in a different project, get the project ID and update your CLI profile:

   nebius profile update --parent-id <project_ID>
   

3. Depending on your project’s region, select an InfiniBand fabric for VM interconnection and save it to an environment variable:

   export INFINIBAND_FABRIC=<fabric>
   

4. Create a GPU cluster and save its ID:

   export GPU_CLUSTER_ID=$(nebius compute gpu-cluster create \
     --name <gpu_cluster_name> \
     --infiniband-fabric $INFINIBAND_FABRIC \
     --format json \
     | jq -r ".metadata.id")
   

   Where:

   • name: A cluster name that you can use to quickly find the cluster.

5. Create a boot disk optimized for VMs with NVIDIA GPUs:

   export BOOT_DISK_ID=$(nebius compute disk create \
     --name <boot_disk_name> \
     --size-gibibytes 200 \
     --type network_ssd \
     --source-image-family-image-family ubuntu24.04-cuda13.0 \
     --block-size-bytes 4096 \
     --format json \
     | jq -r ".metadata.id")
   

   For compatible boot disk images (--source-image-family-image-family), see Boot disk images.

6. Create a virtual machine with GPUs and specify the GPU cluster ID in its parameters.

   All virtual machines added to the GPU cluster, including Managed Service for Kubernetes nodes, must be in the same project.

   For example:

   nebius compute instance create \
     --resources-platform gpu-h100-sxm \
     --resources-preset 8gpu-128vcpu-1600gb \
     --gpu-cluster-id $GPU_CLUSTER_ID \
     --boot-disk-existing-disk-id $BOOT_DISK_ID \
     ...
   

   Specify a VM platform with GPUs in --resources-platform, and a preset compatible with GPU clusters in --resources-preset. For more information, see Types of virtual machines and GPUs in Nebius AI Cloud.

How to test the connection with the NCCL tests

To test InfiniBand performance in a Compute cluster, you can run the NVIDIA NCCL test in it. For instructions, see our tutorial on running distributed jobs with MPIrun: it uses the NCCL test as an example.

How to delete a GPU cluster

Before deleting a GPU cluster, make sure all virtual machines in the cluster are deleted or moved to another cluster.

Web console

1. In the sidebar, go to Compute → GPU clusters.
2. In the row of the GPU cluster you want to delete, click → Delete.
3. In the window that opens, confirm the deletion.

CLI

1. Get the ID of the GPU cluster you want to delete:

   nebius compute gpu-cluster list
   

2. Delete the GPU cluster:

   nebius compute gpu-cluster delete <GPU_cluster_ID>
   

See also

• How to test a GPU cluster physical state in Compute
• InfiniBand networking for Compute virtual machines with GPUs
• How to create a virtual machine in Nebius AI Cloud
• Running the all-reduce NCCL performance test in Soperator clusters

---

InfiniBand and InfiniBand Trade Association are registered trademarks of the InfiniBand Trade Association.