To passthrough GPU device from host to Guest OS running by QEMU/KVM the x86 system shall support hardware virtualization VT-x/SVM and IOMMU VT-d/AMD-V.
To make sure that IOMMU was initialized, grep it in dmesg:
[ 0.172074] iommu: Default domain type: Translated [ 0.286258] pci 0000:00:00.2: AMD-Vi: Unable to read/write to IOMMU perf counter. [ 0.286763] pci 0000:00:01.0: Adding to iommu group 0 [ 0.286922] pci 0000:00:01.1: Adding to iommu group 1 [ 0.287136] pci 0000:00:01.2: Adding to iommu group 2 [ 0.287290] pci 0000:00:08.0: Adding to iommu group 3 [ 0.287317] pci 0000:00:08.1: Adding to iommu group 3 [ 0.287514] pci 0000:00:08.2: Adding to iommu group 4 [ 0.287714] pci 0000:00:14.0: Adding to iommu group 5 [ 0.287731] pci 0000:00:14.3: Adding to iommu group 5 [ 0.287901] pci 0000:00:18.0: Adding to iommu group 6 [ 0.287916] pci 0000:00:18.1: Adding to iommu group 6 [ 0.287931] pci 0000:00:18.2: Adding to iommu group 6 [ 0.287947] pci 0000:00:18.3: Adding to iommu group 6 [ 0.287961] pci 0000:00:18.4: Adding to iommu group 6 [ 0.287976] pci 0000:00:18.5: Adding to iommu group 6 [ 0.287990] pci 0000:00:18.6: Adding to iommu group 6 [ 0.288005] pci 0000:00:18.7: Adding to iommu group 6 [ 0.288238] pci 0000:01:00.0: Adding to iommu group 7 [ 0.288280] pci 0000:01:00.0: Using iommu direct mapping [ 0.288311] pci 0000:01:00.1: Adding to iommu group 7 [ 0.288434] pci 0000:02:00.0: Adding to iommu group 8 [ 0.288459] pci 0000:02:00.1: Adding to iommu group 8 [ 0.288483] pci 0000:02:00.2: Adding to iommu group 8 [ 0.288500] pci 0000:03:00.0: Adding to iommu group 8 [ 0.288515] pci 0000:03:01.0: Adding to iommu group 8 [ 0.288529] pci 0000:03:04.0: Adding to iommu group 8 [ 0.288544] pci 0000:03:06.0: Adding to iommu group 8 [ 0.288559] pci 0000:03:07.0: Adding to iommu group 8 [ 0.288584] pci 0000:04:00.0: Adding to iommu group 8 [ 0.288609] pci 0000:05:00.0: Adding to iommu group 8 [ 0.288625] pci 0000:09:00.0: Adding to iommu group 3 [ 0.288641] pci 0000:09:00.2: Adding to iommu group 3 [ 0.288655] pci 0000:09:00.3: Adding to iommu group 3 [ 0.288669] pci 0000:09:00.4: Adding to iommu group 3 [ 0.288683] pci 0000:09:00.6: Adding to iommu group 3 [ 0.288819] pci 0000:0a:00.0: Adding to iommu group 9 [ 0.289041] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
In order to passthrough GPU device to Guest OS running by QEMU/KVM the driver shall be unbind from the device and control shall be passed to VFIO.
The VFIO driver is an IOMMU/device agnostic framework for exposing direct device access to userspace, in a secure, IOMMU protected environment.
However if you try unbind the driver from the device manually you might get the error like this:
# echo -n '0000:01:00.0' > /sys/bus/pci/drivers/amdgpu/unbind # dmesg [ 121.265219] [drm:amdgpu_pci_remove [amdgpu]] ERROR Device removal is currently not supported outside of fbcon
To prevent probing of the device by amdgpu driver and pass control to VFIO right after start of the kernel, command line options shall be added to /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash vfio pci.ids=1002:67df,1002:aaf0"
Don't forget to run sudo grub-mkconfig -o /boot/grub/grub.cfg to pass CMDLINE to grub bootloader.
After reboot grep dmesg to make sure VFIO reserved our device:
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-5.4.0-33-generic root=UUID=e9f2bb67-cbce-4767-97f3-74cc69b57ec9 ro quiet splash vfio-pci.ids=1002:67df,1002:aaf0 vt.handoff=7 [ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-5.4.0-33-generic root=UUID=e9f2bb67-cbce-4767-97f3-74cc69b57ec9 ro quiet splash vfio-pci.ids=1002:67df,1002:aaf0 vt.handoff=7 [ 0.320700] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none [ 0.340830] vfio_pci: add [1002:67df[ffffffff:ffffffff]] class 0x000000/00000000 [ 0.360879] vfio_pci: add [1002:aaf0[ffffffff:ffffffff]] class 0x000000/00000000
Also check lspci to make sure that device was not taken by vendor driver:
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7) (prog-if 00 [VGA controller]) Kernel driver in use: vfio-pci
Now it's time to start QEMU and passthrough the GPU device to the Guest. Unfortunately, virt-manager doesn't allow to run guest with options -vga none so instead run QEMU manually as follows:
qemu-system-x86_64 -enable-kvm -cpu host,kvm=off \ -m 4G -name "VM_GPU" -smp cores=2 \ -device vfio-pci,host=000:01:00.0,x-vga=on,multifunction=on \ -device vfio-pci,host=0000:01:00.1 -nographic -vga none \ -usb -device usb-host,vendorid=0x046d,productid=0xc52b,id=mouse \ -device usb-host,vendorid=0x1532,productid=0x011a,id=keyboard \ -drive file=/var/lib/libvirt/images/ubuntu18.04-clone.qcow2
In the example above USB keyboard and mouse were passed through to the Guest as well.
