truenas best practices passthrough

Can GPU passthrough work on a TrueNas Core VM? Or should I switch to Scale?

reddit.com › r › truenas › comments › 1dzezyv › can_gpu_passthrough_work_on_a_truenas_core_vm_or

I just did this! It wasn't too bad. In my case I passed an NVidia P400 through to Docker in an Ubuntu 20.04 VM. The version of Ubuntu may matter for old GPUs. Here's my summary approach: 1) Prevent TrueNAS from using the device. 1a) From the TrueNAS shell, find the device: pciconf -lv You should see something like the following: vgapci1@pci0:131:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb3 subvendor=0x10de subdevice=0x11be vendor = 'NVIDIA Corporation' device = 'GP107GL \[Quadro P400\]' class = display subclass = VGA none146@pci0:131:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x0fb9 subvendor=0x10de subdevice=0x11be vendor = 'NVIDIA Corporation' device = 'GP107GL High Definition Audio Controller' class = multimedia subclass = HDA You will need both the GPU device and the audio device. Note the IDs in the first part of each entry: vgapci1@pci0:131:0:0 none146@pci0:131:0:1 1b) Set system tuneables to exclude the device Add vmm_load with a value of YES Add ptdevs with a value of 131/0/0 131/0/1 (that is, the IDs found in step 1a above, separated with a space) 1c) Reboot TrueNAS 2) Prepare the VM OS (necessary for old GPUs) 2a. Once your Ubuntu VM is up and running, head to the shell and CD into /etc/modprobe.d/ cd /etc/modprobe.d/ 2b. Create a new file: sudo nano blacklist-nvidia-nouveau.conf 2c. Add the following to the new file then save/close: blacklist nouveau options nouveau modeset=0 2d. Create another new file: sudo nano nvidia.conf 2e. Add the following to the new file then save/close: options nvidia NVreg\_OpenRmEnableUnsupportedGpus=1 2f. Update kernel init ram fs: sudo update-initramfs -u 2g. Reboot the VM. 3) Add the GPU devices to the VM 3a. Shutdown the VM 3b. For each device: Click Devices, Add, Change the device to PCI Passhthrough, select the device. I believe the order needs to be unique. 3c. Reboot the VM 4) Install the NVidia drivers and Utilities sudo apt install nvidia-driver-460-server sudo apt install nvidia-utils-460-server 5) Verify that the VM OS can access the GPU. nvidia-smi will return a table of output if the GPU is found. lspci | grep NVIDIA nvidia-smi The GPU is now available to the VM. Continue if you want to pass the GPU through to Docker. 6) Install Docker 7) Install the NVidia Container Toolkit distribution=$(. /etc/os-release;echo $\text{[math]}$ VERSION\_ID) \\ && curl -s -L [https://nvidia.github.io/libnvidia-container/gpgkey] \\ (https://nvidia.github.io/libnvidia-container/gpgkey) | sudo apt-key add - \\ && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit 8) Verify that the GPU can be accessed from Docker. sudo ctr image pull docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 sudo ctr run --rm -t \\ --runc-binary=/usr/bin/nvidia-container-runtime \\ --env NVIDIA\_VISIBLE\_DEVICES=all \\ docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 \\ cuda-11.0.3-base-ubuntu20.04 nvidia-smi I am deeply indebted to craftycraft who covered most of this in the TrueNAS community thread listed below. I only had to make minor modifications to his approach in order to make this work for me. Resources: https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/nouveau.html https://www.truenas.com/community/threads/guide-nvidia-quadro-p400-gpu-passthrough-to-vm-and-docker-for-jellyfin-transcoding.104750/ https://ubuntu.com/server/docs/nvidia-drivers-installation Answer from raw65 on reddit.com

TrueNAS Community

truenas.com › forums › virtualized truenas

Passthrough for SATA drives | TrueNAS Community

August 6, 2022 - If you are serious about PCI Passthrough, you really should get a proper HBA card. The LSI 9211-8i (SAS 2008 family) is excellent for this application. Passing through USB drives may not be the best option for stability. Cheers, ... TrueNAS Core 13.0-U6 HP DL380 G9 1@E-2630Lv4 192GB ECC DDR4 RAM 4@1GigE LAG (Access) 2@10GigE iSCSI (Round-Robin) 9300-8i 9300-8e 2 @ Intel 3700 16GB SSD (SLOG) - spinning rust pool 35TiB production pool (2x6 RaidZ2, SSD Mirrored ZIL) 36TiB bigbackup pool (2x2 Mirror) 910GiB SSD fast pool (2x2 Mirror)

reddit.com › r/truenas › can gpu passthrough work on a truenas core vm? or should i switch to scale?

r/truenas on Reddit: Can GPU passthrough work on a TrueNas Core VM? Or should I switch to Scale?

July 9, 2024 -

I’ve heard it both ways from articles, forums, and other posts, but most of them are several years old.

I originally went with Core due to hearing it’s more stable and easier for beginners. I’m in my first month of usage and feel like I made a mistake. I’d like to pass through my GPU to transcode media for premiere proxies and Plex, but am having a tough time considering my build. Not sure if I’m doing something wrong or if it’s just that Core still doesn’t support hardware transcoding well enough.

My Build:

Motherboard: Asus Rog Zenith Extreme Alpha X399 CPU: Ryzen Threadripper 2970wx (24 Core, 4.2GHz) GPU: 2080TI Second GPU: Radeon Pro WX2100 RAM: 128GB Dominator Platinum DDR4 3200Mhz

These parts (aside from the second Gpu I added) were from my first computer and I just reused them as I experiment with my first server. As I’m new, there were several things I didn’t take into account. Threadripper has no iGPU, so I added the second GPU in hopes I could use that as the main GPU and then passthrough the original one to the TrueNas Core VM (Hypervisor is Proxmox).

So… can I do this? Or do I have to upgrade to Scale? I’ve heard a couple people have issues upgrading and I was afraid that my inexperience may put me in the that camp. I would hate to mess something up or lose data. It also would suck to lose my jails, but if this is the only way to get transcoding, I might just need to figure it out. Any thoughts or resources that a newbie could understand would be extremely appreciated.

Top answer

1 of 3

I just did this! It wasn't too bad. In my case I passed an NVidia P400 through to Docker in an Ubuntu 20.04 VM. The version of Ubuntu may matter for old GPUs. Here's my summary approach: 1) Prevent TrueNAS from using the device. 1a) From the TrueNAS shell, find the device: pciconf -lv You should see something like the following: vgapci1@pci0:131:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1cb3 subvendor=0x10de subdevice=0x11be vendor = 'NVIDIA Corporation' device = 'GP107GL \[Quadro P400\]' class = display subclass = VGA none146@pci0:131:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x0fb9 subvendor=0x10de subdevice=0x11be vendor = 'NVIDIA Corporation' device = 'GP107GL High Definition Audio Controller' class = multimedia subclass = HDA You will need both the GPU device and the audio device. Note the IDs in the first part of each entry: vgapci1@pci0:131:0:0 none146@pci0:131:0:1 1b) Set system tuneables to exclude the device Add vmm_load with a value of YES Add ptdevs with a value of 131/0/0 131/0/1 (that is, the IDs found in step 1a above, separated with a space) 1c) Reboot TrueNAS 2) Prepare the VM OS (necessary for old GPUs) 2a. Once your Ubuntu VM is up and running, head to the shell and CD into /etc/modprobe.d/ cd /etc/modprobe.d/ 2b. Create a new file: sudo nano blacklist-nvidia-nouveau.conf 2c. Add the following to the new file then save/close: blacklist nouveau options nouveau modeset=0 2d. Create another new file: sudo nano nvidia.conf 2e. Add the following to the new file then save/close: options nvidia NVreg\_OpenRmEnableUnsupportedGpus=1 2f. Update kernel init ram fs: sudo update-initramfs -u 2g. Reboot the VM. 3) Add the GPU devices to the VM 3a. Shutdown the VM 3b. For each device: Click Devices, Add, Change the device to PCI Passhthrough, select the device. I believe the order needs to be unique. 3c. Reboot the VM 4) Install the NVidia drivers and Utilities sudo apt install nvidia-driver-460-server sudo apt install nvidia-utils-460-server 5) Verify that the VM OS can access the GPU. nvidia-smi will return a table of output if the GPU is found. lspci | grep NVIDIA nvidia-smi The GPU is now available to the VM. Continue if you want to pass the GPU through to Docker. 6) Install Docker 7) Install the NVidia Container Toolkit distribution=$(. /etc/os-release;echo $ID$VERSION\_ID) \\ && curl -s -L [https://nvidia.github.io/libnvidia-container/gpgkey] \\ (https://nvidia.github.io/libnvidia-container/gpgkey) | sudo apt-key add - \\ && curl -s -L [https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list](https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list) | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit 8) Verify that the GPU can be accessed from Docker. sudo ctr image pull docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 sudo ctr run --rm -t \\ --runc-binary=/usr/bin/nvidia-container-runtime \\ --env NVIDIA\_VISIBLE\_DEVICES=all \\ docker.io/nvidia/cuda:11.0.3-base-ubuntu20.04 \\ cuda-11.0.3-base-ubuntu20.04 nvidia-smi I am deeply indebted to craftycraft who covered most of this in the TrueNAS community thread listed below. I only had to make minor modifications to his approach in order to make this work for me. Resources: https://docs.nvidia.com/ai-enterprise/deployment-guide-vmware/0.1.0/nouveau.html https://www.truenas.com/community/threads/guide-nvidia-quadro-p400-gpu-passthrough-to-vm-and-docker-for-jellyfin-transcoding.104750/ https://ubuntu.com/server/docs/nvidia-drivers-installation

2 of 3

When I researched this sometime last year, passthrough involved a script to get it to work. I don't remember all the details now but I decided to go with Scale because passthrough was a simple drop-down to enable in the app. Was much easier than messing with a script but there was that learning curve going from jails to apps. And now TrueNAS is moving away from true chart apps and going to docker, which i personally prefer. Passthrough in docker is easier too if I remember correctly. With that said, i suggest getting a separate drive to use as boot drive and install Scale to that. Play around with it and compare. That's how I did it with mine, so that if Scale didn't work I could just go back to Core easy peasy. Of course, make sure you back up your configuration so you have it in case something goes wrong.

Videos