{"id":19,"date":"2025-02-15T20:34:51","date_gmt":"2025-02-15T20:34:51","guid":{"rendered":"https:\/\/homelab.computer\/?p=19"},"modified":"2025-02-19T20:50:03","modified_gmt":"2025-02-19T20:50:03","slug":"qubes-os-setting-up-a-qube-for-gpu-pass-through","status":"publish","type":"post","link":"https:\/\/homelab.computer\/index.php\/2025\/02\/15\/qubes-os-setting-up-a-qube-for-gpu-pass-through\/","title":{"rendered":"Qubes OS: AI Integration for Productivity Environments"},"content":{"rendered":"\n

This guide explains how to enable GPU pass-through using an Nvidia GPU and how to install Ollama.<\/em><\/strong><\/p>\n\n\n\n

Qubes OS does not have built-in AI features. However, you can integrate AI functionality into common productivity applications by running local language models (LLMs) on your system.<\/p>\n\n\n\n

There are advantages and disadvantages to this approach. On the positive side, it allows you to run the AI locally on your hardware, giving you full control over your data. The downside is that it requires significant system resources, especially for larger models. While you can technically run an LLM without GPU acceleration, performance will be poor, limiting model size and drastically slowing down processing times.<\/p>\n\n\n\n

GPU pass-through for AI integration is simpler to configure compared to setups used for gaming. In this scenario, you won’t need a video output from the GPU, eliminating the need to configure xorg or use a dedicated display for the GPU. This also means that a qube designed exclusively for AI integration cannot be used for gaming.<\/p>\n\n\n\n

If you also want to play video games, see this guide<\/a>.<\/p>\n\n\n\n

Before proceeding with GPU pass-through, you must first hide the GPU from dom0. See this guide<\/a> for hiding the GPU.<\/p>\n\n\n\n

Setting up qube<\/strong> <\/p>\n\n\n\n

Create a standalone qube, don\u2019t use a minimal template unless you are familiar with using minimal templates.<\/p>\n\n\n\n

\"\"
Qubes OS: Create qube dialog<\/figcaption><\/figure>\n\n\n\n

In the qube settings disable memory balancing, set the kernel to provided by qube, and the mode to HVM.<\/p>\n\n\n\n

\"\"
Qubes OS: Qube settings menu<\/figcaption><\/figure>\n\n\n\n

In the Device tab, add the GPU you want passed to the qube.<\/p>\n\n\n\n

\"\"
Qubes OS: Qube settings menu, devices tab<\/figcaption><\/figure>\n\n\n\n

Install software requirements<\/strong><\/p>\n\n\n\n

sudo apt install software-properties-common apt-transport-https curl git<\/code><\/p>\n\n\n\n

Install Nvidia CUDA driver<\/strong><\/p>\n\n\n\n

curl -fsSL https:\/\/developer.download.nvidia.com\/compute\/cuda\/repos\/debian12\/x86_64\/cuda-keyring_1.1-1_all.deb<\/a> -o cuda-keyring_1.1-1_all.deb<\/a>
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo add-apt-repository contrib
sudo apt update
sudo apt install nvidia-kernel-dkms cuda-drivers<\/code><\/p>\n\n\n\n

At this point, reboot the system and run nvidia-smi<\/strong>, to confirm the GPU is available, and ready for use.<\/p>\n\n\n\n

Install docker<\/strong><\/p>\n\n\n\n

sudo curl -fsSL https:\/\/download.docker.com\/linux\/debian\/gpg -o \/etc\/apt\/keyrings\/docker.asc
sudo chmod a+r \/etc\/apt\/keyrings\/docker.asc
echo \"deb [arch=$(dpkg --print-architecture) signed-by=\/etc\/apt\/keyrings\/docker.asc] https:\/\/download.docker.com\/linux\/debian \\
$(. \/etc\/os-release && echo \"$VERSION_CODENAME\") stable\" | sudo tee \/etc\/apt\/sources.list.d\/docker.list > \/dev\/null
sudo apt update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo usermod -aG docker user<\/code><\/p>\n\n\n\n

Exit the shell and log back in, to apply user permissions.<\/p>\n\n\n\n

Install Ollama<\/strong><\/p>\n\n\n\n

curl -fsSL https:\/\/ollama.com\/install.sh | sh<\/code><\/p>\n\n\n\n

Now Ollama should be installed and ready for use, test it by running a model.<\/p>\n\n\n\n

ollama run llama3<\/code><\/p>\n\n\n\n

Allowing other qubes to use Ollama<\/strong><\/p>\n\n\n\n

Using qubes.ConnectTCP<\/strong> is a good way to allow other qubes to access the Ollama qube. The two main advantages are that you can run Ollama offline, and because connection is done though a local port, it makes integration easier.<\/p>\n\n\n\n

In dom0, the file \/etc\/qubes\/policy.d\/30-user-networking.policy<\/strong> will control what qubes can Ollama.<\/p>\n\n\n\n

qubes.ConnectTCP +11434 browser-qube @default allow target=ollama-qube<\/code><\/p>\n\n\n\n

Here is an example, browser-qube is accessing ollama-qube<\/p>\n\n\n\n

In the qube that is connecting to Ollama, you can use systemd to automatically initiate the connection at boot.<\/p>\n\n\n\n

In \/rw\/config<\/strong>, create the following two files<\/p>\n\n\n\n

ollama@.service<\/strong><\/p>\n\n\n\n

[Unit]
Description=Ollama service<\/code>

[Service]
ExecStart=qrexec-client-vm '' qubes.ConnectTCP+11434
StandardInput=socket
StandardOutput=inherit
Restart=always
RestartSec=3<\/code><\/p>\n\n\n\n

ollama.socket<\/strong><\/p>\n\n\n\n

[Unit]
Description=Ollama socket
[Socket]
ListenStream=127.0.0.1:11434
Accept=true
[Install]
WantedBy=sockets.target<\/code><\/p>\n\n\n\n

Add the following code to \/rw\/config\/rc.local<\/p>\n\n\n\n

cp -r \/rw\/config\/ollama* \/lib\/systemd\/system\/
systemctl daemon-reload
systemctl start ollama.socket<\/code><\/p>\n\n\n\n

This code til automatically start the Ollama connection when the system boots.<\/p>\n\n\n\n

When the system boots, it binds the local port localhost:11434 to the remote port ollama-qube:11434. To applications running in the qube, it looks like Ollama is running on localhost:11434, and any application that can use Ollama will work out of the box.<\/p>\n\n\n\n

You can repeat this for as many qubes as you need, just add them to the policy file, and set up the systemd files. <\/p>\n","protected":false},"excerpt":{"rendered":"

How to get AI integration in everyday application, like VS Code and web browses, but integrating the Ollama API in Qubes OS.<\/p>\nRead more<\/a>","protected":false},"author":1,"featured_media":66,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,2],"tags":[],"class_list":["post-19","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-pcie-pass-through","category-qubes-os"],"_links":{"self":[{"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/posts\/19","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/comments?post=19"}],"version-history":[{"count":14,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/posts\/19\/revisions"}],"predecessor-version":[{"id":78,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/posts\/19\/revisions\/78"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/media\/66"}],"wp:attachment":[{"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/media?parent=19"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/categories?post=19"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/homelab.computer\/index.php\/wp-json\/wp\/v2\/tags?post=19"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}