Laptop GPU buying advice for FOSS AI with Linux

TL;DR

Disabling Secure Boot in the bios is a requirement for Nvidia. A Secure Boot with custom keys enabled is best. The best laptop cards that exist are 16GBV and are the 4090 or 3080Ti (regular 3080 is 8GBV).

This post is for someone who already knows a desktop PC for FOSS AI is better, but for whatever personal reason still needs a laptop. I am no expert here. I just got out of this rabbit hole in mid July 2023. Maybe I can help shorten your journey. All I can do is write to my former self and tell them what I wish I had known from the start.

Generally speaking, many people say that Nvidia “just works” and AMD is problematic for FOSS AI. I imagine this may be more of a windows user tint, but it also seems to be an issue with tools written for CUDA, and what exact hardware is supported by the HIPS project which aims to bridge ROCm and CUDA. Supposedly AMD is putting a lot of effort into catching up to Nvidia, but I am still unclear about what is going on in this space. Regarding AMD GPUs, on the Level 1 Techs YT channel there have been a few videos where they mentioned that AMD was only working on support for their 7 series consumer grafe cards and the enterprise stuff when it comes to ROCm and HIPS. I tend to trust their advice. I have seen this info elsewhere too, but not experienced it directly.

As far as laptops, the most powerful AMD mobile GPU I could find is the Radeon RX 6850m with 12GBV RAM. I have no idea what this is capable of when it comes to FOSS AI. I can only say, this appeared to be the most powerful laptop GPU option from AMD. This was offered in 2022 on a machine or two but they appear to be a rare outlier product. There is also a somewhat obscure Asus ROC laptop with an external Radeon GPU setup. This hardware has a lot of negative reviews, but it also seems to fetch top dollar on eBay. I’m skeptical of reviews in this space because there is such an enormous range of user skill levels and competence.

I ended up looking into Nvidia even though I really didn’t want to. This is a messy space in practice too. It is far more nuanced than many lead on. So as far as OEM hardware running Linux out of the box, your only real option is System76. It is pricey for the budget minded. Here’s the deal though, sure you’re getting the best new hardware options, but really, the thing no one will tell you to look at and contrast with the DIY side, is that you’re getting the whole bootloader firmware to kernel interface figured out for you, and they are securing it. If you get any other laptop, this is a mess. First of all, this is a world of emotional marketing nonsense where no one will give you the details about their hardware in a way you can compare and contrast your buying options with depth.

In practice, the most important factor is the bios feature to turn off secure boot for UEFI. If you can do this, at least you will be able to run most Linux distros with unprotected firmware for the machine, but you will be able to get the Nvidia binary turd to run with CUDA. This is the bare minimum and is still shit hardware gate keeping and theft of ownership by the OEM.

The worst case scenario would be a situation where you can not turn off secure boot and have a Nvidia GPU. The issue here is that all kernel code must be cryptographically signed by the shitty feudal big tech overlords to run under secure boot. The OEM will have the top level Package Key (PK), and there will be a couple from Microsoft for running Windows that exist at a lower level. You need to generate a key under the same level as Microsoft’s in order to sign your own kernel and run your own stuff however you want. The OEM is never going to issue you a key certificate to exist under their PK, so the only option is to make your own keys for the entire hierarchy. After making custom keys, there is a mechanism available to generate new Windows keys too.

What is secure boot anyways? It is a shitty scheme to try to secure proprietary firmware on the motherboard instead of open sourcing it so that it can be easily checked and verified. The UEFI standard for Secure Boot doesn’t explicitly create this condition. The full specification has the functionality required to allow the user full access to replace the entire key hierarchy, but there is no requirement for the OEM implementing the full specification. In practice, usually only enterprise level hardware has the full functionality enabled for the user to control Secure Boot.

The Linux kernel does not support this shitty Secure Boot scheme at all. There are distros like Fedora, that use a package called lockdown that enables them to work even with Secure Boot. These distros can work even if Secure Boot can not be disabled by the user. There is a software package called a Shim that is used to add a set of cryptographic keys that sign the Linux kernel and enable it to run without modifying the keys for secure boot. This is possible because Microsoft has signed these keys for the respective Linux distros. Yeah. I know. Great idea right.

“Okay,” you’re saying. “So I can still run Linux even if I can’t mess with secure boot. So what is the problem?” Well, you know about those shitty nvidia binary turds you need to enable? That whole reason we don’t like nvidia thing? Yeah, those must be enabled and run as kernel modules. I’m not aware of any distro that offers those modules pre-signed with the Microsoft 3rd party key. Without a signature, you can’t run the proprietary binary. So you can’t use CUDA. None of the Nvidia alternatives are possible for FOSS AI, you need that binary. “But what about nvidia open sourcing their drivers?” They did open sourced the Kernel side. This is relevant to AI stuff, but you still have the problem of an unsigned kernel module that will always be rejected by Secure Boot. This is why it is important to be sure the machine can at least disable secure boot. “So what this is Linux, everything is a file, surely I can just Stack Exchange my way to a fix here too, right?” I’m no kernel hacker. If you’re reading this, I’m betting you’re not a kernel hacker either. Neither of us is likely to even know the difference between ANSI and GNU C. The Shim and Lockdown exist before the kernel so good luck hacking that one. It is likely your OEM private keys are secured in a TPM microcontroller where they are never externalized.

Anyways, the best case scenario is if you can source a machine with a bootloader that allows secure boot to work with a set of custom keys. This process is somewhat tedious but it will make it possible for you to run any Linux, Windows, or other/custom Kernel without the potential of boot/root kits getting into the proprietary (theft) turd running on the motherboard. Boot/Root kits are a threat to anyone running UEFI. It is not obscure or some Windows only thing. If you are forced to run a machine with secure boot disabled, you should probably keep it behind a whitelist firewall.

So, we covered the gritty details. If you are still interested in nvidia machines. The GPU RAM is what determines how big of a batch, and your maximum resolution you can run without tiling. The speed of the GPU will determine your iteration times. The laptop GPU with the most available video RAM is the nvidia 3080Ti, specifically the Ti variant. This has 16GBV of DDR6 RAM, The 4090 also has 16GBV and is faster, but I don’t think these have made it into the wild yet and the price is $3k5+. The 3080Ti is available on several 2022 model gaming laptops. Second hand or NOS, these can be found for $1500-$2000. A good stable diffusion image takes less than 10 seconds to generate on a 3080Ti. I can generate a batch that makes 5 images in 3 sets for 15 total in around 1 minute and it only uses 80% of the available VRAM according to Automatic1111. The fans are not wild and the temperature change is minimal. My last piece of advice is to get a monster size NVME drive, and make sure it is fast, an SSD is not fast enough. Modern machines also have much faster internet. my old fourth gen i7 was doing fine for me, but I only got 5-8mbs on my network. The Aorus YE5 gets 20-28mbs on the same Ethernet connection. This laptop is fine except the inability to set custom keys for secure boot. I learned the hard way. It is possible to set up the custom keys, but when secure boot is reinstated, the custom keys are flushed and new keys are automatically created. If you do this, when you make the custom keys, you’re also destroying the shim if you already installed a Linux distro. This means you’ll either need to always run Linux with secure boot off, or you’ll need to reinstall Linux to apply a new shim. This laptop also has several features built into the firmware controls that are only available on Windows. I am not at all impressed with Gibabyte for how they steal ownership with proprietary nonsense, but I did not find a remotely competitively priced option that could satisfy my requirements. Someone really should be making an utilitarian Linux dev machine with AI in mind and include a 16GBV or larger GPU without all the extra gaming spec hardware and get the price competitive with old stock trailing edge hardware. This is their real competition, not the bleeding edge stuff that OEM Linux machines are aligned with IMO.

You don’t need to spend $2k. You don’t need to have 16GBV to mess with AI. It will enable you to do more and play with bigger models. All of the software used with AI is running on the computer as a server that can be accessible across your home network. This means it is very cost effective to get a cheaper Linux laptop if you need one, and then get a separate PC tower to run a regular PCIE graphics card. This is likely the better priced option, and it opens up the possibility of running an even larger GPU.

A few other really nice features in this space are dual NVME drive slots, and a GPU MUX that can be manually configured in the bios to set which GPU is displaying output on the laptop screen.

While I haven’t confirmed this myself I’ve seen several people mention that most LLMs generally need twice the VRAM of the source weights to run with good performance. So a 7B model for something like Falcon should run well on a 16GBV GPU.

High level description of intent of Secure Boot in UEFI (PDF):

uefi.org/…/UEFI_Secure_Boot_in_Modern_Computer_Se…

Why and how to set keys and use secure boot (PDF):

…defense.gov/…/CTR-UEFI-Secure-Boot-Customization…

Hope this helps someone.

Edit:formatting