Every time I open LinkedIn, I see someone suffering from their local AI setup is “too slow” and without ROI. Someone complaining that their DGX Spark is not fast enough. Someone else has discovered that Intel makes GPUs now... And someone who is trying to stream a huge model from an SSD and calling it a "revolution".
So here is a practical guide to choosing hardware for local AI inference. I am just a nerd, not a hardware engineer, so please be kind if I get something slightly wrong. But this is just what I learned from my own experience.
So let's start with the main lesson that is simple: choose the workload first, then choose the hardware.
-- Inference is not training --
Inference means running an existing model. Training means changing the model’s weights. They are very different workloads.
If you want to train serious models, I would start with cloud compute or a well-supported NVIDIA GPU with optimally +24GB VRAM. NVIDEA is clearly here the go-to with CUDA, fast VRAM, Tensor Cores, and the surrounding software ecosystem make NVIDIA the least painful option for most training workflows. A Google Colab Notebook or RunPod instance can also be a much better deal than building a small data centre under your desk. So if you want to train, maybe start with cloud ressources.
Now Most people do not find the time to create a data set and do a full fine-tune of a model. So mostly the rest of this article will be about interference: running llm, image, audio, or coding models locally.
-- The API-wrapper trap --
The first category of lost LinkedIn Gurus is what I call the API wrapper lostie. This is when somebody buys an expensive computer for “local AI”, but then puts their Claude API key in to use OpenClaw or so one... So actually it is running in a cloud data centre through an API key. It is frightening to see how many of these lost LinkedIn gurus have been out there during the OpenClaw hype.
In that case, the computer is only sending requests and running a few lightweight tools. An old laptop, a cheap Mac mini, or even a Raspberry Pi can usually do that job. The model is not local just because the chat window is on your desk.
If you only need an API client, use what you already own. Save the money for API usage on the best models — or for hardware that actually runs the model (but this will get expensive).
-- The four bottlenecks --
For local AI, four questions matter more than ever:
- Capacity: can the model and its context fit into memory?
- Bandwidth: how quickly can the hardware read that memory?
- Compute: how quickly can it perform the required matrix operations?
- Software: will your model runner, drivers, and libraries work without a three-day debugging adventure?
These four constraints explain most local-AI buying decisions.
-- Storage is not working memory --
AI models are large files containing numbers called weights. Your Harddrive or SSD is where those weights live permanently, but it is not where the calculations to happen.
When a model starts, it needs to be loaded into fast working memory. Streaming weights constantly from an SSD may technically produce output, but it is usually painfully slow and puts unnecessary stress on the drive. “It runs” and “it is usable” are very different claims.
The useful hierarchy is roughly:
registers → cache → RAM or VRAM → SSD → hard drive
The closer memory is to the compute units, the faster it tends to be—and the more expensive and limited it becomes.
System RAM is the normal working memory of your computer. VRAM is the dedicated memory on a GPU. For local AI, VRAM is often the most important number on the entire spec sheet because it determines what can fit on the GPU.
-- Mixture of Experts and DFlash --
If you want to get into Local AI you also need to know about the concepts of mixture of Expert models and DFlash.
So basically there are dense models, which means the whole AI is just one big model, and a mixture of expert model is when there are multiple smaller expert models that build the big AI model. So instead of calculating every parameter of the AI model. The mixture of expert model just calculates a small subset and is therefore much much faster. Here for example you could also just hold the expert model in the memory while the other sits on your hard drive. But again, if there's a flip, this will decrease the speed of your AI by a lot. Also you should keep an eye on technologies like DFlash because they also increase local interference by up to 30%. So always ask ChatGPT what the newest trends on technologies are out there. And maybe if there is some nighlty/experimental builds of vLLM that supports some new technique. Like for example DFlash that is a drafter technique. Here a small model will draft multiple tokens at the same time and then the big model just validates them. You can set the draft count dynamically, if you set it to high the small AI model will do many mistakes and the speedup will be gone again.
To give you some numbers: my local Qwen-27b is a dense model and produced around 70 Token per second. The MoE Model with 35b produced around 150 Token per second. Both are very close to each other on benchmarks. With the Drafter enabled I achieve around 180 Token which is more then fast enough for daily usage. I would assume everything above 70 Tokens as usable.
-- Why bandwidth matters --
A GPU is built to perform many calculations in parallel. Thousands of parallel cores are only useful if they can be fed with data quickly, which is why GPU memory has much higher bandwidth than ordinary desktop RAM.
This is especially important for language-model inference. The hardware repeatedly reads model weights while predicting the next token. If the model fits inside fast VRAM, the GPU can read those weights efficiently. If the model spills into system RAM, performance can drop sharply and if it even spills into your Harddrive, the battle is basically lost and painfully slow.
That is why a five-year-old GPU with 24 GB of VRAM can still be more useful for local AI than a newer card with less memory. More compute is great, but it cannot compensate when the model does not fit.
-- Context length is a hidden cost --
The model weights are not the only thing using memory. A language model also keeps a KV cache, which stores information about the current conversation or prompt.
Short chats may work perfectly, while a coding agent with logs, tool calls, documents, and a long history suddenly uses much more memory. So this is also a question about how long context or conversations you want to have.
So ask more than “Can this run a 70B model?” Ask: at what quantization, context length, and tokens per second? Fitting a model with just one gigabyte left for KV cache will not give you a better experience than using a smaller model, but with a bigger context. Also you can interfere concurrently using for example vllm (more on that later). Here you will also benefit from more free VRAM.
-- Quantization makes local AI possible --
Quantization stores weights with fewer bits: 8-bit, 6-bit, 5-bit, 4-bit, and sometimes lower. This reduces memory use and lets smaller machines run larger models.
The trade-off is quality. Lower precision can mean weaker reasoning, worse coding, or more strange mistakes. A model may technically fit after aggressive quantization, but that does not mean it will be a good experience.
This is why model size alone is not enough. The practical question is how much quality you are willing to trade for speed and capacity. Check the benchmarks and reports of hopefully real humans on Reddit to get an understanding what quality you want to have for your workflow.
There is also now something called NVFP4 quanitization that only has 1% of loss compared to FP8. Older GPUs can store and dequantize 4-bit models but they can't handle these NVFP4 quants. This is something only high end cards like Nvidea Blackwell can actually perform, as the support tensor Core operations on this native block-scaled FP4 data. If you want me to write some article on it, let me know!
-- CUDA cores and Tensor Cores --
CPUs have a small number of powerful, flexible cores. GPUs have many smaller parallel cores, which suits the matrix multiplication used throughout neural networks.
On NVIDIA cards, CUDA cores handle general parallel GPU work. Tensor Cores are specialized for the matrix and tensor operations used by AI. CUDA-core count is therefore useful, but it is not a complete AI performance metric.
-- AMD and Intel --
There is also the AMD and Intel route. Their GPUs can sometimes look extremely attractive because you may get a lot of VRAM or compute for much less money than an equivalent NVIDIA card. The problem is software support.
CUDA is still the path of least resistance for a huge amount of the AI ecosystem. With AMD you may depend on ROCm support, and with Intel IDK even. Some models and runners work perfectly. Others require special builds, specific driver versions, experimental backends, or simply do not work at all.
So I would treat cheap AMD or Intel GPUs very differently from NVIDIA. Do not buy one because the specifications look good on paper. Buy one because you already know that the exact model, quantization, and inference software you want to use has been tested successfully on that exact hardware.
If you enjoy debugging drivers and compiling experimental branches at 2 a.m., this can be a fantastic way to save money. If you just want to download a model and run it, paying the NVIDIA tax can suddenly look surprisingly reasonable.
--A few realistic hardware paths--
Around 8–16 GB of VRAM: good for learning local tools, smaller language models, image generation, embeddings, and experimentation. It becomes limiting quickly for larger models and long contexts.
24 GB NVIDIA GPU: a used RTX 3090 remains an unusually interesting option. It is hot, power-hungry, and old, but 24 GB of VRAM plus CUDA support goes a long way. It is still one of the greates value choices for serious local experimentation! Currently a used one is around 900 euros.
32 GB high-end NVIDIA GPU: a card such as the RTX 5090 offers much faster memory and much more compute than a 3090. It is one of the strongest choices when you want speed, image generation, and broad software compatibility - but it is not a cheap way to buy memory. If you want to run the biggest models, 32GB will be tight.
48 GB or more: professional GPUs can make larger models and longer contexts much easier because everything fits on one card. The cost is substantial, so this is best justified by a real workload or enthusiast. There are also old Nvidea Workstation GPUs like the ADA generation. Those I would not recommend, as they are rare to find and also still very expensive. They are slower then an 3090, so if I would look for more VRAM I would rather invest some more money in some latest Blackwell card or go the unified memory way, that also become crazy expensive.
128 GB unified-memory systems: Macs and machines such as DGX Spark take a different approach. CPU and GPU share one large memory pool, which makes larger models possible in a compact system. The trade-off is that unified memory generally has less bandwidth than high-end dedicated GPU VRAM, so capacity does not automatically mean speed. During this day, it's August of 2026 I would not suggest you to get a DGX Spark or Mac Mini for local AI interference. The reality is that these models are still not good enough and way too slow. But maybe if there's some reasonable technological break breakthroughs in the next year and you get a good model with 60 to 70 tokens per second, I would consider them as a good solution, even if the price has doubled since the original release around half a year ago. But for this day I would say that you are much better just using the cloud AI models.
You should also know that NVIDIA not intended the DGX Spark to be a local interference machine in first place, but more of a researchers system to tests with a lot of memory and then push the training into the clouds.
-- What about two or more GPUs? --
Two 24 GB cards do not automatically become one perfect 48 GB card. Some software can split a model across GPUs, but communication, PCIe bandwidth, cooling, power, and software support all become part of the problem.
Two GPUs can be a good solution when you know your software supports the setup. For a beginner, one larger card is usually easier and quieter. Two RTX 3090s are less a computer and more a space heater. So if you go this way, I cannot tell you much since I haven't tried it out yet. But I think it comes with a lot of problems.
-- Benchmaxxing --
Benchmaxing is also something I need to cover in this post, as I have stumbled across it so many times during my past, and it is a serious disease in the AI ecosystem.
Like for example Alibaba when they dropped Qwen 3.6 27b - They compared their benchmarks with Claude OPUS. A model that is obviously much larger and obviously also gives better results. The Alibaba model was just very efficient at solving these open benchmarks. And at the same time, there was a huge bot army on Reddit trying to discreditate the Claude models.
I spend a lot of time with this model and also with the 35B model. And if somebody had the capacity, they would also prefer the older Qwen 3.5 model with over a hundred parameters just because they all also knew that even if the benchmark told something else, that the older bigger model was just better...
-- Linux, Windows, or macOS? --
The operating system matters more for local AI than many people expect because the operating system determines which drivers, runtimes, containers, libraries, and GPU backends you can actually use.
Linux is generally the safest choice for a dedicated AI workstation. Most serious inference software, CUDA tooling, ROCm tooling, Docker containers, Python packages, and experimental projects are developed with Linux in mind first. If you want to run vLLM, containers, multiple GPUs, servers, experimental kernels, or whatever somebody released on GitHub yesterday, Linux will usually cause the least pain.
The downside is that Linux assumes that you are willing to learn Linux. Driver problems, permissions, networking, Docker, Python environments, and dependency conflicts can still turn your supposedly relaxing Sunday into system-administration training.
Windows is usually the easiest option if the AI machine is also your normal desktop computer. Applications such as LM Studio, Ollama, Stable Diffusion interfaces, and many other consumer tools make local AI fairly painless. NVIDIA support is also generally good.
For more advanced workloads, Windows sometimes becomes awkward because many projects are written primarily for Linux. WSL2 solves a lot of this by essentially giving you a Linux environment inside Windows, and for many people that is a very good compromise. But I had situation when it was not.
MacOS is its own ecosystem. On Apple Silicon, the CPU and GPU share unified memory, which means a Mac with a large amount of memory can load models that would require an extremely expensive GPU setup on a traditional PC.
Projects such as MLX, llama.cpp, Ollama, and software using Apple's Metal framework can work extremely well on Macs. They are also quiet, compact, and ridiculously power efficient compared with a workstation containing several large GPUs.
But again: capacity is not the same thing as speed or compatibility.
A Mac with 128 GB or more of unified memory may technically load an enormous model, but that does not mean it will run it as quickly as a dedicated NVIDIA GPU with fast VRAM. It will be slow! So my simplified operating-system recommendation would be:
Linux: best if the machine exists primarily for AI. Windows: best if it is also your gaming or everyday workstation. Windows + WSL2: a very useful compromise when you need Linux AI tools without abandoning Windows. macOS: excellent when the exact models and applications you want already have good Metal or MLX support and you value large unified memory, silence, and efficiency.
I personally have dualboot with Windows and Linux. Then on linux I also have many systemd services that expose the PC so that I can conviniently connect with my macbook onto it.
And everything brings us back to the same rule as with the GPU itself: Do not choose the operating system first. Choose the models and software you actually want to run, check what they support best, and build the machine around that.
-- My decision tree --
- If you use AI occasionally, use an API.
- If you mainly generate images or learn local tools, start with a reasonably priced NVIDIA GPU with enough VRAM.
- If you want serious local inference on a budget, look at a used RTX 3090.
- If you want fast inference and have the budget, consider a high-end NVIDIA card.
- If you want quiet, efficient, compact hardware, consider a Mac—but check benchmarks for your exact model.
- If you need a large memory pool more than maximum speed, look at unified-memory systems such as DGX Spark.
- If you want to train large models, use cloud compute first unless you already know exactly why you need local training hardware.
Interference Engine
If you now decided to go the local AI path you will certainly come across the question what software to use to run you models. So here a short crash course:
OLLAMA: Very easy to use and to get into. Has an UI and great support. LLAMA.cpp: The fast one. Has better performance than the others. vLLM: What I use. Since I can easily host an openai compatible endpoint and can run multiple requests against it in parallel. Perfect for agent swarms.
-- Final thought --
Do not start with the GPU. Start with the model.
What do you want to run? At what quantization? With what context length? How many tokens per second do you need? Which software will you use?
VRAM determines what fits. Bandwidth determines how quickly it can be read. CUDA and Tensor Cores determine how much parallel AI math the chip can perform. Software support determines whether you spend your weekend building something—or debugging a driver.
And if the model is actually running in the cloud, congratulations: you do not need a local AI workstation. You need an API client. Your old laptop is probably fine.