Best GPU for Local LLMs in 2026: VRAM by Model Size
Updated
Buy for VRAM first. At 4-bit precision, a language model needs about 0.6GB of graphics memory per billion parameters, plus a few GB for the conversation. That puts 8B models on 8GB cards, models up to about 30B on 24GB, and 70B models beyond any single consumer card. Memory bandwidth then decides how fast it answers.
How Much VRAM Does a Local LLM Need?
A 4-bit version of a model is under a third the size of the original 16-bit one. In llama.cpp, an open-source program for running models locally, the Q4_K_M format averages 4.89 bits per weight. That is about 0.61 bytes per parameter, or 0.6GB per billion parameters. The project's own measurements bear it out: Llama 3.1 8B comes to 4.9GB and Llama 3.1 70B to 43.1GB.
The model weights are not the only thing in memory. While it generates, a model keeps a key-value cache of the conversation so far, and that cache grows with the length of the context. NVIDIA's own worked example, an older 7B model holding 4,096 tokens in 16-bit, needs about 2GB for it. Newer models that share attention heads store less, but long chats and long documents still add up.
So the working rule is: 0.6GB per billion parameters, then leave 2 to 4GB free for the context and the runtime. It is an estimate, not a guarantee. A long context or a higher-precision format needs more.
What Fits on Each Amount of VRAM?
| VRAM | Comfortable at 4-bit | Example cards |
|---|---|---|
| 8GB | 7B to 8B, short chats | RTX 5060, RX 9060 XT 8GB |
| 12GB | Up to about 14B | RTX 5070, Arc B580 |
| 16GB | Up to about 20B | RTX 5060 Ti 16GB, RX 9070 XT, RTX 5080 |
| 24GB | Up to about 30B | Arc Pro B60, RTX PRO 4000 Blackwell, used RTX 3090 or RTX 4090 |
| 32GB | Up to about 45B | RTX 5090, Radeon AI PRO R9700, Arc Pro B70 |
| 96GB | 70B with a long context, up to about 150B | RTX PRO 6000 Blackwell |
Sizes come from the rule above, rounded down. The 8B and 70B anchors are llama.cpp's measured figures. If a model does not fit, llama.cpp can split it between the GPU and system memory, which it describes as partially accelerating models larger than your VRAM. It works, but it is slower than running the whole model on the card.
Why Memory Bandwidth Is the Second Spec
Once a model fits, speed comes mostly from how fast the card can read its memory, not from raw compute. NVIDIA's engineers describe the token-by-token generation step as memory-bound: the time to move the weights and the cache out of memory dominates, not the math. Two cards with the same VRAM can hold the same model and still answer at very different speeds.
That is why the table below lists bandwidth next to memory size. Among 16GB cards, for example, the RTX 5080 has three times the bandwidth of the RX 9060 XT 16GB.
Consumer and Workstation Cards Compared
Gaming cards
| GPU | Memory | Bandwidth |
|---|---|---|
| RTX 5060 Ti 16GB | 16GB GDDR7 | 448 GB/s |
| RX 9060 XT 16GB | 16GB GDDR6 | 320 GB/s |
| RX 9070 XT | 16GB GDDR6 | 640 GB/s |
| RTX 5070 Ti | 16GB GDDR7 | 896 GB/s |
| RTX 5080 | 16GB GDDR7 | 960 GB/s |
| RTX 5090 | 32GB GDDR7 | 1,792 GB/s |
Workstation cards
| GPU | Memory | Bandwidth |
|---|---|---|
| Intel Arc Pro B60 | 24GB GDDR6 | 456 GB/s |
| NVIDIA RTX PRO 4000 Blackwell | 24GB GDDR7, ECC | 672 GB/s |
| Intel Arc Pro B70 | 32GB GDDR6, ECC | 608 GB/s |
| AMD Radeon AI PRO R9700 | 32GB GDDR6, ECC on Linux | 640 GB/s |
| NVIDIA RTX PRO 6000 Blackwell | 96GB GDDR7, ECC | 1,792 GB/s |
Four of the five support error-correcting (ECC) memory; AMD lists it for Linux only on the R9700. More memory per card is the reason to look at them for local AI.
Which GPU Should You Buy for Local LLMs?
Best place to start: a 16GB gaming card
The RTX 5060 Ti 16GB and RX 9060 XT 16GB are the lowest-priced 16GB cards in NVIDIA's and AMD's current lineups, launched at $429 and $349. Both run models up to about 20B at 4-bit. The RTX has more bandwidth (448 against 320 GB/s), so it answers faster. Skip the 8GB versions of either if local AI is part of the plan.
Faster 16GB: RX 9070 XT, RTX 5070 Ti or RTX 5080
These hold the same models as any 16GB card but read memory two to three times faster than the RX 9060 XT 16GB, so replies come quicker. The RX 9070 XT (640 GB/s) is the cheapest of the three at launch price; the RTX 5070 Ti (896 GB/s) and RTX 5080 (960 GB/s) are faster still. They make sense if the same card also has to game.
Best gaming card for local AI: RTX 5090
The RTX 5090 is the only card in NVIDIA's U.S. GeForce RTX 50 lineup with 32GB, and its 1,792 GB/s matches NVIDIA's 96GB workstation flagship. It holds 30B-class models with plenty of context, though not a 70B model at 4-bit, which needs about 43GB. The problem is price: its launch MSRP is $1,999, but Tom's Hardware reported in September that it was regularly listed above $5,000.
32GB for less: Radeon AI PRO R9700 or Arc Pro B70
AMD's Radeon AI PRO R9700 (32GB, 640 GB/s, 300W) launched in October 2025 at $1,299. Intel's Arc Pro B70 (32GB with ECC, 608 GB/s, 230W) launched on March 25, 2026 at a suggested $949. Both give you the RTX 5090's memory size for far less than its street price, at roughly a third of its bandwidth.
Both have also climbed. Notebookcheck reported on October 27 that each now sells above its launch price, with the cheapest store-sold B70 about 58% over $949. AMD pitches the R9700 for multi-GPU setups, and one reviewer pooled four B70s for 128GB.
24GB: Arc Pro B60, RTX PRO 4000 Blackwell or a used RTX 3090 or 4090
Intel's Arc Pro B60 has 24GB, 456 GB/s and a 200W board power. NVIDIA's RTX PRO 4000 Blackwell (24GB with ECC, 672 GB/s) is faster, fits in a single slot and draws 145W, which suits small or multi-card builds.
The used route is a 24GB RTX 3090 or RTX 4090. Read our refurbished GPU guide first, and see our refurbished page for the cards we track. Our cheapest 24GB GPUs page compares the new gaming cards with 24GB or more.
70B and up: RTX PRO 6000 Blackwell or several cards
A 70B model needs more memory than any single gaming card has. The RTX PRO 6000 Blackwell Workstation Edition (96GB with ECC, 1,792 GB/s, 600W) holds one with room to spare. The other route is two or more cards: llama.cpp can split one model across several GPUs, so two 32GB cards give you 64GB to work with.
Before You Buy
Check software support. llama.cpp runs on NVIDIA through CUDA, AMD through HIP, Intel through SYCL, and most GPUs through Vulkan. Other apps may support fewer cards, so check the one you plan to use.
Check power and space. For the RTX 5090, NVIDIA lists 575W of total graphics power and 1,000W of required system power. The RTX PRO 6000 draws up to 600W, and AMD recommends at least a 750W power supply for the R9700. Our power supply guide explains the sizing.
Compare price per GB of VRAM. For local AI, divide the price by the memory size first and look at bandwidth second. The Local AI view of our best value GPUs page ranks cards with 16GB or more by price per GB, using recorded prices with their check dates. Our cheapest 16GB GPUs page is another place to start, and our VRAM guide covers gaming and video editing too.
Frequently Asked Questions
How much VRAM do I need to run a 70B model?
About 43GB for the weights alone at 4-bit: llama.cpp lists Llama 3.1 70B at 43.1GB in its Q4_K_M format. Add room for the context, and you need more than any single consumer card offers. That means 48GB or more split across two or more GPUs, or a 96GB card such as the RTX PRO 6000 Blackwell.
Can I run a local LLM on an 8GB GPU?
Yes, for small models. llama.cpp lists Llama 3.1 8B at 4.9GB in Q4_K_M, which leaves about 3GB on an 8GB card for the context and the runtime. Larger models need a card with more memory, or a split between the GPU and system memory, which runs slower.
Is the RTX 5090 worth it for local AI?
It is the only card in NVIDIA's U.S. GeForce RTX 50 lineup with 32GB and, at 1,792 GB/s, it matches the RTX PRO 6000 Blackwell for memory bandwidth. Its price is the catch: Tom's Hardware reported in September 2026 that the $1,999 card was regularly listed above $5,000. If memory size matters more than speed, the 32GB Radeon AI PRO R9700 and Arc Pro B70 launched at $1,299 and $949.
Do AMD and Intel GPUs work for local LLMs?
Yes. llama.cpp supports NVIDIA through CUDA, AMD through HIP and Intel through SYCL, and has a Vulkan backend that runs on most GPUs. Intel also lists OpenVINO and Intel Extension for PyTorch support for the Arc Pro B60. Check the support list of the specific app you plan to use before you buy.
Compare GPUs on GPUDrip
Specs for every tracked card, with recorded retailer offers and check dates. Confirm current price and stock at the retailer.
Browse GPUsSources
Read on October 30, 2026. Cited for research; none of these sources sponsors or endorses GPU Drip.
- llama.cpp, quantize tool README: Q4_K_M at 4.89 bits per weight; Llama 3.1 8B, 70B and 405B sizes.
- llama.cpp, project README: CUDA, HIP, SYCL and Vulkan backends; CPU and GPU hybrid inference.
- llama.cpp, server README: Splitting a model across multiple GPUs.
- NVIDIA Technical Blog, Mastering LLM Techniques: Inference Optimization: Token generation is memory-bandwidth bound; KV cache size and its example.
- NVIDIA, GeForce graphics card specs compared: RTX 50, 40 and 30 Series memory and bandwidth; RTX 5090 power figures.
- NVIDIA, GeForce RTX 50 Series announcement: RTX 5090, 5080 and 5070 Ti launch prices.
- Tom's Hardware, RTX 5060 Ti and RTX 5060 announcement: April 15, 2025. RTX 5060 Ti 16GB launch price.
- Tom's Hardware, RX 9070 series pricing: February 28, 2025. RX 9070 XT launch price.
- Tom's Hardware, RX 9060 XT pricing: May 21, 2025. RX 9060 XT 16GB launch price.
- AMD, Radeon RX 9070 XT specifications: 16GB GDDR6, up to 640 GB/s.
- AMD, Radeon RX 9060 XT specifications: 16GB GDDR6, up to 320 GB/s.
- AMD, Radeon AI PRO R9700 specifications: 32GB GDDR6, 640 GB/s, ECC on Linux only, 300W board power, 750W minimum power supply.
- HotHardware, Radeon AI PRO R9700 price and release date: October 24, 2025. $1,299, on sale October 27, 2025.
- Intel, Arc Pro B60 specifications: 24GB GDDR6, 456 GB/s, 200W, OpenVINO and IPEX support.
- Intel, Arc Pro B70 specifications: 32GB GDDR6 with ECC, 608 GB/s, 230W.
- Notebookcheck, Arc Pro B70 and Radeon AI Pro R9700 prices: October 27, 2026. B70 launch on March 25 at $949; both cards now above launch price; a four-card B70 setup with 128GB.
- NVIDIA, RTX PRO 4000 Blackwell: 24GB GDDR7 with ECC, 672 GB/s, 145W, single slot.
- NVIDIA, RTX PRO 6000 Blackwell Workstation Edition: 96GB GDDR7 with ECC, 1,792 GB/s, 600W.
- Tom's Hardware, Founders Edition cards at MSRP at PAX West: September 5, 2026. RTX 5090 listings above $5,000.