Local LLM Deployment on 24GB GPUs: Models & Optimizations

GPU VRAM and Model Parameters: The capacity of your GPU's VRAM primarily determines which models you can run. LLMs are often categorized by parameter count (e.g. 7B, 13B, 70B for 7 billion, 13 billion, 70 billion parameters). VRAM usage scales roughly linearly with model size and precision: for example, a 7B model in half-precision (FP16) may require ~14 GB VRAM, whereas a 13B model is ~26 GB (too large to fit 24 GB without compression). A 24GB GPU can handle smaller models at full precision or larger models with quantization (compression) ^[1]. The RTX 40/50 series GPUs also offer high memory bandwidth and tensor core performance, which improve throughput for lower precision inference.

NVIDIA's own product page confirms the GeForce RTX 4090 is built on the Ada Lovelace architecture with 16,384 CUDA cores, 24 GB of GDDR6X memory on a 384-bit bus, and a 450 W board power rating (^[2]). That combination of core count and memory bandwidth is what lets a single 24 GB card sustain useful throughput on 30B-class models once they are quantized down to roughly 4 bits per weight.

GPU Options for 24GB VRAM:

The RTX 4090 remains the most popular choice for serious local LLM deployment. For budget-conscious users, the RTX 3090 offers exceptional value in the used market, providing about 80% of the RTX 4090's performance for less than half the price localllm.in.

GPU availability shifted considerably between this article's original publication and mid-2026. NVIDIA ended RTX 4090 production in October 2024, and with new supply exhausted, street prices for remaining new units climbed to roughly $3,700 to $4,300 by early 2026, with some AIB models exceeding $5,000 (^[3]). Used RTX 4090 units in the $1,400 to $2,000 range remain the more sensible entry point for local inference at this point (^[4]). The RTX 3090's used market has held up better by comparison: tracked listings in July 2026 averaged around $1,254, with a typical range of $995 to $1,504, which keeps its cost per gigabyte of VRAM well below the RTX 4090's (^[5]). Reviewers continue to describe a used RTX 3090 as the value king for local AI workloads even after two subsequent GeForce generations, precisely because 24 GB of VRAM remains the practical threshold for running 30B-class quantized models comfortably (^[6]). For buyers who do not need a display-class gaming card, NVIDIA's RTX 4000 SFF Ada offers 20 GB of ECC GDDR6 memory in a 70 W, half-height workstation form factor that draws power directly from the PCIe slot with no external power connector, trading some VRAM headroom and CUDA core count for a much smaller power and space footprint (^[7]).

Quantization: Quantization reduces memory usage by using lower precision for model weights (and sometimes activations). Common formats include 8-bit (INT8) and 4-bit (INT4) weight compression. For instance, 4-bit quantization cuts memory roughly to one-quarter: a 7B model that needs ~14 GB in FP16 might use only ~4–5 GB in 4-bit form ^[1]. Popular quantization methods are GPTQ (post-training quantization for GPU), bitsandbytes (8-bit loader), and the GGUF 4-bit quant formats used by llama.cpp ^[1]. These enable fitting larger models on 24 GB – with some quality trade-off.

GGUF Format (2025-2026 Standard): The GGUF format has become the de facto standard for local LLM deployment. It offers modular file structure, centralized metadata management, and excellent cross-platform support. llama.cpp now supports 1.5-bit through 8-bit integer quantization, with formats like Q4_K_M providing the best balance of quality and efficiency for most users github.com/ggml-org/llama.cpp. For aggressive quantization (IQ3, IQ2), using an importance matrix (--imatrix) significantly improves quality.

A newer quantization format worth tracking is EXL3, shipped by the ExLlamaV3 project as a streamlined derivative of the QTIP method developed at Cornell RelaxML. It computes Hessians on the fly and uses a fused Viterbi kernel so a model can be converted to a target bitrate in one pass, and a 3-bit-per-weight EXL3 quantization with a 4096-token cache can run in well under 16 GB of VRAM (^[8]). Independent benchmarking of llama.cpp's own quantization ladder on Llama-3.1-8B-Instruct supports the ordering most local users already assume in practice: Q4_K_M and Q5_K_M sit close enough to full precision on the accuracy/size curve that the loss is rarely noticeable in general chat or coding use, while formats below roughly 4 bits per weight show measurably larger degradation without an importance matrix (^[9]). The llama.cpp project's own quantize tool documentation remains the authoritative reference for what each suffix (_S, _M, _L) means and which base precisions (F32, BF16, FP16) can be converted into which GGUF quant (^[10]).

Model Size Guidelines for 24GB:

Example: A 70B model in FP16 needs ~140 GB of memory, but in 4-bit (INT4) it's about 35 GB. This still exceeds 24 GB, but with aggressive quantization or layer offloading to system RAM, single-GPU operation becomes possible with performance tradeoffs.

Inference Speed: Speed is typically measured in tokens generated per second (tok/s). It depends on model size, quantization level, and the efficiency of the software. A larger model means more computation per token, so throughput drops as model size increases. On an RTX 4090-class GPU, a 7B model might generate on the order of ~100–140 tokens/s, whereas a 30B+ model might do ~30–40 tokens/s under similar conditions ^[11]. For example, using the optimized exllama GPU backend, users reported ~140 tok/s for a 7B model and ~40 tok/s for a 33B model on a 24 GB GPU.

RTX 4090 Benchmark Results (Ollama, Q4 quantization):

^[12]

This illustrates the inverse scaling of speed with size. The RTX 4090 excels in hosting lightweight and mid-range LLMs, with evaluation speeds consistently utilizing 92-96% of GPU capacity. Performance drops significantly for 40B+ models, where the 24GB VRAM becomes a limiting factor.

The local LLM landscape has evolved significantly. Here are the top models optimized for 24GB VRAM:

Llama 3.1/3.2 (Meta)

Meta's Llama 3.1 and 3.2 remain among the most popular open-source LLMs. Available in 8B, 70B, and 405B sizes, the 8B variant is ideal for 24GB GPUs running at full precision, while the 70B can run with 4-bit quantization and layer offloading. Features include improved reasoning, multilingual support, and extended context windows up to 128K tokens ^[13]. Meta's own model card for the 8B instruct variant confirms pretraining on roughly 15 trillion tokens with a December 2023 data cutoff, and 39.3 million cumulative GPU hours of training compute on H100-class hardware (^[14]).

Mistral Small 3 (24B)

Released in early 2025, Mistral Small 3 represents the sweet spot for 24GB GPUs. It achieves state-of-the-art performance on benchmarks, handles long contexts well, and fits comfortably with Q4 quantization (~14-15GB). Excellent for general-purpose tasks and coding ^[15].

Qwen 2.5 (Alibaba)

Qwen 2.5 models, particularly the Coder variants, have achieved impressive benchmarks on code generation, reasoning, and debugging. The 14B and 32B versions are well-suited for 24GB GPUs, with the 32B requiring Q4 quantization ^[16].

DeepSeek R1 Distilled Models

The full DeepSeek R1 (671B) requires datacenter hardware, but the distilled variants are excellent for local deployment:

^[17]

Gemma 2 (Google)

Google's Gemma 2 models (9B and 27B) offer competitive performance with efficient architectures designed for consumer hardware. The 27B model fits comfortably on 24GB with Q4 quantization.

Qwen3 (Alibaba)

Qwen3 succeeded the Qwen2.5 line and is Alibaba's current generation of dense and mixture-of-experts models. The Qwen3-32B dense checkpoint carries 32.8B parameters, a native context window of 32,768 tokens (extendable to 131,072 with YaRN), and ships under the Apache 2.0 license, so it can be redistributed and fine-tuned without the restrictions some earlier open-weight releases carried (^[18]). Ollama's library lists a Q4_K_M build of Qwen3-32B at roughly 20GB on disk, which leaves enough headroom on a 24GB card for a moderate context window once KV cache overhead is accounted for (^[19]). Reviewers surveying the local-LLM landscape in 2026 generally treat Qwen3-32B (or the mixture-of-experts Qwen3-30B-A3B variant, which activates only about 3B parameters per token) as the strongest single all-around default for a 24GB card, ahead of the older Qwen 2.5 generation this article originally covered.

Gemma 3 (Google)

Google's Gemma 3 line succeeded Gemma 2 and added native multimodal (vision) input at the 4B, 12B, and 27B sizes. Google published official Quantization-Aware Trained (QAT) checkpoints rather than relying solely on third-party post-training quantization, and the company's developer blog reports that the int4 QAT build of Gemma 3 27B needs approximately 14.1GB of VRAM, comfortably inside a 24GB budget with room left for a large context window (^[20]). The official Q4_0 unquantized-weights release on Hugging Face documents the same QAT approach and provides the checkpoint used by most downstream GGUF conversions (^[21]). Because QAT bakes the quantization error into training rather than applying it afterward, Gemma 3 27B tends to hold up better at 4-bit than post-training-quantized models of similar size.

Mistral Small 3.2 (Mistral AI)

Mistral AI iterated on its 24B "Small" line twice since the 3.0 release covered above: Mistral Small 3.1 added long-context and vision support, and Mistral Small 3.2 (released in June 2025) focused on reliability, notably cutting infinite-generation loops on long or repetitive prompts roughly in half relative to 3.1 (^[22]). Both versions keep the same 24B parameter count and 128K-token context window as the original Small 3 release, so the VRAM math for a 24GB GPU is unchanged. Independent commentary on the release notes that Mistral recommends a low sampling temperature (around 0.15) and pairing the model with vLLM for the best function-calling and tool-use behavior (^[23]).

gpt-oss-20b (OpenAI)

OpenAI's first open-weight release since GPT-2, gpt-oss, shipped in two sizes under an Apache 2.0 license. The smaller gpt-oss-20b is a 21B-parameter mixture-of-experts model with only about 3.6B active parameters per token, and OpenAI's own Hugging Face model card documents native support down to roughly 16GB of memory when quantized, well inside a 24GB budget (^[24]). Community VRAM testing puts a Q4_K_M build of gpt-oss-20b at around 12GB, plus a few gigabytes of KV cache at typical context lengths, meaning a 24GB card can run it alongside a browser, IDE, or a second small model without contention (^[25]). The larger 120B sibling (about 117B total, ~5B active) targets 80GB-class datacenter cards and does not fit a 24GB consumer GPU even quantized.

DeepSeek-V3.2 (DeepSeek)

DeepSeek has continued iterating past the R1 distillations covered above. DeepSeek-V3.2, released December 1, 2025 under the MIT license, introduces DeepSeek Sparse Attention (DSA), which reduces attention computation to near-linear complexity for long-context workloads and supports a 160,000-token context window at the full model size (^[26]). The full V3.2 checkpoint is a large mixture-of-experts model that still targets multi-GPU datacenter deployment rather than a single 24GB card, but DeepSeek's own release notes describe smaller distilled and quantized derivatives intended for single-GPU use in the same way the R1 distillations were (^[27]). For a 24GB GPU today, the practical DeepSeek options remain the R1/V3.2 distillations at the 32B, 14B, and 7B sizes rather than the full flagship checkpoint.

llama.cpp

Written in pure C/C++ with no external dependencies, llama.cpp remains a standout for efficiency and portability. Key 2025-2026 updates include:

• NVFP4 and FP8 quantization support for RTX 40/50 series
• Up to 35% faster token generation with recent optimizations
• Vulkan support for AMD and Intel GPUs
• Feature-rich CLI and web UI under 90MB total size

llama.cpp is ideal for extreme portability, minimal dependencies, and running on consumer-grade hardware github.com/ggml-org/llama.cpp.

vLLM

vLLM is engineered for high-performance, production-grade LLM inference. Key features include:

• PagedAttention technology reducing memory fragmentation by 50%+ and increasing throughput 2-4x for concurrent requests
• vLLM v0.11.0 explicitly supports NVIDIA Blackwell architecture (RTX 5090) with native NVFP4/CUTLASS
• vLLM-Omni (November 2025) enables omni-modality model serving including diffusion transformers

For multi-user applications, vLLM delivered 35x+ the request throughput compared to llama.cpp at peak load ^[28].

Ollama

Ollama offers effortless installation, ready-to-use models, and a plug-and-play API. Perfect for rapid prototyping without complex setup. Supports mixed CPU/GPU inference for partial layer offloading ^[29].

LM Studio

LM Studio's GUI makes it easy for beginners and those who prefer graphical interfaces. Notable for:

• Good performance on lower-spec hardware with Vulkan offloading
• Easy function schema definition for agent workflows
• Interactive model testing and comparison

^[30]

Recommended Workflow: Use Ollama for rapid prototyping, then migrate to vLLM for production deployment where performance becomes critical medium.com/@rosgluk.

ExLlamaV3 (EXL3)

ExLlamaV3 is a purpose-built inference and quantization library for consumer GPUs, and its EXL3 format (described above) is now a third option alongside GGUF and GPTQ/AWQ for users who want the smallest possible footprint at a given quality target. The project ships as a single-step converter: give it a Hugging Face checkpoint and a target bits-per-weight, and it produces a quantized model without a separate calibration pass for most model sizes (^[8]). It remains more niche than llama.cpp or vLLM, but it is a reasonable option when a model needs to be squeezed to 3 or 3.5 bits per weight to fit a 24GB card alongside a large context window.

Version Currency Check (Mid-2026)

All three of the major frameworks referenced in this article have shipped substantial updates since it was first published. vLLM's GitHub release history shows v0.24.0 landing on June 28, 2026, with 408 commits from 200 contributors, continued maturation of DeepSeek-V3.2 support across backends, and Model Runner V2 now selected by default for Llama, Mistral, and Qwen3 dense models (^[31]). LM Studio's public changelog documents a steady cadence of point releases through 2026 covering MLX engine performance work for vision-capable models and Flash Attention now enabled by default on its Vulkan and Metal backends (^[32]). A May 2026 roundup of local AI runtime updates covering Ollama, vLLM, llama.cpp, MLX, and LM Studio together is a useful single reference for tracking this pace of change without following five separate changelogs (^[33]). The practical takeaway for a 24GB deployment is the same one this article made originally: pin a specific framework version for any production workload, and re-test after upgrading, since quantization kernels and default settings do change between releases.

For those considering an upgrade, the NVIDIA RTX 5090 (released January 2025) offers significant improvements:

The RTX 5090 delivers up to 67% improvement over the RTX 4090 for LLM inference. Its 32GB VRAM allows running 70B models more comfortably with Q4 quantization. At $9.38 per token/second, it offers competitive price-performance despite the higher cost ^{[34] [35]}.

1. Choose the right quantization: Q4_K_M provides the best balance for most users. Use Q5_K_M for critical applications.

2. Use importance matrices: For aggressive quantization (IQ3, IQ2), always use --imatrix for better quality.

3. Match model to use case: Coding tasks benefit from specialized models like Qwen Coder or DeepSeek Coder. General chat works well with Llama or Mistral.

4. Monitor VRAM usage: Always reserve 20-30% additional VRAM for context windows and overhead.

5. Consider hybrid inference: Ollama and llama.cpp support offloading some layers to system RAM, enabling larger models at reduced speed.

Running LLMs locally on 24GB GPUs has never been more accessible. The RTX 4090 and RTX 3090 provide excellent performance for models up to 32B parameters with proper quantization. For those seeking maximum capability, the RTX 5090's 32GB VRAM opens the door to comfortable 70B inference. With frameworks like llama.cpp, vLLM, and Ollama continuously improving—now achieving up to 35% faster token generation through NVIDIA's 2026 optimizations—local AI deployment remains a compelling alternative to cloud-based solutions for privacy, cost, and performance-conscious users.