[Revised January 24, 2026]

Running Large Language Models (LLMs) Locally on a 24GB GPU (RTX 4090/3090)

Running large language models (LLMs) on local hardware has become increasingly feasible, with over 42% of developers now running LLMs entirely on local machines to ensure privacy, reduce cloud costs, and boost performance. This report explores the top LLMs that can be deployed on a single high-end GPU with 24 GB VRAM (NVIDIA RTX 4090, RTX 3090, or AMD RX 7900 XTX) – focusing on open-source models, with notes on a few closed models – and covers their architectures, VRAM requirements, speeds, context lengths, and use cases. We also discuss popular local inference frameworks (like llama.cpp, vLLM, LM Studio, Ollama) and optimization techniques (quantization, RoPE scaling, GGUF format) to maximize performance.

Note: The NVIDIA RTX 5090, released in January 2025, features 32GB VRAM (not 24GB), making it capable of running even larger models. This guide focuses on the 24GB VRAM class (RTX 4090/3090/7900 XTX) which remains the most common high-end consumer configuration.

Hardware Considerations: 24 GB VRAM and Model Size

GPU VRAM and Model Parameters: The capacity of your GPU's VRAM primarily determines which models you can run. LLMs are often categorized by parameter count (e.g. 7B, 13B, 70B for 7 billion, 13 billion, 70 billion parameters). VRAM usage scales roughly linearly with model size and precision: for example, a 7B model in half-precision (FP16) may require ~14 GB VRAM, whereas a 13B model is ~26 GB (too large to fit 24 GB without compression). A 24GB GPU can handle smaller models at full precision or larger models with quantization (compression) ^[1]. The RTX 40/50 series GPUs also offer high memory bandwidth and tensor core performance, which improve throughput for lower precision inference.

GPU Options for 24GB VRAM:

The RTX 4090 remains the most popular choice for serious local LLM deployment. For budget-conscious users, the RTX 3090 offers exceptional value in the used market, providing about 80% of the RTX 4090's performance for less than half the price localllm.in.

Quantization: Quantization reduces memory usage by using lower precision for model weights (and sometimes activations). Common formats include 8-bit (INT8) and 4-bit (INT4) weight compression. For instance, 4-bit quantization cuts memory roughly to one-quarter: a 7B model that needs ~14 GB in FP16 might use only ~4–5 GB in 4-bit form ^[1]. Popular quantization methods are GPTQ (post-training quantization for GPU), bitsandbytes (8-bit loader), and the GGUF 4-bit quant formats used by llama.cpp ^[1]. These enable fitting larger models on 24 GB – with some quality trade-off.

GGUF Format (2025-2026 Standard): The GGUF format has become the de facto standard for local LLM deployment. It offers modular file structure, centralized metadata management, and excellent cross-platform support. llama.cpp now supports 1.5-bit through 8-bit integer quantization, with formats like Q4_K_M providing the best balance of quality and efficiency for most users github.com/ggml-org/llama.cpp. For aggressive quantization (IQ3, IQ2), using an importance matrix (--imatrix) significantly improves quality.

Model Size Guidelines for 24GB:

Example: A 70B model in FP16 needs ~140 GB of memory, but in 4-bit (INT4) it's about 35 GB. This still exceeds 24 GB, but with aggressive quantization or layer offloading to system RAM, single-GPU operation becomes possible with performance tradeoffs.

Inference Speed: Speed is typically measured in tokens generated per second (tok/s). It depends on model size, quantization level, and the efficiency of the software. A larger model means more computation per token, so throughput drops as model size increases. On an RTX 4090-class GPU, a 7B model might generate on the order of ~100–140 tokens/s, whereas a 30B+ model might do ~30–40 tokens/s under similar conditions ^[2]. For example, using the optimized exllama GPU backend, users reported ~140 tok/s for a 7B model and ~40 tok/s for a 33B model on a 24 GB GPU.

RTX 4090 Benchmark Results (Ollama, Q4 quantization):

^[3]

This illustrates the inverse scaling of speed with size. The RTX 4090 excels in hosting lightweight and mid-range LLMs, with evaluation speeds consistently utilizing 92-96% of GPU capacity. Performance drops significantly for 40B+ models, where the 24GB VRAM becomes a limiting factor.

Best LLM Models for 24GB GPUs (2025-2026)

The local LLM landscape has evolved significantly. Here are the top models optimized for 24GB VRAM:

Llama 3.1/3.2 (Meta)

Meta's Llama 3.1 and 3.2 remain among the most popular open-source LLMs. Available in 8B, 70B, and 405B sizes, the 8B variant is ideal for 24GB GPUs running at full precision, while the 70B can run with 4-bit quantization and layer offloading. Features include improved reasoning, multilingual support, and extended context windows up to 128K tokens ^[4].

Mistral Small 3 (24B)

Released in early 2025, Mistral Small 3 represents the sweet spot for 24GB GPUs. It achieves state-of-the-art performance on benchmarks, handles long contexts well, and fits comfortably with Q4 quantization (~14-15GB). Excellent for general-purpose tasks and coding ^[5].

Qwen 2.5 (Alibaba)

Qwen 2.5 models, particularly the Coder variants, have achieved impressive benchmarks on code generation, reasoning, and debugging. The 14B and 32B versions are well-suited for 24GB GPUs, with the 32B requiring Q4 quantization ^[6].

DeepSeek R1 Distilled Models

The full DeepSeek R1 (671B) requires datacenter hardware, but the distilled variants are excellent for local deployment:

^[7]

Gemma 2 (Google)

Google's Gemma 2 models (9B and 27B) offer competitive performance with efficient architectures designed for consumer hardware. The 27B model fits comfortably on 24GB with Q4 quantization.

Local Inference Frameworks (2025-2026)

llama.cpp

Written in pure C/C++ with no external dependencies, llama.cpp remains a standout for efficiency and portability. Key 2025-2026 updates include:

• NVFP4 and FP8 quantization support for RTX 40/50 series
• Up to 35% faster token generation with recent optimizations
• Vulkan support for AMD and Intel GPUs
• Feature-rich CLI and web UI under 90MB total size

llama.cpp is ideal for extreme portability, minimal dependencies, and running on consumer-grade hardware github.com/ggml-org/llama.cpp.

vLLM

vLLM is engineered for high-performance, production-grade LLM inference. Key features include:

• PagedAttention technology reducing memory fragmentation by 50%+ and increasing throughput 2-4x for concurrent requests
• vLLM v0.11.0 explicitly supports NVIDIA Blackwell architecture (RTX 5090) with native NVFP4/CUTLASS
• vLLM-Omni (November 2025) enables omni-modality model serving including diffusion transformers

For multi-user applications, vLLM delivered 35x+ the request throughput compared to llama.cpp at peak load ^[8].

Ollama

Ollama offers effortless installation, ready-to-use models, and a plug-and-play API. Perfect for rapid prototyping without complex setup. Supports mixed CPU/GPU inference for partial layer offloading ^[9].

LM Studio

LM Studio's GUI makes it easy for beginners and those who prefer graphical interfaces. Notable for:

• Good performance on lower-spec hardware with Vulkan offloading
• Easy function schema definition for agent workflows
• Interactive model testing and comparison

^[10]

Recommended Workflow: Use Ollama for rapid prototyping, then migrate to vLLM for production deployment where performance becomes critical medium.com/@rosgluk.

RTX 5090: The New Consumer Champion (32GB)

For those considering an upgrade, the NVIDIA RTX 5090 (released January 2025) offers significant improvements:

The RTX 5090 delivers up to 67% improvement over the RTX 4090 for LLM inference. Its 32GB VRAM allows running 70B models more comfortably with Q4 quantization. At $9.38 per token/second, it offers competitive price-performance despite the higher cost ^{[11] [12]}.

Optimization Best Practices

1. Choose the right quantization: Q4_K_M provides the best balance for most users. Use Q5_K_M for critical applications.

2. Use importance matrices: For aggressive quantization (IQ3, IQ2), always use --imatrix for better quality.

3. Match model to use case: Coding tasks benefit from specialized models like Qwen Coder or DeepSeek Coder. General chat works well with Llama or Mistral.

4. Monitor VRAM usage: Always reserve 20-30% additional VRAM for context windows and overhead.

5. Consider hybrid inference: Ollama and llama.cpp support offloading some layers to system RAM, enabling larger models at reduced speed.

Conclusion

Running LLMs locally on 24GB GPUs has never been more accessible. The RTX 4090 and RTX 3090 provide excellent performance for models up to 32B parameters with proper quantization. For those seeking maximum capability, the RTX 5090's 32GB VRAM opens the door to comfortable 70B inference. With frameworks like llama.cpp, vLLM, and Ollama continuously improving—now achieving up to 35% faster token generation through NVIDIA's 2026 optimizations—local AI deployment remains a compelling alternative to cloud-based solutions for privacy, cost, and performance-conscious users.