Self-Hosted LLM Guide: GPU Selection, VRAM and Deployment

Self-hosted LLMs run language models on infrastructure you control instead of sending every request to a third-party model API. A self hosted LLM environment can provide greater control over deployment, data handling, model versions, and available GPU resources.

This guide explains how to choose an LLM server by comparing LLM GPU requirements, VRAM sizing, deployment frameworks, and benchmark results. Always validate the model, quantization, context length, expected concurrency, and software stack before selecting a server.

Choose a Self-Hosted LLM Deployment Path

Start with your self hosted LLM workload, then compare model size, VRAM requirements, concurrency, and framework compatibility before choosing an LLM server.

Quick GPU Recommendations by Workload

These are starting points, not guaranteed capacity limits. Check current configurations and pricing before ordering.

Why Teams Switch Away from Cloud AI APIs

Three pain points that surface once your self-hosted LLM text generation workloads move from prototype to production at scale.

Why Self-Host? The Four Deployment Paths Compared

Understanding structural limits matters more than comparing spec sheets — especially for production LLM inference at scale.

GPU Mart Dedicated GPU Hosting: Key Advantages

What Hardware Determines LLM Performance?

Understanding LLM GPU requirements, including VRAM, memory bandwidth, and precision support, is essential before selecting hardware for an LLM inference server.

1 — VRAM: LLM GPU Requirements by Model Size

VRAM is usually the first constraint when sizing a self-hosted LLM. The GPU must hold the model weights, KV cache, runtime overhead, and sufficient headroom for the expected context length and concurrency. The examples below use a 14B-parameter model as a reference; larger models require proportionally more memory.

Note on Q4_K_M (Ollama default): to avoid the significant quality degradation of pure 4-bit quantization, Ollama uses a "mixed precision" approach — core weights at 6-bit, less critical weights at 4-bit. This keeps quality loss minimal while keeping a 35B model at approximately 22–23 GB VRAM.

KV Cache VRAM Requirements: Context Length & Concurrency

Beyond model weights, inference requires KV Cache memory for every active request. For Qwen2.5-14B at FP16 (24 layers, 8 KV heads, 64 dims per head):

2 — Memory Bandwidth: How Fast Tokens Generate

Generating each token requires loading the full model weight set from VRAM into Tensor Cores. Memory bandwidth — a key LLM GPU requirement — is the hard ceiling on tok/s, not TFLOPS. A100-40G at 1,555 GB/s on a 20B FP16 model: theoretical max ≈ 38 tok/s for a single request.

3 — Tensor Core Precision: Where Blackwell Pulls Ahead

On GPUs with native FP4/FP8 support: FP4 compute is 2× FP8, and FP8 is 2× FP16. FP8-quantized model weights also use half the VRAM of FP16, and FP4 uses a quarter — freeing more VRAM for context and concurrency. Only Blackwell GPUs can run LLMs at FP4 precision, achieving maximum possible throughput.

Best GPUs for LLM Workloads

Finding the best GPU for LLM workloads depends on model size, concurrency, and budget. GPU specifications should be checked against the exact installed model and form factor. Current pricing: gpu-mart.com/pricing.

Tensor Cores Dense: FP16 dense compute (TFLOPS).  AI TOPS: max compute at lowest supported precision (sparse).  Precision: natively supported precisions.

LLM Inference Server Benchmarks

vLLM framework · Input 1,024 tokens + Output 512 tokens · GPU Mart test hardware

Ollama Single-User Benchmarks

llama.cpp backend · Q4_K_M quantization · Suitable for self-hosted LLM development on a GPU VPS · No KV Cache pre-allocation · Input 1,024 tokens + Output 512 tokens · Single user · Average of 10 requests

VRAM Usage & Maximum Supported Models

Context reduction tip: set num_ctx to reduce VRAM and run 35B models on 32 GB cards:

Generation Speed by GPU

Inference Framework Selection

Framework choice affects throughput and latency as much as GPU selection — pick the right one for your LLM inference use case.

Deployment Optimization Tips

Auxiliary Models for <16 GB GPUs

Real Customer Deployments

Representative deployment examples. Identifying details are omitted; performance varies by model, software, and workload.

Additional Industry Deployment Records

Who This Is (and Isn't) For

Choosing the right GPU for self-hosted LLM workloads on dedicated infrastructure is not right for every team. Here is the honest breakdown.

Frequently Asked Questions

What type of LLM server should I choose?

Choose a GPU VPS for smaller models, development, and moderate inference workloads. Choose a dedicated GPU server when you need stable resources, more VRAM, continuous operation, or higher concurrency. Consider a multi-GPU server when the model and KV cache exceed the capacity of one GPU.

What is the difference between a GPU VPS and a dedicated GPU server?

A GPU VPS can provide dedicated GPU access within a virtualized server, while a dedicated GPU server provides the entire physical machine. CPU, RAM, storage, network, operating system, and deployment time vary by plan, so check the selected configuration before ordering.

How do I calculate VRAM requirements for a self-hosted LLM?

Start with model weights, then add KV cache and operating headroom. A practical estimate is model weights plus KV cache plus 20–30% headroom. Context length, concurrency, framework, precision, and quantization can materially change the result.

Can I self-host an LLM on a GPU VPS?

Yes, when the model and runtime fit the available GPU memory and server resources. Smaller quantized models may fit on 16–24 GB GPUs, while larger models, long contexts, and higher concurrency may require 48–96 GB or multi-GPU configurations.

Should I use Ollama or vLLM?

Ollama is convenient for development, evaluation, and simpler single-user deployments. vLLM is generally better suited to production APIs that need continuous batching and higher concurrency. Validate compatibility with the exact model and version you plan to run.

Should I use one large-VRAM GPU or multiple GPUs?

Prefer one GPU when the model and required KV cache fit comfortably in memory. Multi-GPU tensor parallelism is useful when the workload exceeds single-card capacity, but it adds communication overhead and deployment complexity.

How do I deploy Hugging Face or private models?

Install the required NVIDIA driver, CUDA runtime, framework, and model dependencies, then download from Hugging Face or upload your private model through your approved transfer process. Deployment steps depend on the operating system, model license, and serving framework.

Is self-hosting cheaper than a model API?

It can be for sustained, high-utilization workloads, but not in every case. Compare monthly infrastructure cost, expected tokens, utilization, engineering time, storage, backups, monitoring, and support against current API pricing.

Where can I find current GPU server prices?

Prices, promotions, configurations, and availability change. Review the current GPU hosting pricing page before ordering.