Self-Hosted LLMs: GPU Selection, Benchmarks, VRAM Requirements & Hosting Guide

Self-hosted LLMs allow organizations to run AI models on dedicated GPU infrastructure instead of relying on third-party APIs. Compared with token-based AI services, self-hosting offers greater control over text generation performance, data privacy, compliance, and long-term operating costs.

This guide compares GPU requirements, VRAM sizing, benchmark results across 14 GPU configurations, deployment frameworks, and LLM hosting architectures for running 7B–70B language models in production. Whether you are evaluating GPU VPS or dedicated GPU server options, the data here covers LLM GPU requirements, real inference benchmarks, and hosting cost comparisons to support your decision.

Why Teams Switch Away from Cloud AI APIs

Three pain points that surface once your self-hosted LLM text generation workloads move from prototype to production at scale.

Why Self-Host? The Four Deployment Paths Compared

Understanding structural limits matters more than comparing spec sheets — especially for production LLM inference at scale.

GPU Mart Dedicated GPU: Four Core Advantages

What Hardware Determines LLM Performance?

Understanding LLM GPU requirements — VRAM, memory bandwidth, and precision support — is essential before selecting the best GPU for LLM hosting. These three variables determine what model sizes you can run and how fast tokens generate.

1 — VRAM: LLM GPU Requirements by Model Size

VRAM is the top LLM GPU requirement — the hard ceiling for any self-hosted LLM. If the model doesn't fit in VRAM, no other spec matters. The VRAM requirements below use a 14B parameter model as reference — scale proportionally for larger models.

Note on Q4_K_M (Ollama default): to avoid the significant quality degradation of pure 4-bit quantization, Ollama uses a "mixed precision" approach — core weights at 6-bit, less critical weights at 4-bit. This keeps quality loss minimal while keeping a 35B model at approximately 22–23 GB VRAM.

KV Cache VRAM Requirements: Context Length & Concurrency

Beyond model weights, inference requires KV Cache memory for every active request. For Qwen2.5-14B at FP16 (24 layers, 8 KV heads, 64 dims per head):

2 — Memory Bandwidth: How Fast Tokens Generate

Generating each token requires loading the full model weight set from VRAM into Tensor Cores. Memory bandwidth — a key LLM GPU requirement — is the hard ceiling on tok/s, not TFLOPS. A100-40G at 1,555 GB/s on a 20B FP16 model: theoretical max ≈ 38 tok/s for a single request.

3 — Tensor Core Precision: Where Blackwell Pulls Ahead

On GPUs with native FP4/FP8 support: FP4 compute is 2× FP8, and FP8 is 2× FP16. FP8-quantized model weights also use half the VRAM of FP16, and FP4 uses a quarter — freeing more VRAM for context and concurrency. Only Blackwell GPUs can run LLMs at FP4 precision, achieving maximum possible throughput.

Best GPUs for LLM Workloads

Finding the best GPU for LLM workloads depends on model size, concurrency, and budget. All specs validated on GPU Mart LLM hosting infrastructure. Source: NVIDIA official specification documents. Pricing: gpu-mart.com/pricing.

Tensor Cores Dense: FP16 dense compute (TFLOPS).  AI TOPS: max compute at lowest supported precision (sparse).  Precision: natively supported precisions.

Quick-Decision: Best GPU for LLM by Use Case

All plans: flat-rate monthly billing, unmetered bandwidth, no hidden fees. Pricing subject to change — verify at gpu-mart.com/pricing.

Real Inference Benchmark Data

vLLM framework · Input 1,024 tokens + Output 512 tokens · GPU Mart production hardware · Optimized for text generation and LLM inference

Ollama Single-User Benchmarks

llama.cpp backend · Q4_K_M quantization · Ideal for LLM VPS hosting in single-user and dev environments · No KV Cache pre-allocation · Input 1,024 tokens + Output 512 tokens · Single user · Average of 10 requests

VRAM Usage & Maximum Supported Models

Context reduction tip: set num_ctx to reduce VRAM and run 35B models on 32 GB cards:

Generation Speed by GPU

Inference Framework Selection

Framework choice affects throughput and latency as much as GPU selection — pick the right one for your LLM inference use case.

Deployment Optimization Tips

Auxiliary Models for <16 GB GPUs

Real Customer Deployments

Three detailed case studies + 13 industry deployment records. Data partially anonymized.

Additional Industry Deployment Records

Who This Is (and Isn't) For

Choosing the right GPU for self-hosted LLM workloads on dedicated infrastructure is not right for every team. Here is the honest breakdown.

Frequently Asked Questions

What is the difference between GPU VPS and a dedicated GPU server?

GPU VPS uses PCIe Passthrough to give you exclusive, non-shared access to physical GPU hardware — near bare-metal performance at lower cost. Suitable for most 7B–70B LLM server workloads. A dedicated GPU server gives you the entire physical machine exclusively: right for production AI server deployments, multi-GPU inference, training, and zero-tolerance performance workloads.

Which inference frameworks come pre-installed, and how do I enable them?

NVIDIA drivers are pre-installed on all plans. At deploy time, select from 20+ pre-configured AI frameworks including Ollama, ComfyUI, Qwen3, and Gemma3. One-click deployment from the control panel under All Products → App.

My LLM inference service is hitting OOM. What can I do without switching hardware?

In priority order: (1) Verify vLLM continuous batching is on (default); (2) quantize to INT4/INT8 via AWQ or GPTQ — 2–4× VRAM reduction; (3) reduce batch_size and max_tokens; (4) in Ollama set num_ctx explicitly; (5) last resort: CPU offloading — significant latency penalty.

How do I calculate the VRAM requirements for my LLM?

LLM GPU requirements for VRAM: Total ≈ model weights + KV Cache + 20–30% headroom. Model weights = param count (B) × precision bytes (FP16=2, INT8=1, INT4=0.5). KV Cache = 2 × layers × KV heads × head dims × precision bytes × context length × concurrency / 1e9 (GB). Example: 7B FP16 → 14 GB; same at INT4 → ~3.5 GB.

Should I use a single large-VRAM GPU or multiple smaller cards for LLM hosting?

Always prefer single large-VRAM where the model fits (A6000 48G, A100 80G, Pro 6000 96G): lowest latency, no inter-GPU communication overhead, simplest deployment. Only use multi-GPU tensor parallelism (--tensor-parallel-size 2+) when the model literally cannot fit on one card.

vLLM vs SGLang — which inference framework should I choose?

Choose vLLM for general production API, widest model compatibility (AMD/TPU/Trainium), mixed workloads. Choose SGLang for RAG, multi-turn dialogue, AI Agents (RadixAttention +29% vs vLLM), DeepSeek-class reasoning models.

How do I design rate limiting and concurrency control for a production LLM API?

Per-user RPM/TPM limits; max 1–3 concurrent requests per user; queue excess requests rather than rejecting; set max input/output tokens. Best practice: API gateway (Kong/Nginx) for auth + rate limiting, vLLM backend for batching and queuing.

What is the GPU hosting pricing model and how do I estimate monthly cost?

Transparent monthly billing with 1/3/12/24-month options. VPS: RTX Pro 2000 (16G) $95.20/mo (20% OFF); RTX Pro 4000 (24G) $159/mo (20% OFF); RTX Pro 5000 (48G) $269/mo; RTX 5090 (32G) $399/mo; RTX Pro 6000 (96G) $479/mo. Dedicated: A6000 $409/mo; A100-40G $360/mo (55% OFF); A100-80G $1,559/mo; H100-80G $2,099/mo. No hidden fees. See gpu-mart.com/pricing.

How do I deploy Hugging Face models or private models on a GPU server?

CUDA and drivers are pre-installed. Private models upload directly from local; HuggingFace models via git clone or huggingface-cli. Launch your self hosted LLM inference service with vLLM/TGI/Ollama (OpenAI-compatible API); expose REST API via port. Most users complete deployment in minutes.

Is OpenAI-compatible API supported? How do I migrate existing code to a self-hosted LLM?

vLLM/SGLang/Ollama all expose OpenAI-format compatible endpoints on your GPU server. Migration requires only changing base_url and api_key — no business logic changes. Most teams complete the switch in under 5 minutes.

Can I upgrade my GPU VPS or dedicated server configuration later?

You can upgrade to a higher GPU VPS tier or dedicated server at any time. Adding extra GPUs to the same server after deployment is not supported — select a multi-GPU plan at initial deployment.

Is there a free trial available for GPU hosting?

GPU Mart offers hourly pay-as-you-go billing for quick testing. A 24-hour free trial is also available — verify your actual workload before committing to a monthly plan.

Can I self-host an LLM on a GPU VPS?

Yes. GPU Mart LLM VPS hosting uses KVM PCI GPU Passthrough, giving you exclusive access to physical GPU hardware — the same GPU performance as a dedicated server at a lower price. You get root access to install vLLM, Ollama, or any inference framework, and can expose an OpenAI-compatible API endpoint. VPS plans start from $95/mo with 16 GB VRAM, suitable for running 7B–14B parameter models. For larger models (27B–70B+), 48–96 GB VRAM VPS options are available from $269/mo.

How much VRAM is required to self-host a 70B LLM?

VRAM requirements for a 70B model depend on quantization: FP16 (full precision) requires ~140 GB — beyond a single card; INT8 quantization requires ~70 GB, fitting on an A100-80G or H100-80G; INT4/AWQ/GPTQ quantization reduces this to ~35 GB, runnable on an RTX Pro 5000-48G or A6000-48G with two cards. For most production use cases, INT4 quantization of a 70B model on a single 48 GB GPU delivers good quality with practical latency. If quality loss from quantization is unacceptable, use two 48 GB GPUs with tensor parallelism.

What is the best GPU for self-hosted LLMs?

The best GPU for self-hosted LLM workloads depends on model size and budget. For 7B–14B models: RTX Pro 4000 (24G, $159/mo) or RTX 5090 (32G, $399/mo) for speed. For 27B–35B models: RTX Pro 5000 (48G, $269/mo) is the best value — 48 GB VRAM, Blackwell FP8/FP4 native, 178 tok/s on Ollama. For 70B+ models: A100-80G ($1,559/mo) or H100-80G ($2,099/mo). For maximum price-to-performance on FP8 models: RTX 5090 hits 144 tok/s on 8B-FP8 at $399/mo, comparable to H100 at 1/5 the cost.

Is self-hosting an LLM cheaper than using a cloud API?

For high-volume use cases, yes — significantly. A team calling GPT-4o at $5/M input + $15/M output tokens, running 2M tokens/day, spends ~$20/day or $600/month. A GPU VPS at $269/mo (RTX Pro 5000, 48G) handles the same workload with no per-token cost. Break-even is typically 1–2 months, with 50–70% long-term savings. For very low-volume usage (under 500K tokens/day), API pricing may still be more economical than maintaining dedicated GPU infrastructure.

What GPU do I need to run a 32B or 70B parameter LLM?

For a 32B model: INT4 quantization (AWQ/GPTQ) requires ~16–18 GB VRAM — fits on an RTX Pro 4000 (24G) or RTX 5090 (32G). FP16 requires ~64 GB — needs an A100-80G or two 48G cards. For a 70B model: INT4 quantization requires ~35–40 GB — fits on an RTX Pro 5000 (48G) or A6000 (48G). FP16 requires ~140 GB — requires two A100-80G or H100-80G cards with tensor parallelism. Recommended for most teams: run 32B at INT4 on RTX Pro 5000 ($269/mo) or 70B at INT4 on two A6000s ($818/mo combined).