Ollama is a self-hosted AI platform designed to run open-source large language models. It provides quantized versions of popular models, significantly reducing model size and GPU requirements, making it ideal for small-scale projects, lightweight deployments, or early-stage testing. On this page, you’ll find benchmark results across various GPU servers, along with step-by-step setup guides to help you get started quickly and easily.
Excellent AI, Deep Learning
Limited-Time GPU Server Sale!
Powerful AI Servers (GPU Dedicated Server/GPU VPS), Supporting Diverse Models and Platforms!
AI Hosting Sales for Nvidia GPU Server
Enjoy up to 56% off on high-performance GPU hosting servers! Develop and deploy your models with GPU Mart’s Nvidia GPUs, starting at just $56.76 per month.
New Product
GPU Dedicated Server
GPU VPS
New Arrival
Professional GPU VPS- RTX Pro 2000
$ 99.00/mo
1mo3mo12mo24mo
- 30GB RAM
- 16 CPU Cores
- 240GB SSD
- 300Mbps Unmetered Bandwidth
- Once per 2 Weeks Backup
- OS: Linux / Windows 10/ Windows 11
- Dedicated GPU: Nvidia RTX Pro 2000
- CUDA Cores: 4,352
- Tensor Cores: 5th Gen
- GPU Memory: 16GB GDDR7
- FP32 Performance: 17 TFLOPS
New Arrival
Advanced GPU VPS- RTX Pro 4000
$ 159.00/mo
1mo3mo12mo24mo
- 60GB RAM
- 24 CPU Cores
- 320GB SSD
- 500Mbps Unmetered Bandwidth
- Once per 2 Weeks Backup
- OS: Linux / Windows 10/ Windows 11
- Dedicated GPU: Nvidia RTX Pro 4000
- CUDA Cores: 8,960
- Tensor Cores: 280
- GPU Memory: 24GB GDDR7
- FP32 Performance: 34 TFLOPS
New Arrival
Advanced GPU VPS- RTX Pro 5000
$ 269.00/mo
1mo3mo12mo24mo
Order Now- 60GB RAM
- 24 CPU Cores
- 320GB SSD
- 500Mbps Unmetered Bandwidth
- Once per 2 Weeks Backup
- OS: Linux / Windows 10/ Windows 11
- Dedicated GPU: Nvidia RTX Pro 5000
- CUDA Cores: 14,080
- Tensor Cores: 440
- GPU Memory: 48GB GDDR7
- FP32 Performance: 66.94 TFLOPS
New Arrival
Enterprise GPU VPS- RTX Pro 6000
$ 479.00/mo
1mo3mo12mo24mo
- 90GB RAM
- 32 CPU Cores
- 400GB SSD
- 1000Mbps Unmetered Bandwidth
- Once per 2 Weeks Backup
- OS: Linux / Windows 10/ Windows 11
- Dedicated GPU: Nvidia RTX Pro 6000
- CUDA Cores: 24,064
- Tensor Cores: 852
- GPU Memory: 96GB GDDR7
- FP32 Performance: 126 TFLOPS
LLM Frameworks&Tools
LLM frameworks and tools simplify the complexities of working with LLMs by providing APIs, libraries, and utilities that streamline processes like training, inference, and model optimization.
vLLM is a high-performance LLM inference engine built for speed, scalability, and production readiness. Unlike Ollama, vLLM typically runs full-size, non-quantized models downloaded from Hugging Face, offering greater accuracy and performance—ideal for enterprise-grade applications and real-time deployment. On this page, explore vLLM’s capabilities, performance benchmarks across GPU servers, and optimized setup instructions for seamless deployment.
LLM Hosting with Ollama — GPU Recommendation
| Model Name | Size (4-bit Quantization) | Recommended GPUs | Tokens/s |
|---|---|---|---|
| deepSeek-r1:7B | 4.7GB | T1000 < RTX3060 Ti < RTX4060 < A4000 < RTX5060 < V100 | 26.70-87.10 |
| deepSeek-r1:8B | 5.2GB | T1000 < RTX3060 Ti < RTX4060 < A4000 < RTX5060 < V100 | 21.51-87.03 |
| deepSeek-r1:14B | 9.0GB | A4000 < A5000 < V100 | 30.2-48.63 |
| deepSeek-r1:32B | 20GB | A5000 < RTX4090 < A100-40gb < RTX5090 | 24.21-45.51 |
| deepSeek-r1:70B | 43GB | A40 < A6000 < 2A100-40gb < A100-80gb < H100 < 2RTX5090 | 13.65-27.03 |
| deepseek-v2:236B | 133GB | 2A100-80gb < 2H100 | -- |
| llama3.2:1b | 1.3GB | P1000 < GTX1650 < GTX1660 < RTX2060 < T1000 < RTX3060 Ti < RTX4060 < RTX5060 | 28.09-100.10 |
| llama3.1:8b | 4.9GB | T1000 < RTX3060 Ti < RTX4060 < RTX5060 < A4000 < V100 | 21.51-84.07 |
| llama3:70b | 40GB | A40 < A6000 < 2A100-40gb < A100-80gb < H100 < 2RTX5090 | 13.15-26.85 |
| llama3.2-vision:90b | 55GB | 2A100-40gb < A100-80gb < H100 < 2RTX5090 | ~12-20 |
| llama3.1:405b | 243GB | 8A6000 < 4A100-80gb < 4*H100 | -- |
| gemma2:2b | 1.6GB | P1000 < GTX1650 < GTX1660 < RTX2060 | 19.46-38.42 |
| gemma3:4b | 3.3GB | GTX1650 < GTX1660 < RTX2060 < T1000 < RTX3060 Ti < RTX4060 < RTX5060 | 28.36-80.96 |
| gemma3n:e2b | 5.6GB | T1000 < RTX3060 Ti < RTX4060 < RTX5060 | 30.26-56.36 |
| gemma3n:e4b | 7.5GB | A4000 < A5000 < V100 < RTX4090 | 38.46-70.90 |
| gemma3:12b | 8.1GB | A4000 < A5000 < V100 < RTX4090 | 30.01-67.92 |
| gemma3:27b | 17GB | A5000 < RTX4090 < A100-40gb < H100 = RTX5090 | 28.79-47.33 |
| qwen3:14b | 9.3GB | A4000 < A5000 < V100 | 30.05-49.38 |
| qwen2.5:7b | 4.7GB | T1000 < RTX3060 Ti < RTX4060 < RTX5060 | 21.08-62.32 |
| qwen2.5:72b | 47GB | 2A100-40gb < A100-80gb < H100 < 2RTX5090 | 19.88-24.15 |
| qwen3:235b | 142GB | 4A100-40gb < 2H100 | ~10-20 |
| mistral:7b / openorca / lite / dolphin | 4.1–4.4GB | T1000 < RTX3060 < RTX4060 < RTX5060 | 23.79-73.17 |
| mistral-nemo:12b | 7.1GB | A4000 < V100 | 38.46-67.51 |
| mistral-small:22b / 24b | 13–14GB | A5000 < RTX4090 < RTX5090 | 37.07-65.07 |
| mistral-large:123b | 73GB | A100-80gb < H100 | ~30 |
LLM Hosting with vLLM + Hugging Face — GPU Recommendation
| Model Name | Size (16-bit Quantization) | Recommended GPU(s) | Concurrent Requests | Tokens/s |
|---|---|---|---|---|
| deepseek-ai/deepseek-coder-6.7b-instruct | ~13.4GB | A5000 < RTX4090 | 50 | 1375–4120 |
| deepseek-ai/DeepSeek-R1-Distill-Llama-8B | ~16GB | 2A4000 < 2V100 < A5000 < RTX4090 | 50 | 1450–2769 |
| deepseek-ai/deepseek-coder-33b-instruct | ~66GB | A100-80gb < 2A100-40gb < 2A6000 < H100 | 50 | 570–1470 |
| deepseek-ai/DeepSeek-R1-Distill-Llama-70B | ~135GB | 4*A6000 | 50 | 466 |
| meta-llama/Llama-3.2-3B-Instruct | 6.2GB | A4000 < A5000 < V100 < RTX4090 | 50–300 | 1375–7214.10 |
| meta-llama/Llama-3.3-70B-Instruct / 3.1-70B / Meta-3-70B | 132GB | 4A100-40gb, 2A100-80gb, 2*H100 | 50 | ~295.52–990.61 |
| google/gemma-3-4b-it | 8.1GB | A4000 < A5000 < V100 < RTX4090 | 50 | 2014.88–7214.10 |
| google/gemma-2-9b-it | 18GB | A5000 < A6000 < RTX4090 | 50 | 951.23–1663.13 |
| google/gemma-3-12b-it | 23GB | A100-40gb < 2*A100-40gb < H100 | 50 | 477.49–4193.44 |
| google/gemma-3-27b-it | 51GB | 2*A100-40gb < A100-80gb < H100 | 50 | 1231.99–1990.61 |
| Qwen/Qwen2-VL-2B-Instruct | ~5GB | A4000 < V100 | 50 | ~3000 |
| Qwen/Qwen2.5-VL-3B-Instruct | ~7GB | A5000 < RTX4090 | 50 | 2714.88–6980.31 |
| Qwen/Qwen2.5-VL-7B-Instruct | ~15GB | A5000 < RTX4090 | 50 | 1333.92–4009.29 |
| Qwen/Qwen2.5-VL-32B-Instruct | ~65GB | 2*A100-40gb < H100 | 50 | 577.17–1481.62 |
| Qwen/Qwen2.5-VL-72B-Instruct-AWQ | 137GB | 4A100-40gb < 2H100 < 4*A6000 | 50 | 154.56–449.51 |
| mistralai/Pixtral-12B-2409 | ~25GB | A100-40gb < A6000 < 2*RTX4090 | 50 | 713.45–861.14 |
| mistralai/Mistral-Small-3.2-24B-Instruct-2506 | ~47GB | 2*A100-40gb < H100 | 50 | ~1200–2000 |
| mistralai/Pixtral-Large-Instruct-2411 | 292GB | 8*A6000 | 50 | ~466.32 |
Explanation:
Recommended GPUs: From left to right, performance from low to high
Tokens/s: from benchmark data.
Ollama GPU Benchmarks – Model Performance
We've benchmarked LLMs on GPUs including from P1000 to H100. These benchmarks provide insights into how different GPUs perform with Ollama across various model sizes, helping you choose the ideal AI hosting server
vLLama GPU Benchmarks – Model Performance
We've benchmarked LLMs on GPUs includingA5000, T1000, A40, A6000,RTX 4090,Dual RTX 4090, A100 40GB, Dual A100,4xA100 3xV100,A100 80GB,H100, ,4xA6000. Explore the results to select the ideal GPU server for your workload.
What Clients Say about our AI hosting GPU Server?
Delivering exceptional service and support is our highest priority at GPU Mart. Here’s a glimpse of what our clients have said about their experience with our GPU server services.
We’ve been using their GPU servers to run 70B models, and the performance is incredible. The 48GB GPUs handle everything seamlessly, and the setup process was a breeze. Highly recommend for anyone working with large AI models!
We evaluated multiple GPU server providers, and they offer the best value for money. Compared to other vendors, we achieved better performance at a lower cost here. It's especially suitable for teams with limited budgets but high computing power demands.
We needed a reliable server for our 14B models, and their 16GB GPU plan was exactly what we needed. The flexibility to choose our preferred inference engine made deployment so easy. Great service!
We started with a single GPU and upgraded to a multi-GPU setup as our needs grew. The scalability is fantastic, and the pricing is very competitive. Perfect for growing AI teams!
We tested their servers with a free trial, and the performance was so impressive that we signed up immediately. Being able to benchmark our models before committing was a huge plus. Highly recommend!
We needed a server that could handle our unique environment, and they delivered. The ability to customize the setup and choose our own tools made all the difference.
We use the RTX 4090 server for AI-generated artwork, style transfer, and automated image editing, and its performance has exceeded our expectations. Compared to consumer-grade GPUs, this server offers superior stability and computing power, making it ideal for professional AI image processing tasks.
Questions About AI Hosting Promotion
Find answers to your most common questions in our FAQ section. For personalized recommendations or further assistance, don't hesitate to reach out to our online support team.
1. What is an AI hosting server, and how does it work?
GPU Mart's provide GPU-powered physical servers (bare metal) with dedicated IP access. You can remotely log in, choose your preferred LLM inference engine, and deploy your AI models effortlessly.
2. Which platforms are supported?
There are no platform restrictions. However, different platforms may quantize models differently, which can affect the final model size and performance.
3. What GPU memory is required for a 14B model?
We recommend a 16GB GPU for running 14B models efficiently.
4. What GPU memory is required for a 32B model?
For 32B models, we recommend a GPU with 24GB or more memory.
5. What GPU memory is required for a 70B model?
To run 70B models smoothly, we recommend a GPU with 48GB or more memory.
6. When should I choose a multi-GPU plan?
A multi-GPU plan is ideal when a single GPU cannot handle higher concurrency or larger model sizes. If your workload demands more power, consider upgrading to a multi-GPU setup.
7. Can I upgrade my server configuration later?
Yes! You can upgrade GPU memory and storage space. Some servers also support adding additional GPUs. Contact us for custom upgrade options.
8. Can I run benchmarks on my own models before committing?
Yes, we offer free trials for select products. Reach out to us to request a free trial and test your models.
9. Is server maintenance included, or am I responsible for it?
We handle all server maintenance, so you can focus on running your AI tasks without worrying about hardware management.
10. Can I customize the server environment to fit my needs?
Absolutely! You have full control to configure the server environment according to your requirements.
11. Can I use your servers for both inference and training tasks?
Our servers are optimized for inference and reasoning tasks. For training, please contact us to discuss your specific needs.
12. How many GPU servers can I buy with the AI hosting promotion?
Limited for 3 GPU dedicated server plans. If you require bulk purchasing, please contact our sales team for a unique discount arrangement.
13. What's the minimum duration for a GPU server order?
You can order AI hosting GPU server for any duration of one month or longer.
14. What's the meaning of recurring discount?
'Recurring discount' means your discount will still be available when you renew a AI hosting/machine learning server.
15. Can I get a discount for my existing GPU server?
Unfortunately, AI hosting promotions are only available for new GPU server orders. However, you can contact our sales team to inquire about special renewal discounts.
16. Will the discount remain if I upgrade/downgrade the plan after the promotion?
No, the discount will not be valid if the target plan is excluded from the AI hosting GPU server promotion.
17. What payment methods do you accept?
We accept Visa, MasterCard, American Express, JCB, Discover, Diners Club, PayPal, Wire Transfer, and Check. Note that non-instant payment methods will delay service deployment until the payment clears. Wire Transfers must be over $100. Paper checks are only for U.S. clients.
18. How long will it take to set up my server?
Typically, GPU dedicated server setup takes 20-40 minutes. Customized GPU server will take longer.
19. Can I get a free trial before payment?
We offer a 24-hour free trial for new clients who wish to test our GPU server. To request a trial server, please follow these steps:
Step 1: Submit a Free Trial Request
Select a plan, click 'Order Now,' and leave a note saying 'Need free trial.' Then, click 'Check Out' and proceed to the Order Confirm page. On this page, you must click 'Confirm' to complete the free trial request.
Step 2: Security Verification
This process takes about 30 minutes to 2 hours. Once verified, you will receive the server login details in the console and can start using it.If your trial request is not approved, you will be notified via email.
Custom Servers
Can't find your ideal server? Send us your custom requirements, and our sales rep will provide a tailored solution for you.
Server Inquiry
Confused about choosing a server or have questions? Consult online support for recommendations.