Local AI has become realistic — within limits
Since Llama 2 and its quantized derivatives came out, the same question reaches our support every week: "can I run an LLM on one of your VPS?". Honest answer: yes, provided you pick the right model size and know what a CPU can — and cannot — do.
Ollama in five minutes
curl -fsSL https://ollama.com/install.sh | sh
ollama run llama2:7b-chat-q4_K_MThat's it. Ollama downloads the quantized model (about 4 GB for a 7B in Q4) and exposes a local API compatible with most tools in the ecosystem.
CPU versus GPU: the real numbers
On our test bench, a Quantum VPS (16 vCore EPYC, 64 GB DDR5):
| Model | Tokens/s (generation) | RAM required |
|---|---|---|
| 7B Q4 | 11 | 6 GB |
| 13B Q4 | 6 | 10 GB |
| 34B Q4 | 2.5 | 22 GB |
| 70B Q4 | 0.8 | 42 GB |
Reading: up to 13B, the experience is comfortable for solo interactive use. Beyond that, generation gets slower than human reading speed. A consumer GPU does 10 to 20 times better — but costs 10 to 20 times more to rent.
Two factors dominate on CPU: memory bandwidth (the 12 DDR5 channels of our EPYCs are a real asset) and frequency. The Ryzen 9 at 5.7 GHz pushes 14 tokens/s on a 7B — better than EPYC for a single user, worse once requests run in parallel.
Use cases that work
- Internal chatbot over your documents (RAG): a 7B or 13B is largely enough.
- Summarization and extraction as background jobs: speed hardly matters.
- Sensitive data processing: nothing leaves your VPS — the number one reason our customers self-host.
- Prototyping before investing in GPU capacity.
What doesn't work
Serving 50 simultaneous users in real time on a 34B, fine-tuning a model, or approaching GPT-4 quality. For those cases you need a GPU — and we'd rather tell you before you learn it in production.
Our recommendation
A Quantum with 64 GB of RAM covers every realistic use case today. And always benchmark with your own prompts: context length influences speed as much as model size does.