These are some notes on how to improve CPU only inference on older hardware in a VM with proxmox.
I have a few use cases where I need to parse chat messages. These parse runs do not require large context and can be done with very low output/input token counts. Also, these tools had to run as cron so they couldn’t reside on my Foundary Machine, which gets shut down when not in use.
The Ironkeep was the chosen location for the VM that would do these tiny inferences. I configured a VM with Ubuntu Server 26.04 in proxmox and quickly got llama.cpp up and running with CPU inference support. Make sure to enable OpenBLAS as recommended on their getting started page.
The model of choice was the Unsloth Gemma 4 E4B GGUF at Q6_K quant with MTP draft model. Here is what I use to run it at decent context sizes:
./llama-server \
--model <path>/gemma-4-E4B-it-Q6_K.gguf \
--model-draft <path>/mtp-gemma-4-E4B-it.gguf \
--spec-type mtp \
--spec-autotune \
-fa on \
--jinja \
-c 65536 \
--temp 1.0 \
--top-p 0.95 \
--top-k 64 \
--host 192.168.1.150 \
--port 8888 \
--verbosity 4
First optimisation
The first run was not satisfying at all. I was getting 10 t/s prompt processing and 5 t/s generation. As per my research, it was not supposed to be this bad. This speed would not do at all, even for basic message parsing. After going through the documentation and some more forums, I noticed that the CPU inference engine was not getting AVX support. This was a quick fix as I remembered that Proxmox can pass through CPU features. Enabling the correct settings and rebooting the VM, I was granted a big boost in performance!
Prompt processing jumped up to around 50 t/s and decode speed to around 11 t/s (5x on PP and 2x on decode).
Speed greed continues
My research had also led me to ik_llama.cpp and with all the praise about it, I couldn’t resist trying out a build. After having built llama.cpp, it was easy enough to clone ik_llama.cpp and cmake away. The command to run llama-server here is almost identical (with just a few params that are old style). The results were not disappointing at all. My prompt processing jumped up from 50 t/s to around 75 t/s and decode went up to 15 t/s from 11 t/s. These are gains worth keeping. Thank you ik_llama.cpp developers for this amazing fork!
Summary
Here is a table with the results of different optimizations:
| Optimization Stage | Prompt Processing (t/s) | Decode Speed (t/s) | Notes |
|---|---|---|---|
| Initial run | 10 | 5 | Basic setup, no AVX support |
| After AVX support | 50 | 11 | 5x improvement on PP, 2x on decode |
| With ik_llama.cpp | 75 | 15 | 1.5x improvement on PP, ~1.4x on decode |
I might move one of my lower-end GPUs to this server at some point, but for now, I am thrilled with the results. It is amazing how local LLMs are improving so quickly, and the community wide effort is enabling any kind of hardware to run basic inference.
Glossary
PP (Prompt Processing) — The phase where the LLM reads and processes the input prompt before generating output. Measured in tokens per second (t/s).
TG (Text Generation) / Decode — The phase where the model generates output tokens one by one after processing the prompt. Also measured in t/s.
t/s (tokens per second) — A metric for measuring inference speed. Higher values indicate faster processing or generation.
AVX (Advanced Vector Extensions) — CPU instruction sets that enable SIMD (Single Instruction, Multiple Data) operations, significantly accelerating mathematical computations used in LLM inference.
GGUF — A file format for storing large language models in a quantised format, optimised for efficient loading and inference with llama.cpp.
Quant / Quantisation — The process of reducing model precision (e.g., from 16-bit to 4-bit or 6-bit) to decrease memory usage and improve inference speed with minimal quality loss. Q6_K refers to a 6-bit quantisation method.
MTP (Multi-Token Prediction) — A draft model technique that predicts multiple tokens ahead to speed up generation through speculative decoding.
Context — The amount of text (measured in tokens) that the model can process at once. Larger context windows (like 65536 in the example) allow processing longer documents but require more memory.
Disclaimer: All posts are handwritten (except glossaries). All views and opinions are my own and do not reflect those of my employer or any other organisation. Feedback welcome: nandeepmali@gmail.com.