LLM inference serving in 2026: revisit vLLM, SGLang and llama.cpp before you put another model behind an API

Manoj Shukla
October 14, 2026

Picture a product engineering team at a mid-size SaaS in Kochi that builds support automation for Indian NBFCs (an invented example, not a real client). About forty engineers, a platform group of six, and a new mandate from the CTO: keep customer chat and document summarisation on infrastructure the company controls, with an OpenAI-compatible HTTP API so product teams do not rewrite clients every quarter. Three proposals land on the same whiteboard. One engineer wants vLLM because every GPU serving blog mentions PagedAttention and continuous batching. A second wants SGLang because the LMSYS stack and RadixAttention look strong for multi-turn agent traffic with shared prefixes. A third wants llama.cpp (or Ollama on top of it) because the laptop demos work, GGUF files are easy to move, and the first pilot might run on a CPU box in the office before anyone buys an H100. They are arguing as if these were three equal products that do the same job. They are not.

Self-hosted LLM inference in 2026 still has two different jobs wearing one phrase. vLLM and SGLang are GPU-first, high-throughput Python serving stacks aimed at packing many concurrent requests onto accelerators. llama.cpp is a portable C/C++ inference engine with llama-server, strong on CPU, Apple Silicon, edge boxes and GGUF quantisation, and usable on GPUs too. Ollama is a convenience wrapper and model packaging layer that builds on llama.cpp ideas and binaries for local and small-team use; it is not a fourth peer engine in the same class as vLLM or SGLang. This post compares where vLLM, SGLang and llama.cpp stand on 14 October 2026: history, what each actually is, OpenAI-compatible APIs, hardware, quantisation, batching and caching, speculative decoding, structured output, multi-GPU, Kubernetes notes, observability, security, and a decision guide. Facts come from GitHub releases, LICENSE files at tags, official docs and CVE or GitHub advisory records verified against NVD or cve.org where an ID exists. The drill near the end is my own run on this Linux box without a GPU, and I label clearly what I could and could not run.

The short version

Decide first what you are serving: high-concurrency GPU API traffic, or portable single-node / CPU / laptop / edge inference. GPU servers and portable engines solve different bottlenecks.
vLLM (vllm-project/vllm) is at v0.31.0 (5 October 2026, 12:14 PM IST). Apache 2.0. Born at UC Berkeley Sky Computing Lab around PagedAttention; ships an OpenAI-compatible server, continuous batching, prefix caching, speculative decoding, structured outputs, tensor and pipeline parallel options, and a /metrics Prometheus endpoint. README lists quantisation including FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF and more.
SGLang (sgl-project/sglang) is at v0.5.21 (2 October 2026, 6:39 AM IST). Apache 2.0. LMSYS-rooted stack built around RadixAttention (prefix-tree KV reuse). OpenAI-compatible APIs (plus Anthropic and Ollama-compatible routes in docs), speculative decoding families, structured outputs (XGrammar default), multi-GPU parallelism, Kubernetes / LeaderWorkerSet docs, and Prometheus metrics behind --enable-metrics. Hardware docs cover NVIDIA, AMD, Apple Metal, Ascend NPU, CPU, TPU, XPU and Moore Threads.
llama.cpp (ggml-org/llama.cpp, formerly under ggerganov) stable tag v0.6.0 (5 October 2026, 10:26 PM IST), with frequent b* build tags (for example b11598 on 14 October 2026, 2:06 AM IST, marked prerelease on GitHub). MIT Licence. Portable C/C++ engine with llama-server: OpenAI-compatible /v1/chat/completions, optional --metrics, GGUF-native quantisation, CPU/Metal/Vulkan/CUDA/ROCm/SYCL style backends via build or release binaries.
Ollama (ollama/ollama) is at v0.40.3 (11 October 2026, 7:09 AM IST), MIT Licence. Treat it as a llama.cpp-based local/ops wrapper with model library UX, not as a third high-throughput GPU cluster server beside vLLM and SGLang.
Security is active on all three codebases. vLLM has a long 2025-2026 advisory history (Mooncake RCE, API DoS, video-processing RCE, config auto_map RCE, and several October 2026 advisories fixed in 0.31.0). SGLang has critical deserialization and endpoint issues (including CVE-2026-5760, CVE-2026-7304, CVE-2026-93034). llama.cpp has repeated GGUF parser and (out-of-scope per SECURITY.md) RPC issues; load only trusted models and do not expose RPC. Pin versions, put a reverse proxy with auth and body limits in front, and never treat an open /v1 port as safe.

Where things stand on 14 October 2026

Dates are from each project’s GitHub release published_at unless marked otherwise; times are converted from UTC to IST (UTC+5:30). Licences are from the LICENSE files at the tagged releases.

Project

Release

Date (IST)

Notes

Licence

vLLM (vllm-project/vllm)

v0.31.0

5 October 2026, 12:14 PM

Current release at research time; several GHSA fixes land in this line

Apache 2.0

vLLM

v0.30.0

22 September 2026, 10:50 AM

Prior minor; multiple medium advisories fixed here

Apache 2.0

vLLM

v0.29.0

9 September 2026, 2:24 PM

Prior minor

Apache 2.0

SGLang (sgl-project/sglang)

v0.5.21

2 October 2026, 6:39 AM

Current release

Apache 2.0

SGLang

v0.5.20

19 September 2026, 4:11 AM

Prior

Apache 2.0

SGLang

v0.5.19

5 September 2026, 7:57 AM

Last noted CUDA 12 image lane in install docs narrative

Apache 2.0

llama.cpp (ggml-org/llama.cpp)

v0.6.0

5 October 2026, 10:26 PM

Current stable v* tag (not prerelease)

MIT

llama.cpp

v0.5.0

24 September 2026, 2:20 AM

Prior stable

MIT

llama.cpp

b11598

14 October 2026, 2:06 AM

Latest b* build used in the drill; GitHub marks prerelease

MIT

Ollama (ollama/ollama)

v0.40.3

11 October 2026, 7:09 AM

Current non-prerelease; wrapper context only

MIT

Ollama

v0.40.0

25 September 2026, 9:01 AM

Prior minor

MIT

Practical notes on the table. llama.cpp publishes both infrequent stable v* tags and very frequent b* build tags with platform binaries; production pin policy should say whether you follow v* or a known-good b*, and should rebuild or redeploy when GGUF security fixes land. SGLang install docs on this research day state CUDA 13 as the current wheel/image line and call out that CUDA 12 images stopped after the 0.5.19 era; read the install page for the week you upgrade, not a blog from last quarter. Ollama versions move quickly; cite it when the team wants a laptop or single-host UX, not when comparing datacenter throughput servers.

Timeline

Date (IST)

Event

10 March 2023 onwards

llama.cpp public history (repository active in March 2023; MIT; GGUF ecosystem grows through 2023-2024)

20 June 2023

vLLM v0.1.0 GitHub release; PagedAttention blog and paper path (arXiv 2309.06180)

August 2023 era

GGUF format work in the ggml / llama.cpp ecosystem (superseding older ggml bin layouts for distribution)

17 January 2024

LMSYS blog introducing SGLang / RadixAttention

18 January 2024

Early SGLang GitHub release line (v0.1.5 among early tags)

29 April 2025

vLLM GHSA-hj4w-hm2g-p6w5 / CVE-2025-32444 (Mooncake pickle/ZMQ RCE) published

14 June 2025

llama.cpp GHSA-8wwf-w4qm-gpqr (malicious GGUF vocabulary buffer overflow); fixed from b5662

10 July 2025

CVE-2025-53630 / GHSA-vgg9-87g3-85w8 (GGUF parser integer overflow path)

21 August 2025

vLLM CVE-2025-48956 / GHSA-rxc4-3w6r-4v47 (HTTP header DoS) published

1 December 2025

vLLM CVE-2025-66448 / GHSA-8fr4-5q9j-m8gm (get_config / auto_map RCE) published

2 February 2026

vLLM CVE-2026-22778 / GHSA-4r2x-xpjr-7cvv (video processing RCE) published

12 March 2026

llama.cpp CVE-2026-27940 / GHSA-3p4r-fq3f-q74v (GGUF mem_size overflow bypass); fixed in b8146

20 April 2026

SGLang CVE-2026-5760 (rerank chat_template RCE); NVD end-excluding 0.5.11

18 May 2026

SGLang CVE-2026-7304 / GHSA-36m8-w8qf-g76p (custom logit processor dill RCE)

2 October 2026

SGLang v0.5.21

5 October 2026

vLLM v0.31.0 and llama.cpp v0.6.0

8 October 2026

SGLang CVE-2026-93034 / GHSA-w4hv-c72g-pqx6 (ZMQ pickle path) published on NVD

11 October 2026

Ollama v0.40.3

14 October 2026

llama.cpp b11598; this post’s research and CPU drill day

The history matches the framing: vLLM popularised paged KV management for high-throughput GPU serving; SGLang pushed radix/prefix-tree reuse for multi-turn and structured workloads; llama.cpp made local and quantised GGUF serving ordinary on machines without a datacenter GPU.

First question: what are you putting behind the API?

Before comparing logos, write down the failure the Kochi team actually fears:

Many concurrent chat sessions on one or more NVIDIA GPUs, with shared system prompts and tool schemas: this is vLLM or SGLang territory.
A regulated pilot on a CPU VM or an Apple Mac mini, with a 3B-8B GGUF and no CUDA: this is llama.cpp (or Ollama for a simpler install), not a fair fight against a CUDA-only cluster stack.
Product clients already speak OpenAI SDKs: all three can expose OpenAI-shaped routes, but feature completeness (tools, vision, responses API, grammars) differs by version; test the exact routes you need.
Embeddings and vector search sit next door: serving an embedding model is related but not identical to chat serving; this site’s vector search post covers the database side.

If you only install Ollama, you still have not designed multi-GPU tensor parallel for a 70B peak load. If you only install vLLM, you still have not solved offline laptops for field engineers. If you only install llama.cpp on one CPU box, you still have not solved thousand-QPS GPU batching. The first design choice is the deployment class, not the Twitter thread.

What each project actually is

vLLM 0.31.0

vLLM is a library and server for fast LLM inference and serving. The project README states it was originally developed in the Sky Computing Lab at UC Berkeley and centres on efficient KV memory management with PagedAttention, continuous batching, chunked prefill and prefix caching. The OpenAI-compatible server is the usual production entrypoint: clients talk HTTP /v1/... while the engine schedules batches onto GPU workers.

Conceptual pieces you will configure in real clusters:

Engine and scheduler: continuous batching of decode and prefill work.
KV cache manager: paged layout so fragmentation wastes less GPU memory than naive contiguous caches.
Parallelism: tensor parallel and related distributed strategies for multi-GPU; docs also cover data parallel and expert parallel deployment pages in the current doc tree.
Features: automatic prefix caching, speculative decoding, structured outputs, quantisation backends, multimodal paths.
Ops: Docker and Kubernetes deployment docs, Helm and several ecosystem operators (llm-d, KServe, production-stack and others listed in the Kubernetes guide), Prometheus metrics at /metrics.

vLLM is the default mental model when someone says “self-host an OpenAI-compatible GPU API” in 2026. Its limits are the ones operators hit in practice: CUDA/accelerator stack complexity, large images, and a long security advisory history that rewards pinning and fronting the server.

SGLang 0.5.21

SGLang is a high-performance serving framework for large language and multimodal models, documented at docs.sglang.io. The distinctive idea from the January 2024 LMSYS introduction is RadixAttention: a radix-tree style reuse of KV cache across requests that share prefixes (system prompts, few-shot templates, multi-turn history). The docs index also covers session-aware radix cache, hierarchical KV caching (HiCache), speculative decoding (EAGLE, MTP, DFlash, ngram and others), structured outputs, PD disaggregation, expert parallelism and more.

API surface from the docs:

OpenAI-compatible completions, vision and embeddings.
Anthropic-compatible /v1/messages.
Ollama-compatible API routes.
Native SGLang APIs and offline engine APIs.

Hardware pages list NVIDIA GPUs, AMD GPUs, Apple Silicon with Metal, Ascend NPUs, CPU server, NVIDIA Jetson, TPU, XPU and Moore Threads. Install docs emphasise Docker images (lmsysorg/sglang) and pip/uv on CUDA 13 for current releases. Kubernetes guidance includes example manifests and LeaderWorkerSet-oriented multi-node notes, plus the OME operator link for enterprise-style management.

SGLang is not “vLLM with a different logo”. It optimises hard for prefix-heavy and structured agent workloads, and its security story in 2026 includes several critical issues around pickle/dill deserialization and exposed admin-style endpoints. Treat network exposure as hostile.

llama.cpp (v0.6.0 / b11598) and Ollama

llama.cpp is LLM inference in C/C++. The README describes install via curl installer, Docker, pre-built release binaries and source builds. llama-server is an HTTP server based on cpp-httplib with:

OpenAI-compatible chat completions, responses and embeddings routes.
Optional Anthropic-compatible /v1/messages.
Parallel slots / multi-user decoding.
Schema-constrained JSON / grammar sampling.
Speculative decoding options.
Optional Prometheus metrics via --metrics.
Prompt caching controls.

Weights are typically GGUF. Quantisation is a first-class distribution format (Q4_K_M, Q5_K_M, IQ variants and many others in the ecosystem), which is why llama.cpp dominates laptop and CPU demos. Multi-GPU split modes in the server help text include none, layer (default, pipelined), row and experimental tensor. Release assets on b* tags cover ubuntu-x64 CPU, CUDA, ROCm, Vulkan, SYCL, OpenVINO, macOS, Android and more.

Ollama packages model pull/run UX around this world. Its README points at llama.cpp among foundations; licence at v0.40.3 is MIT. For this post it is a footnote: useful for developer laptops and small internal tools, weaker as the comparison point for multi-node GPU serving next to vLLM and SGLang.

Architecture comparison

Concern

vLLM 0.31

SGLang 0.5.21

llama.cpp llama-server

Primary design centre

High-throughput GPU serving (PagedAttention)

High-throughput serving with RadixAttention prefix reuse

Portable C++ engine + HTTP server (GGUF)

Typical language / runtime

Python + CUDA/HIP kernels

Python + accelerator kernels

C/C++ (optional Python bindings elsewhere)

OpenAI-compatible HTTP

Yes (OpenAI-compatible server)

Yes (plus Anthropic / Ollama-compatible docs)

Yes (/v1/chat/completions and related)

Prefix / KV reuse story

Automatic prefix caching

RadixAttention + HiCache family

Prompt cache / slot cache (--cache-prompt, slot save/restore)

Structured output

Structured outputs feature docs

XGrammar / Outlines / llguidance

Grammar and JSON schema sampling

Speculative decoding

Feature docs present

EAGLE / MTP / DFlash / ngram and more

Draft / ngram speculative flags on server

Multi-GPU

Tensor / pipeline / expert / data parallel doc areas

TP/PP/EP/DP and disaggregation docs

--split-mode layer/row/tensor

Metrics

/metrics on OpenAI server

/metrics with --enable-metrics

/metrics with --metrics

Licence (tag)

Apache 2.0

MIT

Best first fit

GPU API clusters

GPU API clusters with heavy shared prefixes / agents

CPU, Apple, edge, simple GGUF hosts

Symptom to stack (same table, different verbs)

Symptom

Likely fit

GPU util low while request queue grows

vLLM or SGLang continuous batching tuning; check sched and max concurrency

Multi-turn agent with huge identical system prompt

SGLang radix / prefix features; or vLLM prefix caching

Need a 7B model on a CPU VM this week

llama.cpp GGUF (or Ollama)

Product already uses OpenAI Python SDK

Any of the three; verify tools/vision routes on your pin

Security review hates pickle on the wire

Read advisories; avoid optional unsafe flags; keep servers off the public internet

OpenAI-compatible API support

All three advertise OpenAI-shaped HTTP APIs, which is why the Kochi team can share one client. Details differ:

vLLM: OpenAI-compatible server is a first-class doc section; /metrics rides on that server. Tool calls, reasoning outputs and multimodal routes exist in the current doc tree; test the exact model and flags.
SGLang: dedicated OpenAI API docs for completions, vision and embeddings; also Anthropic /v1/messages and Ollama-compatible routes for broader client ecosystems.
llama.cpp: server README documents /v1/models, /v1/completions, /v1/chat/completions, /v1/responses, /v1/embeddings, plus Anthropic /v1/messages. Optional API key and CORS warnings appear in server logs when left open (the drill saw a warning that no API key was set and CORS allowed all origins).

Compatibility is “enough for many apps”, not “byte-identical to OpenAI for every field”. Pin server versions in contract tests.

Hardware support (honest matrix)

Platform

vLLM

SGLang

llama.cpp

NVIDIA CUDA

Primary path

Primary path (docs: CUDA 13 current images)

CUDA release binaries / builds

AMD ROCm / HIP

Documented GPU install paths; quant chart includes AMD GPU column

AMD GPUs hardware page; ROCm binaries in wild

ROCm ubuntu release assets

Intel GPU (XPU)

Intel GPU column in quant docs; XPU install area

XPU hardware page

SYCL / OpenVINO release assets

Intel / x86 CPU

CPU install / demos (docs note CPU is not on par with GPU)

CPU server hardware page

Strong CPU path (used in drill)

Apple Metal

vLLM-Metal docs area in nav

Apple Silicon with Metal page

First-class Metal builds / macOS assets

Google TPU

TPU docs / tpu-inference project notes

TPU hardware page; 2026 LMSYS/Google blog mentions

Not the headline path

Ascend NPU / others

Ecosystem varies

Ascend and Moore Threads pages

Vendor backends vary by build

Rule of thumb for Kochi: if the budget is one NVIDIA L40S or A100, start with vLLM or SGLang. If the budget is “prove the prompt on a CPU box before procurement”, start with llama.cpp.

Quantisation formats

vLLM: docs list AutoAWQ, BitsAndBytes, GPTQModel, Intel Neural Compressor, LLM Compressor (FP8/INT4/INT8 paths), NVIDIA Model Optimizer, online quantisation, AMD Quark, TorchAO, GGUF and quantized KV cache. Hardware compatibility differs by method (see the project’s quantisation compatibility table).
SGLang: offline and online quantisation; methods in docs include fp8, mxfp4, mxfp8, awq, gptq / gptq_marlin, awq_marlin, gguf, modelopt / modelopt_fp4, bitsandbytes, petit_nvfp4 on ROCm, and Ascend-oriented MX paths.
llama.cpp: GGUF quant families are the distribution standard (Q and IQ types). Server also allows KV cache typed at lower precision (--cache-type-k/v).

Do not assume a GPTQ file that runs in vLLM loads unchanged in llama.cpp. Plan conversion and evaluation as a task.

Batching, prefix caching, speculative decoding, structured output

Feature

vLLM

SGLang

llama.cpp server

Continuous / concurrent batching

Core

Multi-slot parallel decoding

Prefix caching

Automatic prefix caching docs

RadixAttention + session/HiCache docs

--cache-prompt, slot cache, cache RAM limits

Speculative decoding

Speculative decoding feature

EAGLE2/3, MTP, DFlash, ngram, adaptive, …

--spec-type draft/ngram family

Structured output

Structured outputs docs

XGrammar (default), Outlines, llguidance

--grammar, --json-schema

For agent traffic with a 2k-token system prompt repeated thousands of times per hour, prefix reuse is not a micro-optimisation; it is the difference between fitting the KV cache and OOMing. That is exactly where SGLang’s radix story and vLLM’s prefix caching both matter, and where a naive one-request-at-a-time llama.cpp setup will look fine in a demo and fall over in production concurrency.

Multi-GPU and Kubernetes notes

Multi-GPU. vLLM and SGLang document tensor parallel size flags and richer parallel modes (pipeline, data, expert, disaggregation). llama.cpp server exposes --split-mode and tensor split options for multi-GPU hosts without pretending to be a full cluster scheduler.

Kubernetes. vLLM’s Kubernetes guide covers CPU and GPU Deployments plus a long list of higher-level options (Helm, llm-d, KServe, production-stack, LWS and others). SGLang documents Kubernetes manifests, Docker Compose, SkyPilot and LeaderWorkerSet-style multi-node examples, and points at the OME operator. llama.cpp usually enters the cluster as a container running llama-server behind a Service; it does not need a specialised operator for a single replica, but you still need GPU operators, probes, and model volume management. Metrics from all three can feed Prometheus; long-term storage choices are covered in the Prometheus long-term storage post. Continuous profiling of native kernels is a separate discipline; see the continuous profiling post if you chase GPU or CPU flame graphs.

Observability

vLLM: Production Metrics docs; scrape /metrics on the OpenAI-compatible server (Prometheus text format).
SGLang: enable with --enable-metrics; scrape /metrics; docs ship Grafana dashboard JSON under examples/monitoring.
llama.cpp: enable with --metrics; scrape /metrics for llamacpp:* series (prompt tokens, predicted tokens, speculative counters, request gauges).

Minimum useful red panel for Kochi: request latency histogram, tokens/second, KV/cache hit rate if exposed, GPU memory if from DCGM or vendor exporters, and 5xx rate at the reverse proxy.

Security (verified advisories)

I checked project GitHub advisory APIs and, for CVE IDs below, NVD (totalResults: 1) and cve.org HTTP 200 on 14 October 2026. Severity labels come from GitHub advisories unless noted.

vLLM (selected)

Severity

Affected (advisory)

Fixed

Notes

CVE-2025-32444 / GHSA-hj4w-hm2g-p6w5

Critical

>=0.6.5 <0.8.5

0.8.5

Mooncake integration pickle/ZMQ RCE; non-Mooncake deployments not in scope per advisory

CVE-2025-48956 / GHSA-rxc4-3w6r-4v47

High

>=0.1.0 <0.10.1.1

0.10.1.1

Oversized HTTP header DoS

CVE-2025-66448 / GHSA-8fr4-5q9j-m8gm

High

<0.11.1

0.11.1

auto_map / get_config RCE even with trust_remote_code=False

CVE-2026-22778 / GHSA-4r2x-xpjr-7cvv

Critical

>=0.8.3 <0.14.1

0.14.1

RCE in video processing path

GHSA-h3rc-6mm3-gc2m (no CVE in API)

High

>=0.7.3 <0.31.0

>=0.31.0

Request-controlled mm_processor_kwargs.code_revision RCE

Several medium GHSAs

Medium

ranges ending <0.30.0 / <0.31.0

0.30.0 / 0.31.0

Multimodal cache, salt, Prometheus label cardinality, and related

Also note GHSA-ggpf-24jw-3fcw (critical): malicious model load issues tied to PyTorch <2.6.0 / vLLM <0.8.0 guidance. Keep torch and vLLM pairs inside the project’s supported matrix.

SGLang (selected)

Severity

Affected / fixed notes

Notes

CVE-2026-5760

(NVD record; CERT/CC refs)

NVD CPE end-excluding 0.5.11

/v1/rerank path with malicious tokenizer chat_template / unsandboxed Jinja2

CVE-2026-7304 / GHSA-36m8-w8qf-g76p

Critical

pip sglang >=0.4.1.post7, <=0.5.12; first_patched_version null in GHSA

Unauthenticated RCE when --enable-custom-logit-processor deserialises via dill.loads

CVE-2026-93034 / GHSA-w4hv-c72g-pqx6

Critical

Published 8 October 2026; no patched package version in the advisory snapshot I pulled

ZMQ message decoder pickle path; dangerous if data-parallel attention is enabled with non-loopback dist init

Older 2026 CERT write-ups also discuss pickle/ZMQ style issues in multimodal and disaggregation features. Practical mitigation shared across them: do not expose SGLang ports to untrusted networks, turn off experimental deserialization features, bind control planes to localhost, and track advisories weekly while the fixed-version metadata is still catching up.

llama.cpp (selected)

Severity

Affected

Fixed

Notes

CVE-2026-27940 / GHSA-3p4r-fq3f-q74v

High

<= b8145

>= b8146

GGUF mem_size integer overflow (bypass of CVE-2025-53630 fix)

CVE-2025-53630 / GHSA-vgg9-87g3-85w8

High

per advisory

commit/fix path in advisory

Earlier GGUF parser overflow family

GHSA-8wwf-w4qm-gpqr

High

before patch

>= b5662

Malicious GGUF vocabulary buffer overflow

GHSA-j8rj-fmpv-wcxw

Critical

<= b7991 (RPC)

(see advisory)

Unauthenticated RCE in RPC backend; project SECURITY.md tells people not to use RPC

Ollama: GitHub security advisories list for ollama/ollama was empty in this research pass. Still apply OS and binary updates; supply-chain and model-file trust remain your problem.

Shared controls that matter more than logo choice: mutual TLS or gateway auth in front of /v1, request size limits, no public anonymous admin endpoints, trusted model registries only, and GPU node IAM so a compromised serving pod is not a cloud admin key.

Decision guide

Situation

Prefer

Avoid thinking

Multi-user GPU API, OpenAI SDK clients, NVIDIA fleet

vLLM or SGLang

Ollama as the cluster server

Heavy shared system prompts / agents

SGLang strongly; vLLM with prefix caching

Per-request full prefill forever

CPU-only pilot, Apple Mac mini, edge appliance

llama.cpp (or Ollama for UX)

“We failed because vLLM was slow on CPU”

Need GGUF from Hugging Face today

llama.cpp

Assuming GGUF equals best GPU throughput

Strict Apache 2.0 only

vLLM or SGLang

Forgetting MIT is also fine for many orgs (llama.cpp)

Security team asks for recent CVEs

Pin latest stable; read tables above

Assuming open-source serving is “safe by default”

Team is two developers on laptops

Ollama or llama.cpp

Standing up SGLang disaggregation first

For the invented Kochi NBFC SaaS: use llama.cpp for the offline evaluation kit on CPU; pick vLLM or SGLang for the CUDA service behind the API gateway once traffic is real; keep Ollama as a developer convenience, not as the production SLA path.

Checklist

Write down GPU vs CPU/Apple vs hybrid before choosing a server.
Pin exact versions (vLLM / SGLang / llama.cpp tag) in deployment manifests.
Verify LICENSE at that tag (Apache 2.0 vs MIT) against procurement policy.
List the OpenAI routes your apps call; contract-test them on a staging server.
Decide quantisation format per model and per engine; record conversion steps.
Enable metrics (/metrics) and scrape them into Prometheus from day one.
Put auth, TLS and body limits on a reverse proxy; do not rely on default open CORS.
Load models only from registries you trust; treat GGUF/HF repos as code.
For multi-GPU, write down TP size, memory headroom and failure domains before go-live.
Re-read advisories on upgrade day; confirm CVE fixed versions, not only changelog emoji.

Common mistakes

Treating vLLM, SGLang and llama.cpp as interchangeable benchmarks without matching hardware.
Publishing a raw serving port to the internet because “it speaks OpenAI”.
Enabling SGLang custom logit processors or other dill/pickle features without reading CVE-2026-7304.
Using llama.cpp RPC in production despite SECURITY.md guidance.
Comparing tokens/second from a project blog as if it were your workload (if you quote numbers, label them as that project’s run and link them; this post deliberately skips unsourced perf tables).
Forgetting prefix caching on multi-turn agents, then buying more GPUs.
Running Ollama in Kubernetes and expecting vLLM-like continuous batching behaviour.
Ignoring multimodal/video code paths in vLLM after CVE-2026-22778.
Pinning an old vLLM (pre-0.14.1 / pre-0.11.1 / pre-0.10.1.1) because “it works”.
Skipping /metrics and debugging latency only from client anecdotes.

A drill I ran on 14 October 2026

Environment: shared Linux VM (x86_64), about 15 GiB RAM, no NVIDIA GPU, no Kubernetes cluster. Port 8766 left alone for the existing static file server. Artefacts under /workspace/revisit/drill37/.

What I did:

Cloned ggml-org/llama.cpp at tag v0.6.0 and configured a CPU Release build with CMake (GGML_NATIVE=OFF). The build produced build/bin/llama-server and llama-cli.
Downloaded the official CPU binary archive llama-b11598-bin-ubuntu-x64.tar.gz from the b11598 GitHub release (published 14 October 2026, 2:06 AM IST) and used that llama-server for the live HTTP drill.
Downloaded a tiny GGUF: SmolLM2-135M-Instruct-Q2_K.gguf from bartowski/SmolLM2-135M-Instruct-GGUF on Hugging Face (about 85 MiB on disk).
Started llama-server on 127.0.0.1:18080 with -c 512 and --metrics.
Confirmed GET /health returned HTTP 200 {"status":"ok"}.
Posted POST /v1/chat/completions with a one-line user prompt; HTTP 200 with an assistant message; response JSON recorded system_fingerprint beginning b11598- and usage token counts.
Fetched GET /metrics and saved Prometheus text including llamacpp:prompt_tokens_total and llamacpp:tokens_predicted_total.

What I did not run: no vLLM install, no SGLang install, no Ollama install, no CUDA/ROCm, no multi-GPU, no Kubernetes manifests applied, no load test. Those need accelerators or a larger dedicated host than this shared box gave me today.

Finding that is still useful: on CPU, llama.cpp’s OpenAI route and metrics endpoint work with a small GGUF in minutes when you use the ubuntu-x64 release binary. That validates the portable engine path. It does not validate vLLM or SGLang throughput claims, and I am not inventing any.

Unlearn

Unlearn the idea that “LLM serving” is one product category. GPU high-throughput servers and portable GGUF engines differ.
Unlearn treating Ollama as the same decision as vLLM or SGLang.
Unlearn that OpenAI compatibility means feature parity or security parity.
Unlearn loading random GGUF or Hugging Face repos as if they were inert data.
Unlearn that metrics are optional once GPT-style demos look good on stage.

Re-learn

Re-learn PagedAttention (vLLM) and RadixAttention (SGLang) as different KV strategies with different strengths.
Re-learn GGUF as a distribution and quantisation format that llama.cpp centres on.
Re-learn version pins from GitHub releases, with IST timestamps recorded in your runbook.
Re-learn CVE hygiene: Mooncake, header DoS, video paths, chat_template/Jinja, dill/pickle, GGUF parsers.
Re-learn that a reverse proxy, auth and trusted weights are part of the serving architecture.

Closing

Self-hosted LLM inference in 2026 is still a wording trap. vLLM and SGLang are the open-source GPU-first servers you put behind an internal OpenAI-compatible API when concurrency and prefix reuse matter. llama.cpp is the portable engine that makes CPU, Apple and GGUF mobilisation realistic, with Ollama as a wrapper for people who want a smoother local UX. The Kochi team’s whiteboard argument only works after they separate those jobs. Learn what each project optimises. Unlearn three-way bake-offs on mismatched hardware. Re-learn licence and CVE pins on the tag you actually deploy. Practise with a tiny GGUF and /metrics on a safe loopback port before you buy another GPU node. Apply the stack that matches the bottleneck you measured, not the logo that won the last conference talk.

Sources

vLLM

SGLang

llama.cpp and Ollama

Internal

comments powered by Disqus

Releted Posts

Kubernetes autoscaling in 2026: revisit Karpenter, Cluster Autoscaler and KEDA before you scale another node

Manoj Shukla
October 13, 2026

Picture a mid-size ecommerce SaaS in Ahmedabad with about 2,200 active sellers (an invented example, not a real client). Its platform team of eight runs checkout, catalogue, search and about 55 other services on Kubernetes on a public cloud.

Manoj Shukla
October 13, 2026

Picture a GST invoicing SaaS in Pune with about 1,800 business customers (an invented example, not a real client). Its platform team of six runs about 40 services, mostly Go with a few Java and Python ones, on a 12-node Kubernetes cluster.

Manoj Shukla
October 13, 2026

Picture a two-wheeler loan company in Coimbatore with about ninety branches (an invented example, not a real client). Its data team of five runs about 140 cron jobs on two Linux VMs: shell scripts that pull collections files from a payment gateway SFTP, Python scripts that clean them, and SQL that loads a PostgreSQL reporting database every night.