llama.cpp: The Engine Under Local AI (GitHub, Scanned)

llama.cpp is the engine. Ollama wraps it, LM Studio and Jan embed it, and the GGUF model format it defined is what most quantised models on Hugging Face are published in. It is plain C and C++ with no Python runtime, runs on CPUs as well as Apple, NVIDIA, AMD and Intel GPUs, and is where the tricks that make a 70B model fit in consumer memory tend to appear first.

Using it directly buys you control. The llama cli command downloads a model straight from Hugging Face and chats with it; llama serve exposes the same model through an OpenAI-compatible HTTP API with a small web interface. Every knob the wrappers hide, from context length to which layers go on the GPU, is a flag here, and the project's 400-plus contributors keep it moving with several merges a day.

• Repository: github.com/ggml-org/llama.cpp
• Licence: MIT (MIT License)
• Language: C++. Stars: 128.4K. Forks: 23.3K. Last push: Sep 16, 2026.
• Scan: clean, Sep 16, 2026, commit 83078fe

Who it is for

Developers who want the thinnest possible layer between a model file and their code, people squeezing a model onto hardware the friendlier tools give up on, and anyone curious how local inference actually works.

Getting started

1. Install with Homebrew on macOS or Linux (Windows: winget install llama.cpp; or take a release binary)

brew install llama.cpp

2. Download a model from Hugging Face and chat

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

3. Serve the same model as an OpenAI-compatible API

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Safety scan

We cloned ggml-org/llama.cpp at commit 83078fe on Sep 16, 2026 and ran the checks described on the GitHub Tools page: credential patterns, decode-and-execute code, install-time scripts, committed binaries, risky CI workflows, every host the code talks to, known vulnerabilities in pinned dependencies, and project hygiene. A person read every hit. This is what we found.

• No secrets, no self-decoding code and no bare-IP URLs.
• The three pattern hits are two Dockerfiles installing Rust and uv into build images, and the update command of the new llama app, which reruns the installer from the project's own llama.app domain when the build was installed that way.
• Two committed binaries: an Android Gradle wrapper jar, standard in any Android sample, and a Keynote presentation in the docs with an executable bit it does not need.
• The known advisories are not in the engine. They sit in the web UI's npm build tooling (tools/ui/package-lock.json) and in pinned Python helpers for a Qualcomm script (transformers, urllib3) that most users never run. The C++ core vendors its few dependencies.
• 51 workflows. Two use pull_request_target, the issue labeler and the draft-PR labeler, and neither checks out the pull request. 16 of 60 third-party actions are pinned to a commit. Security policy and contributing guide present; no Dependabot or CodeQL.

What the scanner counted

The raw findings

Every hit the scanner wrote out, with a link to the exact line at the scanned commit. Secrets candidates are redacted.

By the numbers

Lines by language: C++ 450.1K, C/C++ header 88K, C 76.5K, Python 72.9K, TypeScript 54.7K, JavaScript 38.3K.

Questions

What is the difference between llama.cpp and Ollama?

Ollama is built on llama.cpp. llama.cpp is the inference engine and the GGUF format; Ollama adds a model registry, a background service, automatic GPU setup and a simpler command set. If you want convenience, use Ollama. If you want every flag, the newest features first, or no extra service, use llama.cpp directly.

What is a GGUF file?

GGUF is llama.cpp's model file format: a single file holding the weights, usually quantised to 4 or 8 bits, plus the tokenizer and metadata needed to run it. Most open models on Hugging Face have community GGUF conversions, and llama cli can download one by name with the -hf flag.

Does llama.cpp run without a GPU?

Yes. It was written for CPUs first and still runs well on them; small and medium models are usable on a modern laptop processor. A GPU makes it faster, and the build detects Metal, CUDA, ROCm, Vulkan and SYCL back ends.

This post is part of GitHub Tools, where every repository is cloned and scanned before it is written up. The scan is a snapshot of one commit on one day; the repository has moved on since, so check it before you install.