# Distributed LLaMA Inference Engine Built from Scratch (KV Cache, GQA, RoPE)

**URL:** <https://discuss.huggingface.co/t/distributed-llama-inference-engine-built-from-scratch-kv-cache-gqa-rope/172528>\
**Category:** 🤗Transformers\
**Created:** [January 16, 2026, 10:18am UTC](https://discuss.huggingface.co/t/distributed-llama-inference-engine-built-from-scratch-kv-cache-gqa-rope/172528 "2026-01-16T10:18:40Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![llmRinoCool](https://avatars.discourse-cdn.com/v4/letter/l/67e7ee/32.png) [@llmRinoCool](https://discuss.huggingface.co/u/llmRinoCool)\
**Post date:** [January 16, 2026, 10:18am UTC](https://discuss.huggingface.co/t/distributed-llama-inference-engine-built-from-scratch-kv-cache-gqa-rope/172528/1 "2026-01-16T10:18:40Z")

</div>

I built a **from-scratch LLaMA-style inference engine** to deeply understand transformer internals and inference mechanics.

The project implements:

- Token-by-token **autoregressive decoding**

- **KV cache** (per-layer, per-head)

- **Grouped-Query Attention (GQA)** with correct KV expansion

- **RoPE** applied to Q/K

- **RMSNorm** + **SwiGLU MLP**

- **Distributed matrix execution** across multiple nodes (custom cluster backend)

The goal was correctness and clarity rather than performance tuning, and to learn how modern LLM inference actually works under the hood.

Code + README:  
👉 [https://github.com/rinoScremin/Open-Cluster](https://github.com/rinoScremin/Open-Cluster)

I’d love feedback from anyone working on inference engines or distributed LLM systems.