README.md

---
license: apache-2.0
library_name: lclm
base_model:
  - Qwen/Qwen3-Embedding-0.6B
  - Qwen/Qwen3-4B-Instruct-2507
tags:
  - latent-context-language-model
  - context-compression
  - long-context
---

# 0.6b-4b LCLM, 8× compression

Latent Context Language Model: an encoder–decoder compressor described in
*End-to-End Context Compression at Scale*.

The text to compress should be wrapped between `<|memory_start|>` and
`<|memory_end|>`.

> Running these checkpoints requires the LCLM codebase:
> <https://github.com/LeonLixyz/LCLM>. Standard `transformers.AutoModel` /
> `vllm.LLM` will not load this format on its own.

## Quick load

```python
from latent_context import LCLM

model = LCLM.from_pretrained("latent-context/0.6b-4b-LCLM-8x")

prompt = (
    "<|memory_start|>"
    "<long document, code, or text to compress>"
    "<|memory_end|> "
    "Summarize the document above."
)
# model.generate(...) — see latent_context/inference/hf.py
```

## vLLM serving (two-stage CLI)

The vLLM path runs the encoder and the decoder in **separate
processes** that hand off via a `.pt` file on disk. Running both in
one process OOMs — vLLM grabs all GPU memory at init, leaving none
for the HF encoder.

```bash
# Step 1: HF encoder over a jsonl of prompts → embeds.pt
python -m inference.vllm_inference.encode     --checkpoint latent-context/0.6b-4b-LCLM-8x     --prompts-jsonl prompts.jsonl     --out embeds.pt

# Step 2: vLLM decoder reads embeds.pt → completions.jsonl
python -m inference.vllm_inference.decode     --checkpoint latent-context/0.6b-4b-LCLM-8x     --embeds-pt embeds.pt     --out completions.jsonl
```

See `inference/examples/README.md` in the codebase for the
`prompts.jsonl` schema and an end-to-end RULER NIAH eval driver.

## Configuration

| field | value |
|---|---|
| encoder | Qwen/Qwen3-Embedding-0.6B |
| decoder | Qwen/Qwen3-4B-Instruct-2507 |
| compression_ratio | 8 |
| encoder_window_size | 1024 |
| pooling | mean |
| encoder_mask_type | causal |
| boundary_overlap | 0 |
| adapter_type | mlp |

Code: <https://github.com/LeonLixyz/LCLM>