spark-servitor / README.md
wxsys's picture
docs: note deterministic crc32 tokenizer fix in Phase 0
64a45a3 verified
|
Raw History Blame Contribute Delete
12.5 kB
---
language:
- en
- id
license: apache-2.0
library_name: transformers
tags:
- code
- security
- code-review
- static-analysis
- vulnerability-detection
- sarif
- sliding-window-attention
- spark
- cpp
- avx2
- awq
- experimental
base_model: XHToken/Spark-X2.5-1.7B
pipeline_tag: text-classification
inference: false
---
# ⚑ Spark-Servitor
<p align="center">
<strong>Experimental rule-assisted code diff gate with an unvalidated 1.7B neural head &amp; OASIS SARIF v2.1.0 output</strong>
</p>
<p align="center">
<a href="https://huggingface.co/wxsys/spark-servitor"><img src="https://img.shields.io/badge/πŸ€—%20HuggingFace-wxsys%2Fspark--servitor-ffd21e.svg" alt="Hugging Face Model"></a>
<img src="https://img.shields.io/badge/License-Apache_2.0-red.svg" alt="License">
<img src="https://img.shields.io/badge/Status-Experimental-orange.svg" alt="Status: experimental">
<img src="https://img.shields.io/badge/Held--out_accuracy-~50%25_(chance)-critical.svg" alt="Held-out accuracy">
</p>
---
> [!WARNING]
> **Status: experimental. Do not use as a security gate.**
>
> - On an independent held-out benchmark (PrimeVul paired CVE fixes, 433 pairs) the shipped pipeline scores **50.0% accuracy** with **1.8% recall** on vulnerability-introducing diffs. That is chance level.
> - Earlier versions of this card reported "93.5% Real-World PR Accuracy" and "100% Critical Vulnerability Recall". Those numbers were **withdrawn**: 93.5% is the *training-set* accuracy of the last training epoch (see [What went wrong](#known-issues--what-was-wrong-with-earlier-claims)), and no run produced the 100% recall figure.
> - Most verdicts produced by the `--weights` / native path come from **string-matching rules and a taint extractor**, not from neural inference.
> - Full evaluation details are in [Evaluation](#evaluation).
---
## Overview
Spark-Servitor is a code diff screening tool. It canonicalizes a unified diff, runs it through rule-based and taint-flow checks, and (optionally) a 1.7B-parameter neural verdict/risk/pathology head, then emits a verdict (`APPROVE` / `QUARANTINE` / `REJECT`), a risk score, pathology labels and an OASIS SARIF v2.1.0 report with exploit flows.
It is a sibling of [`wxsys/qwen-servitor`](https://huggingface.co/wxsys/qwen-servitor).
What works today:
- Diff canonicalization, multi-file parsing, SARIF v2.1.0 emission, CLI / daemon / FFI plumbing. The repo test suite passes (149 passed, locally re-run).
- A rule/taint layer that catches classic web/shell patterns (e.g. `os.system(user_input)`, string-concatenated SQL, `eval` on request data, double-lock deadlocks) and passes the safe forms of those patterns.
What does **not** work yet:
- Detection of vulnerabilities outside those patterns (memory safety, auth/logic flaws, IDOR, hardcoded secrets, `verify=False`, XSS sinks, …).
- A neural head that adds measurable value over the rules (see [Evaluation](#evaluation)).
---
## Architecture (as implemented)
```text
[ unified diff ]
β”‚
β–Ό
DiffCanonicalizer (parse, tokenize)
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β–Ό β–Ό β–Ό
Rule / taint Native C++ engine Neural cortex (PyTorch)
extractor (string-match heuristics) SparkServitorModel
(Python) used by --weights / 28-layer SWA/full-attn backbone
daemon / FFI path + DiffCAMQP cortex head
β”‚ β”‚ β”‚ (not used by the native path)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β–Ό
Hierarchical veto + (QUARANTINE) Z3 SMT gate
β–Ό
verdict Β· risk Β· pathologies Β· SARIF
```
Important implementation facts:
- The native C++ engine (`src/spark_simd_engine.cpp`) decides verdicts with substring checks such as `system(`, `shlex.quote`, `Lock()`/`Unlock()`. Weights are loaded into memory, but the verdict path does not run a forward pass over them.
- In `spark_servitor/engine.py`, an unmitigated taint flow forces `REJECT` with risk β‰₯ 0.95, overriding the neural output.
- The CLI defaults to `--mock-mode` (rule-based emulation) unless `--weights` is given.
- The PyTorch `SparkServitorModel` neural path exists and can be run by dequantizing a `.servitor` file; it was evaluated separately (below).
---
## Specifications
| Parameter | Value |
|---|---|
| **Architecture** | Spark-X2.5-inspired decoder: d_model 2048, 28 layers (21 SWA window 512 + 7 full attention), 16 Q heads Γ— head_dim 128 / 2 KV heads, SiLU MLP, vocab 151,936, headwise sigmoid output gate. The upstream `XHToken/Spark-X2.5-1.7B` config differs (8 heads Γ— head_dim 256, GELU, vocab 131,072, partial rotary on full-attention layers), so its checkpoint is **not drop-in compatible** with this implementation. |
| **Heads** | 3-way verdict, continuous risk, 10-label pathology |
| **Weights provenance** | Per `scripts/train_spark_l4.py` and `models/training.log`: model is instantiated from a fresh `SparkConfig` (no pretrained backbone load found in the script), then DoRA (r=16, ~1% of parameters) is trained for 3 epochs on ~250 samples. The repo does **not** demonstrate that Spark-X2.5 pretrained weights were used. |
| **Training data** | ~250 samples (62 steps Γ— batch 4 per epoch), generated by `data/generate_dataset.py` (synthetic templates). No separate validation split in the log. |
| **Formats** | Outlier-preserving AWQ INT4 / W8A8 packed `.servitor` files |
| **Context window** | Architecture supports a long circular SWA buffer; long-context quality was **not** evaluated |
> The previously listed figures "4.08 MB resident memory" and "0.044 ms per diff" describe the C++ rule path only, were not independently re-measured, and are not neural-inference numbers. Loading the weights alone takes ~1.3–1.6 GB on disk.
---
## Model Files
| Variant | Path in Repository | Size on Disk | Notes |
|---|---|---|---|
| Slim | `slim/spark-servitor-slim.servitor` | ~1.3 GB (1,312 MB) | INT8-quantized embeddings |
| Baseline | `int4/spark-servitor-awq_int4.servitor` | ~1.6 GB (1,608 MB) | FP16 embeddings. (An earlier target of ~720 MB was not met.) |
The repository does **not** contain `model.safetensors` or `cortex_head.pt`. Older docs that mention them are wrong.
---
## Evaluation
All numbers below were produced on 2026-10-16 on a CPU-only machine with data **not** used for training.
### 1. PrimeVul paired test (real CVEs, C/C++) β€” shipped pipeline
Source: [`colin/PrimeVul`](https://huggingface.co/datasets/colin/PrimeVul), `paired` config, `test` split. 433 usable (vulnerable, fixed) function pairs. For each pair two diffs were built with `difflib`:
- fixed β†’ vulnerable ("a PR that introduces the bug", label 1)
- vulnerable β†’ fixed ("a PR that fixes the bug", label 0)
| Configuration | Recall on bug-introducing diffs | False-positive rate on fix diffs | Accuracy |
|---|---|---|---|
| `SparkServitorEngine` + `--weights` (final verdict) | 1.8% | 1.8% | 50.0% |
| Default CLI (mock / rule-based) | 11.1% | 6.5% | 52.3% |
| Raw native engine output | 1.8% | 1.8% | 50.0% |
97.5% of pairs were approved on both sides. The rules do not target C/C++ memory-safety bugs. PrimeVul is a hard benchmark; this result does not claim the task is easy, only that this tool does not solve it.
### 2. Hand-written multi-language set (38 diffs) β€” shipped pipeline
8 obvious vulnerabilities, 10 subtle vulnerabilities, 10 "looks dangerous but safe" cases, 10 clean diffs (Python, Go, JS, C). Labels written by the evaluator.
| Configuration | Accuracy | Recall | False-positive rate |
|---|---|---|---|
| Final (with weights) | 0.68 | 0.50 | 0.15 |
| Default CLI (mock) | 0.61 | 0.44 | 0.25 |
| Raw native only | 0.63 | 0.39 | 0.15 |
| Taint rules only | 0.68 | 0.44 | 0.10 |
The final pipeline equals the taint rules alone (0.68). Missed examples: `strcpy` overflow, hardcoded AWS key, `innerHTML = location.hash`, `or True` auth bypass, IDOR, `verify=False`, path traversal on upload. Wrongly rejected: `ast.literal_eval`, a `realpath`-checked file open.
### 3. Neural head run directly (PyTorch, dequantized INT4 weights, fp32, CPU)
> [!NOTE]
> **Tokenizer caveat.** When these `.servitor` weights were trained, `DiffCanonicalizer.tokenize_canonical_text` mapped words to vocabulary ids using Python's process-salted `hash()` instead of the Spark-X2.5 tokenizer. `DiffCanonicalizer` now uses deterministic `zlib.crc32` across processes, but the existing `.servitor` checkpoints remain mismatched with any inference token mapping until retrained in Phase 3.
| Test | Result |
|---|---|
| 38 hand-written diffs, AUC of risk score | 0.59 |
| 38 hand-written diffs, argmax verdict: recall / false-positive rate | 0.67 / 0.40 |
| PrimeVul, 50 random pairs, AUC (bug-introducing vs fix) | 0.67 |
| PrimeVul, 50 random pairs, argmax flags bug-introducing / fix diffs | 94% / 90% |
| PrimeVul, 50 random pairs, risk(intro) > risk(fix) | 80% |
As a gate the neural head flags almost everything. The 80% pairwise figure is not evidence of skill: a trivial "more deleted lines β‡’ riskier" rule reaches ~73% on the same pairs, and 50 pairs gives a wide confidence interval. Verdict probabilities for a plain `strcpy` overflow were near-uniform (0.37 / 0.36 / 0.28).
### Limits of this evaluation
- Small samples (38 hand-written diffs, 50 neural pairs, tokens truncated to 512). Treat results as indicative, not precise.
- PrimeVul is C/C++ only; Python/JS/Go were covered only by the 38 hand-written diffs.
- The hand-written labels were created by the evaluator, not an independent annotator.
- Latency and memory were not re-measured for the neural path.
---
## Known issues &amp; what was wrong with earlier claims
| Earlier claim | Reality |
|---|---|
| "Real-World PR Accuracy 93.5%" | Training-set accuracy of epoch 3 (`models/training.log`: `Acc: 93.50% \| Steps: 62`), on ~250 synthetic samples. Not a held-out measure. |
| "Critical Vulnerability Recall 100%" | No run supports it. Measured recall on real CVE diffs is 1.8% (shipped pipeline). |
| "Neural code decision gate" | Verdicts on the weights/native path come from string-matching rules; taint flows override model output. |
| "Zero-shot" tag | Removed. |
| Spark-X2.5 vocabulary / tokenizer | Checkpoints were trained with a per-process `hash()` tokenizer (not Spark-X2.5 BPE). Code now uses deterministic `crc32` pending Phase 3 real tokenizer integration. |
| INT4 ~720 MB | File is 1,608 MB. |
| `model.safetensors`, `cortex_head.pt` in the repo | Not present. |
| `SparkServitor.summon(...)`, `spark-servitor judge`, `--model` flag (older USAGE) | Not implemented in the code. |
---
## Quickstart (what actually exists)
### CLI
```bash
# Rule-based emulation (default, no weights)
git diff HEAD~1..HEAD | spark-servitor --format tree
# With the packed weights file (still rule/taint driven, see warnings above)
spark-servitor --file patch.diff --weights models/spark-servitor-awq_int4.servitor --format json
```
### Resident daemon
```bash
spark-servitord --socket /run/user/1000/spark-servitor.sock --weights models/spark-servitor-awq_int4.servitor --daemon
spark-servitor --socket /run/user/1000/spark-servitor.sock --file patch.diff
```
### Python
```python
from spark_servitor.engine import SparkServitorEngine
engine = SparkServitorEngine(mock_mode=False, weights_path="models/spark-servitor-awq_int4.servitor")
verdict = engine.evaluate_diff(diff_text)
print(verdict.status, verdict.risk, verdict.pathology)
```
---
## Intended use
- Research and development of diff-screening pipelines, SARIF plumbing, and as a lightweight rule-based pre-filter for a small set of web/shell patterns.
- **Not** suitable for gating merges, replacing SAST tools, or any decision where a missed vulnerability matters.
## Roadmap
A plan to rebuild the neural component properly (pretrained backbone, real training data, held-out evaluation, honest reporting) is tracked in the project's improvement plan.
---
## Related Repositories
- [**wxsys/qwen-servitor**](https://huggingface.co/wxsys/qwen-servitor): Sister model based on Qwen3.5-0.8B Linear Attention (DeltaNet), specialized for sub-15ms local pre-commit hooks.