--- language: - en - id license: apache-2.0 library_name: transformers tags: - code - security - code-review - static-analysis - vulnerability-detection - sarif - sliding-window-attention - spark - cpp - avx2 - awq - experimental base_model: XHToken/Spark-X2.5-1.7B pipeline_tag: text-classification inference: false --- # โšก Spark-Servitor

Experimental rule-assisted code diff gate with an unvalidated 1.7B neural head & OASIS SARIF v2.1.0 output

Hugging Face Model License Status: experimental Held-out accuracy

--- > [!WARNING] > **Status: experimental. Do not use as a security gate.** > > - On an independent held-out benchmark (PrimeVul paired CVE fixes, 433 pairs) the shipped pipeline scores **50.0% accuracy** with **1.8% recall** on vulnerability-introducing diffs. That is chance level. > - Earlier versions of this card reported "93.5% Real-World PR Accuracy" and "100% Critical Vulnerability Recall". Those numbers were **withdrawn**: 93.5% is the *training-set* accuracy of the last training epoch (see [What went wrong](#known-issues--what-was-wrong-with-earlier-claims)), and no run produced the 100% recall figure. > - Most verdicts produced by the `--weights` / native path come from **string-matching rules and a taint extractor**, not from neural inference. > - Full evaluation details are in [Evaluation](#evaluation). --- ## Overview Spark-Servitor is a code diff screening tool. It canonicalizes a unified diff, runs it through rule-based and taint-flow checks, and (optionally) a 1.7B-parameter neural verdict/risk/pathology head, then emits a verdict (`APPROVE` / `QUARANTINE` / `REJECT`), a risk score, pathology labels and an OASIS SARIF v2.1.0 report with exploit flows. It is a sibling of [`wxsys/qwen-servitor`](https://huggingface.co/wxsys/qwen-servitor). What works today: - Diff canonicalization, multi-file parsing, SARIF v2.1.0 emission, CLI / daemon / FFI plumbing. The repo test suite passes (149 passed, locally re-run). - A rule/taint layer that catches classic web/shell patterns (e.g. `os.system(user_input)`, string-concatenated SQL, `eval` on request data, double-lock deadlocks) and passes the safe forms of those patterns. What does **not** work yet: - Detection of vulnerabilities outside those patterns (memory safety, auth/logic flaws, IDOR, hardcoded secrets, `verify=False`, XSS sinks, โ€ฆ). - A neural head that adds measurable value over the rules (see [Evaluation](#evaluation)). --- ## Architecture (as implemented) ```text [ unified diff ] โ”‚ โ–ผ DiffCanonicalizer (parse, tokenize) โ”‚ โ”Œโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ผโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ” โ–ผ โ–ผ โ–ผ Rule / taint Native C++ engine Neural cortex (PyTorch) extractor (string-match heuristics) SparkServitorModel (Python) used by --weights / 28-layer SWA/full-attn backbone daemon / FFI path + DiffCAMQP cortex head โ”‚ โ”‚ โ”‚ (not used by the native path) โ””โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ดโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”ฌโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”˜ โ–ผ Hierarchical veto + (QUARANTINE) Z3 SMT gate โ–ผ verdict ยท risk ยท pathologies ยท SARIF ``` Important implementation facts: - The native C++ engine (`src/spark_simd_engine.cpp`) decides verdicts with substring checks such as `system(`, `shlex.quote`, `Lock()`/`Unlock()`. Weights are loaded into memory, but the verdict path does not run a forward pass over them. - In `spark_servitor/engine.py`, an unmitigated taint flow forces `REJECT` with risk โ‰ฅ 0.95, overriding the neural output. - The CLI defaults to `--mock-mode` (rule-based emulation) unless `--weights` is given. - The PyTorch `SparkServitorModel` neural path exists and can be run by dequantizing a `.servitor` file; it was evaluated separately (below). --- ## Specifications | Parameter | Value | |---|---| | **Architecture** | Spark-X2.5-inspired decoder: d_model 2048, 28 layers (21 SWA window 512 + 7 full attention), 16 Q heads ร— head_dim 128 / 2 KV heads, SiLU MLP, vocab 151,936, headwise sigmoid output gate. The upstream `XHToken/Spark-X2.5-1.7B` config differs (8 heads ร— head_dim 256, GELU, vocab 131,072, partial rotary on full-attention layers), so its checkpoint is **not drop-in compatible** with this implementation. | | **Heads** | 3-way verdict, continuous risk, 10-label pathology | | **Weights provenance** | Per `scripts/train_spark_l4.py` and `models/training.log`: model is instantiated from a fresh `SparkConfig` (no pretrained backbone load found in the script), then DoRA (r=16, ~1% of parameters) is trained for 3 epochs on ~250 samples. The repo does **not** demonstrate that Spark-X2.5 pretrained weights were used. | | **Training data** | ~250 samples (62 steps ร— batch 4 per epoch), generated by `data/generate_dataset.py` (synthetic templates). No separate validation split in the log. | | **Formats** | Outlier-preserving AWQ INT4 / W8A8 packed `.servitor` files | | **Context window** | Architecture supports a long circular SWA buffer; long-context quality was **not** evaluated | > The previously listed figures "4.08 MB resident memory" and "0.044 ms per diff" describe the C++ rule path only, were not independently re-measured, and are not neural-inference numbers. Loading the weights alone takes ~1.3โ€“1.6 GB on disk. --- ## Model Files | Variant | Path in Repository | Size on Disk | Notes | |---|---|---|---| | Slim | `slim/spark-servitor-slim.servitor` | ~1.3 GB (1,312 MB) | INT8-quantized embeddings | | Baseline | `int4/spark-servitor-awq_int4.servitor` | ~1.6 GB (1,608 MB) | FP16 embeddings. (An earlier target of ~720 MB was not met.) | The repository does **not** contain `model.safetensors` or `cortex_head.pt`. Older docs that mention them are wrong. --- ## Evaluation All numbers below were produced on 2026-10-16 on a CPU-only machine with data **not** used for training. ### 1. PrimeVul paired test (real CVEs, C/C++) โ€” shipped pipeline Source: [`colin/PrimeVul`](https://huggingface.co/datasets/colin/PrimeVul), `paired` config, `test` split. 433 usable (vulnerable, fixed) function pairs. For each pair two diffs were built with `difflib`: - fixed โ†’ vulnerable ("a PR that introduces the bug", label 1) - vulnerable โ†’ fixed ("a PR that fixes the bug", label 0) | Configuration | Recall on bug-introducing diffs | False-positive rate on fix diffs | Accuracy | |---|---|---|---| | `SparkServitorEngine` + `--weights` (final verdict) | 1.8% | 1.8% | 50.0% | | Default CLI (mock / rule-based) | 11.1% | 6.5% | 52.3% | | Raw native engine output | 1.8% | 1.8% | 50.0% | 97.5% of pairs were approved on both sides. The rules do not target C/C++ memory-safety bugs. PrimeVul is a hard benchmark; this result does not claim the task is easy, only that this tool does not solve it. ### 2. Hand-written multi-language set (38 diffs) โ€” shipped pipeline 8 obvious vulnerabilities, 10 subtle vulnerabilities, 10 "looks dangerous but safe" cases, 10 clean diffs (Python, Go, JS, C). Labels written by the evaluator. | Configuration | Accuracy | Recall | False-positive rate | |---|---|---|---| | Final (with weights) | 0.68 | 0.50 | 0.15 | | Default CLI (mock) | 0.61 | 0.44 | 0.25 | | Raw native only | 0.63 | 0.39 | 0.15 | | Taint rules only | 0.68 | 0.44 | 0.10 | The final pipeline equals the taint rules alone (0.68). Missed examples: `strcpy` overflow, hardcoded AWS key, `innerHTML = location.hash`, `or True` auth bypass, IDOR, `verify=False`, path traversal on upload. Wrongly rejected: `ast.literal_eval`, a `realpath`-checked file open. ### 3. Neural head run directly (PyTorch, dequantized INT4 weights, fp32, CPU) > [!NOTE] > **Tokenizer caveat.** When these `.servitor` weights were trained, `DiffCanonicalizer.tokenize_canonical_text` mapped words to vocabulary ids using Python's process-salted `hash()` instead of the Spark-X2.5 tokenizer. `DiffCanonicalizer` now uses deterministic `zlib.crc32` across processes, but the existing `.servitor` checkpoints remain mismatched with any inference token mapping until retrained in Phase 3. | Test | Result | |---|---| | 38 hand-written diffs, AUC of risk score | 0.59 | | 38 hand-written diffs, argmax verdict: recall / false-positive rate | 0.67 / 0.40 | | PrimeVul, 50 random pairs, AUC (bug-introducing vs fix) | 0.67 | | PrimeVul, 50 random pairs, argmax flags bug-introducing / fix diffs | 94% / 90% | | PrimeVul, 50 random pairs, risk(intro) > risk(fix) | 80% | As a gate the neural head flags almost everything. The 80% pairwise figure is not evidence of skill: a trivial "more deleted lines โ‡’ riskier" rule reaches ~73% on the same pairs, and 50 pairs gives a wide confidence interval. Verdict probabilities for a plain `strcpy` overflow were near-uniform (0.37 / 0.36 / 0.28). ### Limits of this evaluation - Small samples (38 hand-written diffs, 50 neural pairs, tokens truncated to 512). Treat results as indicative, not precise. - PrimeVul is C/C++ only; Python/JS/Go were covered only by the 38 hand-written diffs. - The hand-written labels were created by the evaluator, not an independent annotator. - Latency and memory were not re-measured for the neural path. --- ## Known issues & what was wrong with earlier claims | Earlier claim | Reality | |---|---| | "Real-World PR Accuracy 93.5%" | Training-set accuracy of epoch 3 (`models/training.log`: `Acc: 93.50% \| Steps: 62`), on ~250 synthetic samples. Not a held-out measure. | | "Critical Vulnerability Recall 100%" | No run supports it. Measured recall on real CVE diffs is 1.8% (shipped pipeline). | | "Neural code decision gate" | Verdicts on the weights/native path come from string-matching rules; taint flows override model output. | | "Zero-shot" tag | Removed. | | Spark-X2.5 vocabulary / tokenizer | Checkpoints were trained with a per-process `hash()` tokenizer (not Spark-X2.5 BPE). Code now uses deterministic `crc32` pending Phase 3 real tokenizer integration. | | INT4 ~720 MB | File is 1,608 MB. | | `model.safetensors`, `cortex_head.pt` in the repo | Not present. | | `SparkServitor.summon(...)`, `spark-servitor judge`, `--model` flag (older USAGE) | Not implemented in the code. | --- ## Quickstart (what actually exists) ### CLI ```bash # Rule-based emulation (default, no weights) git diff HEAD~1..HEAD | spark-servitor --format tree # With the packed weights file (still rule/taint driven, see warnings above) spark-servitor --file patch.diff --weights models/spark-servitor-awq_int4.servitor --format json ``` ### Resident daemon ```bash spark-servitord --socket /run/user/1000/spark-servitor.sock --weights models/spark-servitor-awq_int4.servitor --daemon spark-servitor --socket /run/user/1000/spark-servitor.sock --file patch.diff ``` ### Python ```python from spark_servitor.engine import SparkServitorEngine engine = SparkServitorEngine(mock_mode=False, weights_path="models/spark-servitor-awq_int4.servitor") verdict = engine.evaluate_diff(diff_text) print(verdict.status, verdict.risk, verdict.pathology) ``` --- ## Intended use - Research and development of diff-screening pipelines, SARIF plumbing, and as a lightweight rule-based pre-filter for a small set of web/shell patterns. - **Not** suitable for gating merges, replacing SAST tools, or any decision where a missed vulnerability matters. ## Roadmap A plan to rebuild the neural component properly (pretrained backbone, real training data, held-out evaluation, honest reporting) is tracked in the project's improvement plan. --- ## Related Repositories - [**wxsys/qwen-servitor**](https://huggingface.co/wxsys/qwen-servitor): Sister model based on Qwen3.5-0.8B Linear Attention (DeltaNet), specialized for sub-15ms local pre-commit hooks.