sid1-qwen3-4b: a search agent trained with reinforcement learning

Qwen/Qwen3-4B fine-tuned with reinforcement learning (GRPO) to find the documents that answer a multi-hop question. It does not write an answer. It searches a collection with dense and keyword search, reads, searches again hop by hop, and finally reports a ranked list of document ids. The recipe follows the SID-1 technical report, scaled down to a 4B model, three public datasets and one GPU.

This is an early stop: 175 of a planned 400 training steps (see Training). It is a research artifact, not a finished replication of SID-1.

Model details

Base model Qwen/Qwen3-4B, 4,022,468,096 parameters, all of them trained
Training RL only, no supervised fine-tuning (there is no target text to imitate): the reward is the NDCG of the reported list against the datasets' annotated supporting paragraphs
Format bf16 safetensors, same architecture and tensor names as the base model (loads with AutoModelForCausalLM)
Language English
License Apache 2.0, as the base model
Developed by masterofaudio2077
Demo Agentic Search, Trained with RL: replay recorded runs or run the model live, next to plain vector search
Code The training notebook and the reference agent loop (agent.py, agent_core.py) are in the demo Space

Uses

Direct use. Give it a question and four tools over your own collection: search_dense, search_bm25, read, report. It returns the ids of the documents needed to answer the question. Use it as the retrieval step of a question-answering system, or to study agentic search.

Out-of-scope. It is not a chat model and does not answer questions. It was trained on one tool schema and one system prompt (below) and may not follow different ones. It was never trained on the open web, other languages, or other kinds of collections.

Bias, risks and limitations

  • In-domain evaluation. Training and evaluation questions come from the same three benchmarks (MuSiQue, HotpotQA, 2WikiMultiHopQA) over Wikipedia paragraphs. Train and dev questions do not overlap, but the results do not show how well it works on other question styles or collections.
  • Early stop. Stopped at step 175 of 400. The learning-rate decay phase of the schedule never started. On the 96-question checkpoints the score had flattened at about 0.66 since step 100, so more steps may or may not have helped.
  • It never uses read. During training the read tool was called 0.0 times per trajectory; it relies on the 160-character excerpts that the searches return. Do not expect it to open documents.
  • Lower precision. It reports more documents than the base model (up to 20), so recall and NDCG rise while precision falls (table below).
  • One run, one seed. There are no confidence intervals. The evaluation has 600 questions (200 per dataset), and the 96-question checkpoints during training are noisy.
  • Cost of 4 samples. The best number (4 samples fused) needs four full runs per question.
  • Inherits the biases and failure modes of the Qwen3 base model and of the Wikipedia text it searches.

How to get started

One turn of the agent with transformers. The prompt and tool schema must be exactly these, because the model was trained on them.

import json, re, torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "masterofaudio2077/sid1-qwen3-4b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval()

TOOLS = [
    {"type": "function", "function": {"name": "search_dense",
        "description": "Semantic (embedding) search over the document collection. Returns the top matches as '[doc_id] title: excerpt'.",
        "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "a natural-language query or a short drafted answer passage"}}, "required": ["query"]}}},
    {"type": "function", "function": {"name": "search_bm25",
        "description": "Keyword (BM25) search over the document collection. Best for names, titles and rare terms. Returns the top matches as '[doc_id] title: excerpt'.",
        "parameters": {"type": "object", "properties": {"query": {"type": "string", "description": "keywords"}}, "required": ["query"]}}},
    {"type": "function", "function": {"name": "read",
        "description": "Read the full text of one document.",
        "parameters": {"type": "object", "properties": {"doc_id": {"type": "string"}}, "required": ["doc_id"]}}},
    {"type": "function", "function": {"name": "report",
        "description": "Finish. Report every document needed to answer the question, most relevant first. Must be the only call in its turn.",
        "parameters": {"type": "object", "properties": {"doc_ids": {"type": "array", "items": {"type": "string"}}}, "required": ["doc_ids"]}}},
]

SYSTEM_PROMPT = (
    "You are a retrieval agent. You are given a question. Do not answer it. Find the documents in the collection that are needed to answer it "
    "and report their ids, most relevant first.\n"
    "Search with several different queries in the same turn (semantic and keyword), read documents when an excerpt is not enough, and follow "
    "multi-step questions hop by hop. When you are done, call report once with the ids of all the documents needed. "
    "You have at most 6 turns, and report must be called before they run out."
)

msgs = [{"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": "When did the luxury division of the maker of the Toyopet Master change the RX 350 body style?"}]
prompt = tok.apply_chat_template(msgs, tools=TOOLS, add_generation_prompt=True, enable_thinking=True, tokenize=False)
ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**ids, max_new_tokens=1536, do_sample=True, temperature=0.7, top_k=0, top_p=1.0,
                     eos_token_id=tok.convert_tokens_to_ids("<|im_end|>"))
text = tok.decode(out[0, ids.input_ids.shape[1]:], skip_special_tokens=True)
calls = [json.loads(c) for c in re.findall(r"<tool_call>\s*(.*?)\s*</tool_call>", text.rsplit("</think>", 1)[-1], re.S)]
print(calls)    # a list of {"name": ..., "arguments": {...}}: several searches per turn, or a single report

To run the whole loop (up to 6 turns), run each call against your collection and feed the results back as the model's next input, formatted as [doc_id] title: excerpt lines inside <tool_response> blocks. Stop at report. agent.py and agent_core.py in the demo Space do this, including the parsing and the exact result format.

Sampling. The model was trained and evaluated with temperature only (1.0 in training, 0.7 in evaluation), no top-k and no top-p, and generation stops at <|im_end|>. This repository's generation_config.json is the Qwen3 default (temperature 0.6, top-k 20, top-p 0.95), so pass top_k=0, top_p=1.0 as above.

Training details

Data

Questions from the train splits of three public multi-hop QA sets, with the supporting paragraphs as gold documents:

Dataset Question pool
MuSiQue all 19,938 train questions
HotpotQA (distractor) 50,000 sampled
2WikiMultiHopQA 50,000 sampled

119,938 questions in the pool; 175 steps Γ— 16 questions = 2,800 of them were actually used. The search collection is the pooled, de-duplicated paragraphs of all three sets (train and dev): 578,747 paragraphs with random ids. 200 dev questions per dataset (600 in total) are held out for evaluation; none of them is a training question.

Procedure

  • Algorithm: GRPO in a custom loop (Magistral style, as in SID-1): 16 questions Γ— 8 attempts per step, no KL term, no importance ratio, one update per batch. A group whose attempts all get the same reward gives zero signal and is skipped. The advantage is the reward minus the group mean, divided by the group's total generated tokens (SID-1's length-bias fix).
  • Rollouts: vLLM, tool results appended as tokens (the chat template is never re-applied). At most 6 turns, 1,536 tokens per turn, up to 6 parallel tool calls per turn, context cap 4,608 β†’ 6,144 tokens (from step 100).
  • Tools: search_dense (Qwen3-Embedding-0.6B, top 4), search_bm25 (bm25s, top 4), read (first 1,800 characters), report (at most 20 ids, must be the only call in its turn).
  • Reward: NDCG of the reported list against the gold paragraphs, (0 if it never reported), minus 0.1 for a malformed turn, minus a length penalty that grows linearly from 0 at 3,000 generated tokens to 0.1 at 7,000.
  • Optimizer: Lion with 8-bit momentum and stochastic rounding, peak learning rate 5e-7 on a warmup-stable-decay schedule (40 warmup steps, constant until step 320, then linear decay to 10%), weight decay 0.1, gradient clip 1.0. Linear layers trained with FP8 matmuls (torchao) on bf16 weights; gradient checkpointing.
  • Stopped at step 175 of 400, in the constant phase of the schedule, so the learning rate was still 5e-7.
Hyperparameter Value
Questions Γ— attempts per step 16 Γ— 8
Steps run / planned 175 / 400
Training temperature 1.0
Seed 0 (one run)
Training-time peak GPU memory 32.6 GiB

Training progress

Quick checkpoints on 96 held-out dev questions (32 per dataset, 1 sample, temperature 0.7, so noisy), then the full evaluation:

Step 0 (base) 25 50 75 100 125 150 175 final (600 questions)
NDCG 0.455 0.504 0.618 0.586 0.658 0.660 0.672 0.662 0.684

Evaluation

Setup. 600 held-out dev questions (200 per dataset) searched over the full 578,747-paragraph collection, temperature 0.7, context cap 8,192 tokens, at most 6 turns. "1 sample" is the first attempt; "4 samples fused" combines the four attempts' lists with reciprocal rank fusion and keeps the top 20. Metrics are computed on the reported list of at most 20 ids. The two baselines use the question itself as the query and take the top 10.

System NDCG Recall Precision F1
Qwen3-4B (base), 1 sample 0.455 0.413 0.511 0.436
Qwen3-4B (base), 4 samples fused 0.604 0.598 0.513 0.517
sid1-qwen3-4b, 1 sample 0.684 0.699 0.356 0.461
sid1-qwen3-4b, 4 samples fused 0.729 0.781 0.237 0.355
Vector search only, top 10 0.579 0.602 n/a n/a
BM25 only, top 10 0.508 n/a n/a n/a

Behaviour on the same questions:

Base sid1-qwen3-4b
Well-formed tool calls 94.9% 99.9%
Ends with a report 86.3% 99.9%
Turns per question 2.6 4.1
Tool calls per turn 0.96 1.52
Generated tokens per question 1,437 1,510

The vector-search rows retrieve 10 documents and the agent reports up to 20, so precision is not comparable across those rows; NDCG and recall are the fairer comparison. Per-dataset numbers are in the MLflow run linked below.

Environmental impact

Hardware 1 Γ— NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB)
Wall-clock about 9.0 hours for the whole run, including the base, 25-step and final evaluations
Platform Google Colab
Carbon emitted not measured

Technical specifications

  • Architecture: Qwen3 dense decoder, 36 layers, tied input and output embeddings, 8 key-value heads, head dimension 128.
  • Software: torch 2.14.0, vLLM 0.34.0 (rollouts), torchao 0.18.0 (FP8), transformers 5.22, bm25s 0.3.16, MLflow 3.18.0.
  • Metrics and curves: the run rl-20261120-0500 in the sid1-qwen3-4b-agentic-search experiment on DagsHub MLflow.

Citation

@misc{sid1_qwen3_4b_2026,
  title        = {sid1-qwen3-4b: a search agent trained with reinforcement learning},
  author       = {masterofaudio2077},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/masterofaudio2077/sid1-qwen3-4b}}
}

Method: the SID-1 technical report. Base model: Qwen3.

Model card contact

Open a discussion on this repository.

Downloads last month
-
Safetensors
Model size
4B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for masterofaudio2077/sid1-qwen3-4b

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,304)
this model

Datasets used to train masterofaudio2077/sid1-qwen3-4b

Space using masterofaudio2077/sid1-qwen3-4b 1

Evaluation results

  • NDCG, 1 sample on MuSiQue + HotpotQA + 2WikiMultiHopQA (pooled dev, 600 questions)
    self-reported
    0.684
  • NDCG, 4 samples fused (RRF) on MuSiQue + HotpotQA + 2WikiMultiHopQA (pooled dev, 600 questions)
    self-reported
    0.729
  • Recall, 1 sample on MuSiQue + HotpotQA + 2WikiMultiHopQA (pooled dev, 600 questions)
    self-reported
    0.699