🧠 I built a novel triple-hybrid LLM (Mamba + Attention + 32-expert MoE) from scratch for ~$50 — Titan v1 complete, Titan v2 first cycle done, expanding dataset now

[Mati83moni]

Hi HuggingFace community

I’m Mateusz — independent ML researcher from UK, operating under Project Inkblot. By day I work as a Manufacturing Controller, this project is something I’ve been building in my spare time, driven purely by passion and curiosity. Today I want to share something I’ve been working on for months: a full pre-trained language model designed, implemented, and trained entirely from scratch, combining Mamba SSM, Multi-Head Attention, and fine-grained 32-expert MoE in a single decoder-only architecture — to my knowledge the first to do so under 1B parameters.

---

What is HybridMoE Titan v1?

A 450M parameter decoder-only language model that unifies three fundamentally different sequence processing paradigms in a single architecture:

Component	Role
Mamba SSM (12 layers)	Linear-time local context, O(T) compute
Multi-Head Attention + RoPE (4 layers)	Global context capture
Fine-Grained MoE	32 routed + 2 shared experts, top-2 routing

This is not a wrapper, fine-tune, or modification of any existing model. Every component — architecture, training loop, data pipeline, checkpointing — written from scratch in PyTorch.

---

What’s actually novel here

1. Fixed-Shape Expert Dispatch

Every standard MoE uses torch.nonzero() which creates variable-length tensors incompatible with gradient checkpointing. Titan solves this with a fixed-shape dispatch that works with full gradient checkpointing across all 16 layers:

VRAM with gradient checkpointing: 5.84 GB
Without:                         >24 GB

This is the difference between requiring an A100 cluster and training on a single L4 GPU.

2. Loss-Free EMA Load Balancing

Standard MoE (Switch, Mixtral) adds auxiliary balance loss that competes with the LM objective. Titan uses EMA-tracked router logit biases — zero auxiliary loss, zero competing gradients:

Dead experts:     0 / 32  (entire training run)
usage_std:        0.019 – 0.031  (ideal uniform = 0.031)
Max expert load:  14.7%

Textbook routing health. No dead experts, no collapse.

3. Jamba-Style 1:3 Layer Interleaving

Layer  0:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
Layer  1:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  2:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  3:  [Mamba SSM]      + [SwiGLU Dense FFN]
Layer  4:  [Attention+RoPE] + [MoE: 32 routed + 2 shared]
...

Mamba handles local patterns in O(T). Attention captures global context every 4 layers. MoE provides specialized capacity exactly where global reasoning happens.

4. Single-Line Scalability

The entire model is parameterized through one config. Scaling from 57M to 1.3B+ parameters requires changing one integer:

config = HybridModelConfig(num_layers=8)   # Titan Tiny    57.7M
config = HybridModelConfig(num_layers=16)  # Titan v1     450.4M
config = HybridModelConfig(num_layers=24)  # Titan Mid    690.0M
config = HybridModelConfig(num_layers=48)  # Titan Large  ~1.3B

The 1:3 interleaving, expert counts, RoPE — everything propagates automatically.

---

Training Results (Titan v1)

Config	Value
Hardware	Single NVIDIA L4 24GB
Instance	AWS g6.2xlarge
Precision	bfloat16
Effective batch	64 sequences (~65K tokens)
Peak throughput	~6,200 tok/s
Total tokens trained	~4.2B
Total cost	~$50

Validation Perplexity Progression

Step	Val Loss	PPL	Notes
12,000	2.38	10.81	50-shard val set
20,000	3.47	32.31	Expanded to 200 shards (Polish added)
26,000	3.38	29.26	Monotonic improvement resumes
42,850	3.30	~27.5	Final checkpoint

The PPL increase at step 20K is from validation set expansion (50→200 shards, adding Polish text) — not model degradation. Within the 200-shard regime loss decreases monotonically.

---

Titan v2 — First Training Cycle Complete

Titan v2 completed its first full pre-training cycle (3 epochs) with major improvements across every component:

Aspect	v1	v2
Parameters	450.4M	464.5M
Tokenizer	GPT-2 BPE 50K	Custom BPE 64K (PL + EN + bio + code)
Polish support	Fragmented (ą→2 tokens)	Native single-token
Optimizer	AdamW	AdamW(Tested and Stable)
Training	Single GPU	Single GPU + gradient checkpointing
Data pipeline	Runtime loading	Pre-tokenized binary shards
Data quality	Unknown	Full corruption audit — 0 binary artifacts

Titan v2 Pre-Training — Completed Cycle

Start:          Fresh init, custom 64K tokenizer
Epoch 0:        PPL  87.87  ← cold start, new vocabulary
Epoch 1:        PPL  61.37  ← −26.5 points in one epoch
Epoch 2:        PPL  57.50  ← best_val_loss = 4.0523  ✅ FINAL

Total tokens:   ~1.8B  (3 epochs × ~0.6B per epoch)
Total steps:    27,981
Hardware:       AWS g6.2xlarge (L4 24GB) + benchmarking on A100
Duration:       6 days  (Jun 11 → Jun 17 2026)
Cost:           ~$40

Four-domain corpus: 45% FineWeb-Edu (English) + 18% Polish (articles + Wolne Lektury) + 20% biomedical (PubMed/clinical) + 17% Python/code.

Currently: Building an expanded ~4.56B token dataset for the next training cycle, targeting 9B total tokens — Chinchilla-optimal scale for 464M parameters.

---

Links

• Titan v1 — Model + Weights + Paper: Mati83moni/HybridMoE-Titan-v1 · Hugging Face
• Titan v2 — Live training: Mati83moni/HybridMoE-Titan-v2 · Hugging Face
• Demo Space: HybridMoE Titan - a Hugging Face Space by Mati83moni
• DOI: Mati83moni/HybridMoE-Titan-v1 · Hugging Face

The model weights, full architecture code, training scripts, and paper are all public on HuggingFace. Fully open source, MIT license.

---

Looking for arXiv Endorsement

The paper is pending submission to arXiv cs.LG. As a first-time independent submitter I need one endorsement from an established arXiv author in cs.LG or cs.AI.

If you’re endorsed for cs.LG and find this work interesting — please reach out via HuggingFace messages. One endorsement is all I need.

The paper covers: architecture design, fixed-shape dispatch, EMA load balancing, training dynamics, and scaling analysis.

---

Happy to discuss

• Why 1:3 interleaving over 1:1 or 1:7?
• Fixed-shape dispatch implementation details
• Training stability on a single GPU without loss spikes
• The $50 compute budget — what’s possible and what isn’t
• Roadmap to Titan Standard (~1B) and beyond

Built from scratch, one GPU, one researcher. Project Inkblot

[Mati83moni]

So sorry if answer will be to long but honestly was to many surprises on this project and dont know where to start answer hehe Let’s do from question about interacting with it — yes, quite a lot actually! Just ran a fresh inference test tonight across all four domains the model was trained on. Here’s the honest picture: What works already (after just 1.8B tokens, no SFT): Polish diacritics render perfectly — ą ę ć ł ń ó ś ż ź all clean, including Polish quotation marks „ ". This was the whole point of building a custom 64K tokenizer — GPT-2 fragmented Polish characters, Titan v2 doesn’t. Domain awareness is there — medical prompts produce real terminology (RNA-seq, differential gene expression, chromosome references, Fig. 2 citation format). It knows it’s in a scientific paper. Python syntax surface is correct — def, docstrings, imports, for loops — the structure is right even if the logic isn’t. What doesn’t work yet — and this is expected: Semantic loops (“future of the future of the future”) — classic undertrained behaviour, fixable partially with repetition_penalty without retraining. No real algorithmic reasoning in code — Fibonacci with no Fibonacci logic, factorial accepting a string. Pattern matching, not understanding. Polish text has an archaic 19th-century flavour with was bit funny— that’s the Wolne Lektury (classic Polish literature) dataset leaving its fingerprint. What’s next: First SFT attempt already happened on the 1.8B checkpoint — curious results, data collected. But the real milestone is tomorrow: Titan v2 hits Chinchilla optimum (~9.3B total tokens for 464M parameters) completing epochs 3 and 4 on the expanded dataset. That’s the point where the model should have seen enough data to actually justify its parameter count.
After that — a proper SFT run on a Chinchilla-trained base. The hypothesis is that instruction tuning on top of an undertrained model mostly amplifies existing loops, while SFT on a properly trained base should produce something meaningfully different. Tomorrow we find out if that hypothesis holds.
Either way, it’s data. That’s the fun part of doing this from scratch — every stage teaches you something the papers don’t mention

BEST REGARDS

Mati83moni

[Mati83moni]

Quick clarification on the PPL numbers — I think there’s a misread in the post. The PPL ~57.5 for Titan v2 isn’t a final result — it’s a snapshot from Epoch 2 of an ongoing run. Every model starting cold with a brand new 64K vocabulary begins around PPL 100+ (Titan v2 opened at PPL 87.87 at Epoch 0, which is actually quite clean for a cold start). The post was showing the trajectory, not the destination.

Here’s where things actually stand as of today:

Cycle 1 (original dataset, ~1.8B tokens):

Epoch 0: PPL 87.87 ← cold start, new vocabulary

Epoch 1: PPL 61.37 ← −26.5 points in one epoch

Epoch 2: PPL 57.50 ← Cycle 1 complete

Cycle 2 (expanded dataset, 3.78B tokens/epoch):

Step 45,000: val_loss 3.8319 → PPL ~46.1 ← current BEST

Step 47,000: autoresume checkpoint — still training

Still actively converging. Target is ~9.3B total tokens (Chinchilla optimum for 464M params).

On all your technical points — you’re right across the board, and I’m taking each one seriously:

BPB instead of PPL — absolutely, and this is now on my list before the paper submission. The v1 (50K vocab) vs v2 (64K vocab) comparison in raw PPL is misleading and I should have flagged it myself. BPB is the right cross-tokenizer metric and reviewers will ask for it.

Per-domain eval — planned for the next checkpoint report. The 45/18/20/17 mix is a hypothesis I want to validate, not a result I’ve confirmed. Per-domain BPB across checkpoints will tell me whether FineWeb is carrying everything or whether Polish and bio are genuinely learning.

The ablation — this is the one I keep circling back to. Mamba-only vs Attn+MoE-only vs triple-hybrid at matched params and tokens, run at 57M scale. It’s cheap and it’s the figure the architecture claim actually rests on. I’ll run it.

SFT as a controlled pair — same SFT data, same recipe, two bases: 1.8B checkpoint vs the 9.3B one. That’s a clean figure and I’ll run it exactly that way.

On the Chinchilla framing — noted. 9B is a checkpoint, not a ceiling. LLaMA proved empirically that for small models you train well past compute-optimal because inference is cheap and you’re buying capability, not minimizing FLOPs. If the loops persist past 9B, more tokens is the lever.

On the novelty — fair and appreciated. Jamba is right there and I cite it. The defensible claim is the fixed-shape dispatch + loss-free EMA balancing + doing it from scratch under 1B for ~$50. Leading with the engineering contribution rather than the architectural combination is the stronger move for the paper abstract.

​I’ll post a full checkpoint update this weekend—I’m taking some time off from work to focus entirely on BPB numbers and the per-domain breakdown. Would genuinely love your eyes on the Titan v2 model card if you have a moment — it has the full training config, architecture details, and I just updated it with the corrected training progress.

This is the most useful single comment this project has received.

Thank you!!!

Mati83moni

[vl-dev]

Hi Mateusz, this is a really useful build log. The parts that stood out are the repeated training cycles, the autoresume checkpoint at step 47,000, and keeping the work viable on L4/A100-level budget instead of a cluster.

I’m Rahul, founder of VaultLayer. VaultLayer is live now and helps individual builders run training/fine-tuning jobs on reliable, available, and affordable GPUs with monitoring, checkpointing, and recovery around the run.

If you want to compare it against the current AWS/L4 workflow for Titan v2 checkpoints or the 57M ablation runs, happy to share details.

Rahul
Founder, VaultLayer

[ProtopiaUk]

I am not into the technicalities, but I am very interested in the possibilities that arise from smaller models that can be easily trained because IMO this enables the creation of specialised models for specific tasks.

My own interest is agentic coding, and it frustrates me to heck that to get decent results for a specialised task I need to use huge frontier models that know 50 human languages, the whole general knowledge of wikipiedia and capable of writing erotic poetry in iambic pentameter but they can’t write decent code first time.

Software Engineering is possibly a unique use case - it is probably the single most common use case, and it is also the one with the biggest base of: academic knowledge/best practice, practical knowledge/best practice and training data (github etc).

What I would like to see is a small model that has been trained to understand:

• The most common human languages (or possibly just English as a starter since it is the most common language used by software engineers)
• Modern Software Engineering fundamentals & best practices (programming language independent) i.e. UML, what makes good code, design patterns etc.
• Programming languages themselves as Experts
• Millions of examples of good quality code from Github
• Dual free-text / structured responses approach so you can chat but it can also work in a structured way with harnesses specifically designed to work with it.

I know this is probably not the area of interest for OP, but I did want to chip in to see if there was any interest in this.

[ProtopiaUk]

@KnackAU Obviously anything you decide to do for yourself is automatically OK - but this isn’t my vision which is much much much bigger than this.

My vision is for the AI industry to:

1. Stop forcing everyone to use frontier models which are massively expensive to run and deplete the worlds resources whilst causing climate change.
2. Create a new more flexible architecture for AI which enable specialised SLMs easily to be created which can run on consumer hardware to perform specific functions to high quality at low cost.

If I want to make a shelf for my home that requires me to sand it down to a fine finish and drill a few holes in my wall, I don’t have to go and rent time on a massive mobile drilling/grinding/sanding/milling machine that can do everything from drill an oil well to sand planks and everything else. I want to get out my own local sander and drill and do some sanding and make some holes. YES, I both own and use two specialised tools that are designed to do a specific job and do it well.

Obviously if I have a highly specialised requirement that is far from common and no one has created a consumer tool for it, then I might need to hire an expensive more generalised and flexible tool - but for common tasks, I should be able to buy an inexpensive tool that has economies of scale.

So, what I envisage in AI is the ability for people to buy (or download for free) an off the shelf SLM that is specifically designed for coding in English, adapt it for Python or PHP by adding a language module (that contains the AST rules) and set it going (like having a drill with a both a chuck and a sanding attachment).

Equally obviously the AI industry isn’t interested in doing this when all their investment is in monolithic frontier models which are hugely expensive to train - like most consumerism this needs to be driven by smaller specialised groups creating a solution that everyone wants.

However I don’t want a tool thrown together by a neighbour made from bits of wood and metal he had laying around either which doesn’t fit the drill holder or sanding attachment I already own - I want something that is supported and has a variety of attachments etc.

In other words, I want the SLM to be a community effort by a team of like-minded people who get together to produce something of high quality which is supported and has a future.

(Aside: I honestly don’t understand why there is so little team-work in the AI space to create an entire solution that can survive and be enhanced for decades, but instead individuals create a single specialised part for themselves that doesn’t get synergy and eventually disappears as if it never existed. OPEN SOURCE is well established in other areas of IT - so why not in the AI space.)

[ProtopiaUk]

@KnackAU Ray - I probably should have gone and looked at your contributions here to see what you had done, but I failed to do this earlier for which I apologise - however, honestly wasn’t pointing at you when I was commenting generally about a lack of teamworking.

I guess both you and @Mati83moni Mateusz essentially are doing fundamental experimental research - you have synergistic ideas on how to improve some basics and you do experiments to see how it works out. These are the essential building blocks for useable products and they are typically done by individuals. The biggest exception to this I can think of is CERN - but historically Faraday, Curie, Einstein, and thousands of other scientists worked this way, mostly taking other people’s ideas and building upon them by coming up with something new, but occasionally doing something completely new. Products, however, are typically engineered through team efforts - often mostly commercial and driven by a single CEO, but in recent decades also organically through team-based open-source efforts.

But, if you take a look on Reddit there are literally tens (or possibly hundreds) of people who have vibe coded a memory system for AI. Some of these might gain traction, but most will fall by the wayside, because it doesn’t gain many users, or because the author loses interest. In a scientific analogy these are like new life-forms coming into being - some succeeding in surviving but not evolving, and most dying off. Now imagine what the outcome might have been if these same developers had come together to work on a single memory project as a team - we would have synergy where ideas spark off one another, we would have evolution, we would have longevity, we would have a much much much better chance of achieving critical mass / virality. Now imagine extending that concept to an entire AI ecosystem - harnesses, memory, language models, infrastructure pipelines …

I would imagine my own situation is a good example of the consequences - I have a history of making significant contributions to several open source projects over the past 20 years or so - but I have not yet managed to start such a project myself. I have an idea for an open source project (synergising the ideas but not the code) from two other open source projects I have contributed to) but since getting the idea 6+ years ago I haven’t had the time to start coding it, even though my shower-time thinking has taken the ideas forward. More recently I started to think about doing AI Agentic Coding to get it off the ground, but discovered just how fragmented and the agentic coding world and AI world is. I get that it is so new and fast moving, but nevertheless the lack of use of e.g. UML as part of this and the unstructured free-text approach is astounding - not building on the wealth of academic and practical knowledge as a foundation of agentic coding is an IMO amateur vibe approach.

[ProtopiaUk]

My personal interpretation of the lack of teamwork is:

1. American rugged individualism - don’t rely on others when you can do it yourself.
2. Desire for monetisation - also a trait that originated in the USA
3. Selfishness - not just about hoarding for monetisation, this term can apply equally to believing that you are the only person experiencing this problem.
4. Lack of more generalised knowledge - they install e.g. Claude Code and stay focused entirely within the Clause ecosystem, without much (if any) knowledge about how things work. So when they experience a problem and don’t already know of a solution, they assume that there isn’t one.

To a large extent I think that this happening here is simply a specific example of a more general growing societal problem of selfishness, greed, lack of empathy and compassion, short-termism and focus only on the immediate problem without looking at the generalised issue.

P.S. The American culture of rugged individualism supposedly derives from the Wild West - and it probably does but only in the sense of the cowboy movies and not the real-life “wild” west where people worked very much in teams to create communities necessary for survival.

[ProtopiaUk]

@KnackAU

I think one of the biggest things people are finding out now is that using agentic coding and AI research is so fast and so powerfull, that a single human or even groups of humans simply can not keep up at the same pace. The key (in my mind) is to take a step back. Remember that just because you can do things fast doesn’t mean that you should. Sometimes you need to take your time, go back over what you are doing and refocus, because it is very easy to stray from your original path and if you are using agents for everything you will just push them down that path as well (or they will drag you down it).

Hmmm. What we are seeing right now is a huge number of AI adopters pulling back due to the costs or because the quality isn’t there. In other words its a bubble - and given that Anthropic and OpenAI are subsidising stuff to a massive degree, once full pricing kicks in after IPOs a lot of demand is going to evapourate.

I don’t think that this means that AI is dead - far from it - but it may mean that dominance by a few giant IS Corporates is going to fade away.

I think that the other problem people are going to face is one of code quality. There is a well understood architectural issue in human-developed IT whereby you start with a nice clean architecture and then bolt on new functions and bolt further stuff onto these bolt-ons etc. I suspect (without having done any research or collected any evidence - in other words a pure guess based on decades of IT experiences) that even if the code quality is good, the underlying architecture may not be so good. In other words, AI coding may result in large scale technical debt. To prevent this you need not only diligent code reviews, but also senior reviews of the underlying architecture - assuming that A) there is an underlying architecture conjured up by the AI and B) the architecture is sufficiently documented to allow someone to review it. In other words, quality AI coding is for the diligent and NOT for the lazy.

[ProtopiaUk]

Yes - it is more of a human thing, but it is most obvious in the USA though also noticeable in the UK, and in both countries it started getting worse at the same time (Thatcher / Reagan).

But also yes - we cannot fix society or really even influence it - but we might perhaps be able to make a difference in a much smaller area of AI.

[Mati83moni]

Weekend Research Update — SFT Experiments, BPB Metric & New Pretraining BEST

Project Inkblot | Mateusz Piesiak | 26–28 June 2026

A quick personal note before diving in — I genuinely wish I had more time to participate actively in the discussion here. Life has been quite full lately: day job as a Manufacturing Controller, family, kids, the household, and everything else that comes with being a solo researcher doing this entirely on the side. I read every comment carefully and they matter a lot to me — I just do not always get to respond as quickly as I would like. This weekend report is my way of making up for the silence. Thank you for your patience.

Pretraining Progress — New BEST Checkpoint

Since my last post the pretraining has continued on AWS g6.2xlarge (L4 24GB). Here is the full checkpoint history:

Step	Val Loss	PPL	Notes
30,000	4.0508	57.5	Cycle 2 start (expanded dataset)
45,000	3.8319	46.1	Previous BEST
50,000	3.7447	42.3
53,950	3.7447	42.3	Autoresume STOP
55,000	3.6835	39.8
60,000	3.6361	37.9
65,000	3.5968	~36.5	New BEST

The model continues to converge steadily. At step 65,000 we have processed approximately 4.26B tokens (65,000 steps × batch 4 × grad_accum 16 × seq_len 1024). Chinchilla optimum for 464M params is ~9.3B tokens so we are at 46% of the way.

Weekend SFT Experiments

This weekend I ran three SFT experiments. Here is exactly what happened, including the raw outputs.

Experiment 1 — Colab T4, 120 examples, freeze 10/16 layers

I started with the step 50,000 checkpoint (PPL 42.3) and a small dataset of 120 ChatML examples across 5 domains. Froze the embedding and layers 0–9, training only the top 6 layers. Used fp16 with gradient checkpointing on a T4 (15GB VRAM), 7 epochs at LR 3e-6.

The training curve looked promising on paper:

Ep 1/7  val=7.49  PPL=1796
Ep 2/7  val=5.79  PPL=326
Ep 3/7  val=4.57  PPL=96
Ep 4/7  val=3.77  PPL=43
Ep 5/7  val=3.35  PPL=28
Ep 6/7  val=3.15  PPL=23
Ep 7/7  val=3.07  PPL=21.6  ← saved

But the actual generation told a very different story:

Prompt: "Co to jest szlak mTOR i jaką rolę odgrywa w nowotworach?"

Output: 123466678910101011121314151718181819206070909090899090909190909094909090
        CCC C C C G G G H G G G M G G G A G G G I G G G I G G G I G G G I...

PPL 21.6 after 7 epochs was pure memorisation, not learning. The model reproduced numerical patterns (TPM values 1, 2, 3… 100, 200, 500, 1000) from the RNA-seq training examples. 120 examples × 7 epochs = 742 forward passes on the same data. Classic overfitting on a tiny dataset.

Experiment 2 — Colab T4, 120 examples, full unfreeze

Same 120 examples but this time all 16 layers unfrozen, 2 epochs.

Ep 1/2  val=7.49  PPL=1796
Ep 2/2  val=2.68  PPL=14.5  ← saved (interrupted)

Output:

Prompt: "Explain the mechanism of action of tyrosine kinase inhibitors in CML"

Output: /. / W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W
        W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W W...

Catastrophic forgetting in 88 seconds. Training all 464M parameters on 120 examples erased 5.2B tokens of pretraining knowledge. The model replaced everything with a loop on a single token.

Experiment 3 — AWS L4, 4300 examples, structural MoE-aware freeze

Based on KnackAU’s earlier comment and my own architecture inspection, I moved from sequential layer freezing to structural freezing — freezing by component type rather than by layer index:

for name, param in model.named_parameters():
    freeze = (
        'token_emb' in name or
        any(f'layers.{i}.' in name for i in range(4)) or
        'router' in name or
        'routed_experts' in name or
        'shared_experts' in name or
        'shared_gate' in name
    )
    param.requires_grad = not freeze

The key decision here was freezing the MoE routers and experts. Why? Because the Expert Sparsity Analysis (done earlier this weekend) showed that the router learned domain-specific expert allocation with zero supervision:

Medical vs Python   overlap = 0.00  ← perfect domain specialisation
Medical vs Polish   overlap = 0.15  ← clear specialisation
Python  vs English  overlap = 0.11  ← clear specialisation

This emerged purely from pretraining on the mixed 4-domain corpus. SFT should not destroy this.

The split ended up at 278.6M frozen (embedding + first 4 layers + all MoE routing and experts) and 185.9M trainable (Attention QKV, Mamba SSM, Dense/SwiGLU, lm_head) — about 40% of the model.

The dataset was much larger this time: 4,300 ChatML examples total. 3,500 clinical reasoning examples with step-by-step analysis, 800 examples from MedDialog/MedQA/PubMedQA/ChatDoctor/iCliniq, and 50 of my own Polish medical examples covering RNA-seq omics and drug mechanisms.

Training curve:

Val BEFORE  9.92   PPL=20,257
Ep 1/3      train=0.46  val=0.060  PPL=1.1
Ep 2/3      train=0.04  val=0.042  PPL=1.0  ← BEST saved

Val loss 0.04 looked incredible but it was an artefact of data leakage. When building the dataset I mixed train_final.csv and val.csv into one pool and then split 95/5 randomly. The model was being evaluated on examples it had already seen during training. The actual generalisation is unknown.

The output confirmed the problem:

Prompt: "What is the mTOR signaling pathway and its role in cancer?"

Output: Guthrie struct char ******* Tibpectral vagus neutrophonsai Monaster
        sappveston CoQ provinandelion oracle alp powiatowszeostridochastic...

Root cause is the same as Experiments 1 and 2 — base model PPL 39.8 is simply too weak for SFT to produce anything useful. This confirms KnackAU’s point that Chinchilla is a checkpoint, not a ceiling. The SFT pipeline, dataset, and freeze strategy are all correct. The base model just needs more pretraining first.

Evaluation Results (step 50,000, before SFT)

MCQ Log-Likelihood

I ran a proper MCQ eval with full-text scoring and a BPE fix (the previous version had a merge bug where single-word continuations produced cont=0 tokens — fixed by stripping trailing space from prompt and including it with the continuation, which is the standard lm-evaluation-harness approach).

🟢 Easy    (5 questions)  3/5 (60%)  Paris ✅  print() ✅  Warszawa ✅
🟡 Medium  (5 questions)  2/5 (40%)  mTOR ✅  RNA-seq ✅
🔴 Hard    (5 questions)  0/5 (0%)   expected for base model pre-SFT

Tokenizer Compression — Titan 64K vs Mistral-7B vs GPT-2

Polish General    Titan 4.04 c/t  |  Mistral 2.47 (+67%)  |  GPT-2 1.93 (+110%)
Polish Medical    Titan 3.25 c/t  |  Mistral 2.30 (+41%)  |  GPT-2 2.07 (+57%)
English Tech      Titan 5.19 c/t  |  Mistral 4.84 (+7%)   |  GPT-2 5.05 (+3%)
Python Code       Titan 2.52 c/t  |  Mistral 2.77 (-9%)   |  GPT-2 2.22 (+14%)

The 64K Polish-optimised tokenizer gives +67% better compression for Polish text vs Mistral. This directly reduces compute cost for Polish-language inference.

Expert Sparsity — Router Specialisation across 4 MoE layers (L0/L4/L8/L12)

Medical (top-5)  E24:24  E29:23  E16:21  E23:21  E3:21
Python  (top-5)  E9:52   E28:50  E30:48  E27:47  E10:47
Polish  (top-5)  E24:26  E4:25   E30:23  E29:21  E22:21
English (top-5)  E23:18  E3:18   E30:15  E13:14  E29:14

Domain specialisation (IoU of top-5 expert sets)
  Medical  vs Python   overlap = 0.00  ✅ Perfect specialisation
  Medical  vs Polish   overlap = 0.15  ✅ Clear specialisation
  Python   vs English  overlap = 0.11  ✅ Clear specialisation
  Polish   vs English  overlap = 0.25  ✅ Clear specialisation

The router learned domain-specific expert allocation with zero supervision — this emerged purely from pretraining on the mixed 4-domain corpus.

BPB Evaluator — responding to KnackAU’s feedback

As KnackAU correctly pointed out, PPL is tokenizer-dependent and cannot fairly compare Titan v1 (50K vocab, PPL ~27.5) with Titan v2 (64K vocab, PPL ~36.5).

Bits Per Byte (BPB) is tokenizer-invariant:

BPB = total_NLL_nats / (ln(2) × n_bytes_UTF8)
    = log₂(PPL) / avg_bytes_per_token

Implementation:

@torch.no_grad()
def compute_bpb(model, tok, text: str) -> dict:
    n_bytes  = len(text.encode('utf-8'))
    ids      = tok.encode(text, add_special_tokens=False)

    input_ids = torch.tensor([ids]).to(DEVICE)
    out = model(input_ids)
    logits = (out[0] if isinstance(out, tuple) else out)[0]

    log_probs = F.log_softmax(logits[:-1].float(), dim=-1)
    targets   = input_ids[0, 1:]
    nll_per_token = -log_probs[range(len(targets)), targets]

    total_nll = nll_per_token.sum().item()
    bpb = total_nll / (math.log(2) * n_bytes)
    return {"bpb": bpb, "ppl": math.exp(nll_per_token.mean().item())}

The evaluator tests across 5 domains (English General, Polish General, Medical English, Medical Polish, Python Code). BPB results for v1 vs v2 comparison will be published once v2 reaches the target PPL.

Interpretation scale:

< 1.0    excellent (better than human for English)
1.0–1.5  good
1.5–2.0  average
> 2.0    undertrained

SFT Dataset — Ready for Next Run

The dataset for the next SFT attempt is built and waiting:

sft_master_v2.jsonl — 4,300 high-quality ChatML examples (7.6MB)

• 3,500 medical examples from train_final.csv — clinical reasoning with step-by-step analysis, system prompt “You are an expert medical AI assistant”
• 800 examples from val.csv — MedDialog, MedQA, PubMedQA, ChatDoctor, iCliniq, system prompt “You are an empathetic AI health companion”
• 50 of my own Polish medical examples — RNA-seq omics analysis, drug mechanism explanations (PL/EN), patient education in Polish

Validation split is 197 examples in a separate file (sft_val_v2.jsonl) with no mixing with train. Lesson learned.

SFT Architecture for Next Run

For the next SFT run (once pretraining reaches PPL ~30-32) the plan is the same structural freeze that was validated in Experiment 3:

# Structural freeze — preserves MoE domain specialisation
freeze = (
    'token_emb' in name or
    any(f'layers.{i}.' in name for i in range(4)) or
    'router' in name or
    'routed_experts' in name or
    'shared_experts' in name or
    'shared_gate' in name
)
# Frozen    278.6M (embedding + 4 layers + all MoE routing)
# Trainable 185.9M (Attention, Mamba, Dense, lm_head)

Target config is LR 3e-6, 3 epochs, GRAD_ACCUM 4, fp32 on L4.

What’s Next

Current   step 65,000  |  PPL 36.5  |  4.26B tokens
Target    step ~92,000  |  PPL ~33-34  |  ~6B tokens (early stop considered)
          step ~164,490  |  PPL ~30-32  |  9.3B tokens (Chinchilla optimum)

After pretraining reaches the target:

1. BPB eval — v1 vs v2 cross-tokenizer comparison
2. SFT with sft_master_v2.jsonl (4,300 examples, structural freeze)
3. Per-domain eval post-SFT
4. Architecture ablation at 57M scale (Mamba-only vs Attn+MoE vs triple hybrid)

Key Lessons This Weekend

1. SFT requires a competent base. PPL > 35 means garbage output regardless of dataset quality or freeze strategy. This is not a bug — it is physics.

2. MoE expert routing is precious. Freeze routers and experts during SFT. The domain specialisation (Medical vs Python overlap = 0.00) took billions of tokens to learn and can be destroyed in minutes.

3. Data leakage is subtle. Mixing sources before splitting produces optimistically low val loss that means nothing. Val set must be built separately before any train data is assembled.

4. BPB > PPL for cross-model comparison. Larger vocab mechanically inflates PPL. The models are more comparable than the raw PPL numbers suggest.
   *Update comming soon

5. 120 examples is not enough for 464M params. Minimum viable SFT for this model size is ~500-1000 high-quality instruction pairs. The 4,300-example dataset should work.

So yee that happen on weekend guys have nice reading hehe and yee
Any Question Just Message me Happy to Answer For all Question
@ProtopiaUk @KnackAU @vl-dev @John6666

[Mati83moni]

@KnackAU I just realized we’re on the exact same wavelength! I’ve been meaning to circle back to something you mentioned in your earlier post — that idea about taking Gemma MTP heads and finetuning them for specific tasks like Python generation. That really stuck with me because I’ve actually been down a very similar rabbit hole, just from a slightly different angle.

A while back I took Google’s FunctionGemma-270M and fine-tuned it specifically for on-device mobile function calling — things like send_email(), create_contact(), create_calendar_event(), show_map(), the kind of everyday phone actions people do constantly. Used LoRA, trained it on a Colab T4 in about 30 minutes, and managed to push accuracy from ~58% (base model) to nearly 85%. The whole thing quantizes down to 272MB INT8 and runs in 1-3 seconds on a phone with zero cloud calls.

Here’s the repo if you’re curious: Mati83moni/functiongemma-270m-it-mobile-actions · Hugging Face

What caught my eye even more is your Aiden project — a physical device that watches the screen over HDMI and controls the phone via USB HID. That’s genuinely clever because it sidesteps all the jailbreak/root/ADB problems. And it made me think — something like my functiongemma could potentially serve as a lightweight brain for exactly that kind of setup. You have the hardware layer that sees and controls the phone, and this model could be the part that takes a natural language command like “send email to my boss with today’s report” and translates it into the right function call, all running locally on the device itself.

I see a lot of overlap between what we’re both chasing — small specialized models that actually do useful things on real hardware instead of burning $50/day on API calls to frontier models. Your head-grafting platform sounds fascinating too, especially the interceptor concept. Would love to hear more about how that works in practice.

Also completely agree with what you said about engaging with people doing similar things rather than trying to push ideas — that’s exactly the spirit. Different approaches to the same problem space, and I think we’re both finding that these small Gemma-class models are way more capable than people give them credit for when you train them right for a specific job.

Would be great to collaborate on something down the line