Article
Open R1: Update #2
By
and 6 others
β’
β’
203Elie Bakouch
eliebak
AI & ML interests
Training LLM's @ π€
Recent Activity
posted
an
update
2 days ago
Google just dropped an exciting technical report for the brand-new Gemma3 model! π Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
upvoted
an
article
3 days ago
Open R1: Update #3
liked
a model
3 days ago
RekaAI/reka-flash-3
Organizations
eliebak's activity
published
an
article
about 1 month ago
published
an
article
about 2 months ago
Article
Open-R1: a fully open reproduction of DeepSeek-R1
β’
807
published
an
article
about 2 months ago
Article
Diving into MiniMax01 405B MoE
By
β’
β’
17
published
an
article
4 months ago
Article
SmolVLM - small yet mighty Vision Language Model
β’
221
published
an
article
8 months ago
Article
SmolLM - blazingly fast and remarkably powerful
β’
333