RekaAI/reka-flash-3
Updated
•
1.99k
•
232
Elie Bakouch
eliebak
AI & ML interests
Training LLM's @ 🤗
Recent Activity
posted
an
update
2 days ago
Google just dropped an exciting technical report for the brand-new Gemma3 model! 🚀 Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
upvoted
an
article
3 days ago
Open R1: Update #3
liked
a model
3 days ago
RekaAI/reka-flash-3
Organizations
eliebak's activity
liked
a
model
3 days ago
liked
a
dataset
3 days ago
liked
a
model
5 days ago
liked
a
dataset
6 days ago
liked
a
dataset
8 days ago
liked
a
Space
9 days ago
liked
a
Space
12 days ago
liked
a
dataset
13 days ago
liked
a
model
15 days ago
liked
a
dataset
17 days ago
liked
a
Space
18 days ago
liked
2
models
18 days ago
liked
a
dataset
20 days ago
liked
a
Space
22 days ago
liked
a
Space
23 days ago
liked
a
model
24 days ago
liked
a
dataset
27 days ago
liked
a
dataset
30 days ago
liked
a
dataset
about 1 month ago