Steveeeeeeen/mls_eng_10k
Updated
•
46
Steven Zheng
Steveeeeeeen
AI & ML interests
speech & audio
Recent Activity
updated
a dataset
about 19 hours ago
Steveeeeeeen/mls_eng_10k
liked
a Space
about 20 hours ago
sesame/csm-1b
updated
a dataset
1 day ago
Steveeeeeeen/whisper-leaderboard-evals
Organizations
Steveeeeeeen's activity
updated
a
dataset
about 19 hours ago
liked
a
Space
about 20 hours ago
updated
a
dataset
1 day ago
upvoted
2
articles
1 day ago
Article
Open-R1: Update #1
By
and 7 others
•
•
295Article
Open-R1: a fully open reproduction of DeepSeek-R1
•
808
updated
a
model
3 days ago
published
a
model
3 days ago
reacted to
eliebak's
post with 🔥
3 days ago
Post
1353
Google just dropped an exciting technical report for the brand-new Gemma3 model! 🚀 Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!
2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension
3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?
4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
- 1 reply
upvoted
an
article
3 days ago
Article
Open R1: Update #3
By
and 9 others
•
•
216
published
a
dataset
4 days ago
updated
a
dataset
5 days ago
published
a
dataset
5 days ago
upvoted
an
article
5 days ago
Article
Introducing EuroBERT: A High-Performance Multilingual Encoder Model
By
and 3 others
•
•
121
upvoted
an
article
8 days ago
Article
LLM Inference on Edge: A Fun and Easy Guide to run LLMs via React Native on your Phone!
•
32
liked
a
model
10 days ago
Adding Phi-4-Multimodal
1
#32 opened 11 days ago
by
Steveeeeeeen
updated
a
Space
11 days ago
commented on
From Llasa to Llasagna 🍕: Finetuning LLaSA to generates Italian speech and other languages
11 days ago
Hey!
You're on the right track! The output you're seeing is still tokenized with special markers. To convert this into natural language text, you need to use your tokenizer's decode function.
You can find an example here: https://huggingface.co/HKUSTAudio/Llasa-1B
liked
a
Space
12 days ago
Make Phi4MMForCausalLM.forward's num_logits_to_keep actually optional
2
#20 opened 12 days ago
by
phh