vectara/leaderboard_results
Viewer
•
Updated
•
103k
•
412
•
2
Vectara
company
AI & ML interests
retrieval augmented generation, grounded generation, large language models, LLMs, question answering, chatbot
Recent Activity
vectara's activity
stsui96
updated
a
dataset
about 17 hours ago
stsui96
updated
a
Space
1 day ago
clefourrier
posted
an
update
2 days ago
Post
1536
Gemma3 family is out! Reading the tech report, and this section was really interesting to me from a methods/scientific fairness pov.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
Instead of doing over-hyped comparisons, they clearly state that **results are reported in a setup which is advantageous to their models**.
(Which everybody does, but people usually don't say)
For a tech report, it makes a lot of sense to report model performance when used optimally!
On leaderboards on the other hand, comparison will be apples to apples, but in a potentially unoptimal way for a given model family (like some user interact sub-optimally with models)
Also contains a cool section (6) on training data memorization rate too! Important to see if your model will output the training data it has seen as such: always an issue for privacy/copyright/... but also very much for evaluation!
Because if your model knows its evals by heart, you're not testing for generalization.
joao-vectara
updated
a
Space
3 days ago
joao-vectara
published
a
Space
8 days ago
ofermend
published
a
Space
9 days ago
joao-vectara
published
a
Space
12 days ago
nthakur
authored
a
paper
18 days ago
clefourrier
authored
a
paper
about 1 month ago
clefourrier
authored
a
paper
3 months ago
nthakur
authored
a
paper
4 months ago
nthakur
authored
a
paper
7 months ago
Post
1830
I'm excited to share our updated hallucination evaluation model (called HHEM-2.1-Open) as well as the updated leaderboard that ranks LLM by the propensity to hallucinate.
vectara/Hallucination-evaluation-leaderboard
vectara/Hallucination-evaluation-leaderboard
- 1 reply
clefourrier
authored
2
papers
9 months ago
Post
1756
If you are a debate fan or did this as an extracurricular activity as a kid, you might have fun with this demo - debate bot. Debate against AI/RAG:
vectara/debate-bot
vectara/debate-bot
-
- 4 replies
Post
3405
🦢 The SWIM-IR dataset contains 29 million text-retrieval training pairs across 27 diverse languages. It is one of the largest synthetic multilingual datasets generated using PaLM 2 on Wikipedia! 🔥🔥
SWIM-IR dataset contains three subsets :
- Cross-lingual:
- Monolingual:
- Indic Cross-lingual:
Check it out:
https://huggingface.co/collections/nthakur/swim-ir-dataset-662ddaecfc20896bf14dd9b7
SWIM-IR dataset contains three subsets :
- Cross-lingual:
nthakur/swim-ir-cross-lingual- Monolingual:
nthakur/swim-ir-monolingual- Indic Cross-lingual:
nthakur/indic-swim-ir-cross-lingualCheck it out:
https://huggingface.co/collections/nthakur/swim-ir-dataset-662ddaecfc20896bf14dd9b7
clefourrier
posted
an
update
11 months ago
Post
6119
In a basic chatbots, errors are annoyances. In medical LLMs, errors can have life-threatening consequences 🩸
It's therefore vital to benchmark/follow advances in medical LLMs before even thinking about deployment.
This is why a small research team introduced a medical LLM leaderboard, to get reproducible and comparable results between LLMs, and allow everyone to follow advances in the field.
openlifescienceai/open_medical_llm_leaderboard
Congrats to @aaditya and @pminervini !
Learn more in the blog: https://huggingface.co/blog/leaderboard-medicalllm
It's therefore vital to benchmark/follow advances in medical LLMs before even thinking about deployment.
This is why a small research team introduced a medical LLM leaderboard, to get reproducible and comparable results between LLMs, and allow everyone to follow advances in the field.
openlifescienceai/open_medical_llm_leaderboard
Congrats to @aaditya and @pminervini !
Learn more in the blog: https://huggingface.co/blog/leaderboard-medicalllm
clefourrier
posted
an
update
11 months ago
Post
4764
Contamination free code evaluations with LiveCodeBench! 🖥️
LiveCodeBench is a new leaderboard, which contains:
- complete code evaluations (on code generation, self repair, code execution, tests)
- my favorite feature: problem selection by publication date 📅
This feature means that you can get model scores averaged only on new problems out of the training data. This means... contamination free code evals! 🚀
Check it out!
Blog: https://huggingface.co/blog/leaderboard-livecodebench
Leaderboard: livecodebench/leaderboard
Congrats to @StringChaos @minimario @xu3kev @kingh0730 and @FanjiaYan for the super cool leaderboard!
LiveCodeBench is a new leaderboard, which contains:
- complete code evaluations (on code generation, self repair, code execution, tests)
- my favorite feature: problem selection by publication date 📅
This feature means that you can get model scores averaged only on new problems out of the training data. This means... contamination free code evals! 🚀
Check it out!
Blog: https://huggingface.co/blog/leaderboard-livecodebench
Leaderboard: livecodebench/leaderboard
Congrats to @StringChaos @minimario @xu3kev @kingh0730 and @FanjiaYan for the super cool leaderboard!
clefourrier
posted
an
update
11 months ago
Post
2236
🆕 Evaluate your RL agents - who's best at Atari?🏆
The new RL leaderboard evaluates agents in 87 possible environments (from Atari 🎮 to motion control simulations🚶and more)!
When you submit your model, it's run and evaluated in real time - and the leaderboard displays small videos of the best model's run, which is super fun to watch! ✨
Kudos to @qgallouedec for creating and maintaining the leaderboard!
Let's find out which agent is the best at games! 🚀
open-rl-leaderboard/leaderboard
The new RL leaderboard evaluates agents in 87 possible environments (from Atari 🎮 to motion control simulations🚶and more)!
When you submit your model, it's run and evaluated in real time - and the leaderboard displays small videos of the best model's run, which is super fun to watch! ✨
Kudos to @qgallouedec for creating and maintaining the leaderboard!
Let's find out which agent is the best at games! 🚀
open-rl-leaderboard/leaderboard