Create README.md
Browse files
README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
base_model: AtlaAI/Selene-1-Mini-Llama-3.1-8B
|
| 3 |
+
library_name: transformers
|
| 4 |
+
language:
|
| 5 |
+
- en
|
| 6 |
+
- de
|
| 7 |
+
- fr
|
| 8 |
+
- it
|
| 9 |
+
- pt
|
| 10 |
+
- es
|
| 11 |
+
pipeline_tag: text-generation
|
| 12 |
+
tags:
|
| 13 |
+
- llama
|
| 14 |
+
- atla
|
| 15 |
+
- evaluation
|
| 16 |
+
- llm-as-a-judge
|
| 17 |
+
- meta
|
| 18 |
+
- conversational
|
| 19 |
+
- lm-judge
|
| 20 |
+
- llama-cpp
|
| 21 |
+
- gptq
|
| 22 |
+
license: llama3.1
|
| 23 |
+
---
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
<p align="center">
|
| 27 |
+
<picture>
|
| 28 |
+
<source
|
| 29 |
+
srcset="https://atla-ai.notion.site/image/https%3A%2F%2Fprod-files-secure.s3.us-west-2.amazonaws.com%2Ff08e6e70-73af-4363-9621-90e906b92ebc%2F1bfb4316-1ce6-40a0-800c-253739cfcdeb%2Fatla_white3x.svg?table=block&id=17c309d1-7745-80f9-8f60-e755409acd8d&spaceId=f08e6e70-73af-4363-9621-90e906b92ebc&userId=&cache=v2"
|
| 30 |
+
media="(prefers-color-scheme: dark)"
|
| 31 |
+
width="200"
|
| 32 |
+
/>
|
| 33 |
+
<source
|
| 34 |
+
srcset="https://atla-ai.notion.site/image/attachment%3A230448e8-921f-45df-b2af-a3158b6c04cd%3Aatla_black2x.png?table=block&id=188309d1-7745-805c-87e4-c39ca54d598d&spaceId=f08e6e70-73af-4363-9621-90e906b92ebc&width=2000&userId=&cache=v2"
|
| 35 |
+
media="(prefers-color-scheme: light)"
|
| 36 |
+
width="200"
|
| 37 |
+
/>
|
| 38 |
+
<img
|
| 39 |
+
src="https://atla-ai.notion.site/image/attachment%3A230448e8-921f-45df-b2af-a3158b6c04cd%3Aatla_black2x.png?table=block&id=188309d1-7745-805c-87e4-c39ca54d598d&spaceId=f08e6e70-73af-4363-9621-90e906b92ebc&width=2000&userId=&cache=v2"
|
| 40 |
+
width="200"
|
| 41 |
+
/>
|
| 42 |
+
</picture>
|
| 43 |
+
</p>
|
| 44 |
+
<p align="center">
|
| 45 |
+
🛝 <a href="https://hf.co/spaces/AtlaAI/selene">Playground</a> |
|
| 46 |
+
📄 <a href="https://huggingface.co/spaces/AtlaAI/selene-1-mini-tech-report">Technical report</a> |
|
| 47 |
+
💻 <a href="https://github.com/atla-ai/selene-mini">GitHub</a> |
|
| 48 |
+
👀 <a href="https://www.atla-ai.com/sign-up-waitlist?utm_source=huggingface&utm_medium=community&utm_campaign=WL_HF_all_communitypost_sel1minilaunch" style="background-image: linear-gradient(to right, red, orange, yellow, green, blue, indigo, violet); -webkit-background-clip: text; color: transparent; animation: rainbow 5s ease infinite; text-decoration: underline; text-decoration-color: currentColor;">Sign up for the API</a>
|
| 49 |
+
</p>
|
| 50 |
+
|
| 51 |
+
<style>
|
| 52 |
+
@keyframes rainbow {
|
| 53 |
+
0% { background-position: 0% 50%; }
|
| 54 |
+
50% { background-position: 100% 50%; }
|
| 55 |
+
100% { background-position: 0% 50%; }
|
| 56 |
+
}
|
| 57 |
+
</style>
|
| 58 |
+
|
| 59 |
+
<style>
|
| 60 |
+
@keyframes rainbow {
|
| 61 |
+
0% { background-position: 0% 50%; }
|
| 62 |
+
50% { background-position: 100% 50%; }
|
| 63 |
+
100% { background-position: 0% 50%; }
|
| 64 |
+
}
|
| 65 |
+
</style>
|
| 66 |
+
|
| 67 |
+
# AtlaAI/Selene-1-Mini-Llama-3.1-8B-GPTQ-W8A8
|
| 68 |
+
This model was quantised into an **8-bit** (W8A8) format using GPTQ and SmoothQuant from [`AtlaAI/Selene-1-Mini-Llama-3.1-8B`](https://huggingface.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B).
|
| 69 |
+
This was done using vLLM's llm-compressor library (https://docs.vllm.ai/en/stable/features/quantization/int8.html)
|
| 70 |
+
|
| 71 |
+
Refer to the [original model card](https://huggingface.co/AtlaAI/Selene-1-Mini-Llama-3.1-8B) for more details on the model.
|
| 72 |
+
|
| 73 |
+
This quantisation was calibrated using a sample of 512 datapoints from the data used to train Selene-1-Mini.
|
| 74 |
+
As a result, our quantised models show minimal performance degradation, losing <0.5% overall across benchmarks!
|
| 75 |
+
|
| 76 |
+
For reference, an [Llama-3.1-8B](neuralmagic/Meta-Llama-3.1-8B-quantized.w8a8) shows ~1.5% degradation across benchmarks.
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
|