Awan LLM
commited on
Update README.md
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ OpenLLM Benchmark:
|
|
| 16 |
|
| 17 |
|
| 18 |
Training:
|
| 19 |
-
- 2048 sequence length since the dataset has an average
|
| 20 |
- Training duration is around 1 days on 2xRTX 3090, using 4-bit loading and Qlora 64-rank 128-alpha resulting in ~2% trainable weights.
|
| 21 |
|
| 22 |
|
|
|
|
| 16 |
|
| 17 |
|
| 18 |
Training:
|
| 19 |
+
- 2048 sequence length since the dataset has an average length of under 1000 tokens, while the base model is 8192 sequence length. From testing it still performs the same 8192 context just fine.
|
| 20 |
- Training duration is around 1 days on 2xRTX 3090, using 4-bit loading and Qlora 64-rank 128-alpha resulting in ~2% trainable weights.
|
| 21 |
|
| 22 |
|