Update README.md

README.md

@@ -1,3 +1,40 @@
----
-license: llama3
----
+---
+license: llama3
+---
+Based on Meta-Llama-3-8b-Instruct, and is governed by Meta Llama 3 License agreement:
+https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct
+
+
+This is a fine tune using an improved Dolphin and WizardLM dataset intended to make the model follow instructions better and refuse less. OpenLLM benchmark is running...
+
+
+Will soon have quants uploaded here on HF and have it up on our site https://awanllm.com for anyone to try.
+
+
+OpenLLM Benchmark:
+
+
+
+Training:
+- 2048 sequence length since the dataset has an average lenght of under 1000 tokens, while the base model is 8192 sequence length. From testing it still performs the same 8192 context just fine.
+- Training duration is around 1 days on 2xRTX 3090, using 4-bit loading and Qlora 64-rank 128-alpha resulting in ~2% trainable weights.
+
+
+Instruct format:
+```
+<|begin_of_text|><|start_header_id|>system<|end_header_id|>
+
+{{ system_prompt }}<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+{{ user_message_1 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+
+{{ model_answer_1 }}<|eot_id|><|start_header_id|>user<|end_header_id|>
+
+{{ user_message_2 }}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
+```
+
+
+Quants:
+
+Generating...
+