File size: 2,926 Bytes
785fe8a e8150c5 785fe8a 7143865 76f24b5 cb6f594 7143865 f9a1201 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
---
language:
- en
license: apache-2.0
---
<div align="center">
<b style="font-size: 30px;">LLAMA-3_8B_Unaligned_Alpha_RP_Soup</b>
</div>
<img src="https://i.imgur.com/pXcjpoV.png" alt="LLAMA-3_8B_Unaligned_Alpha_RP_Soup" style="width: 50%; min-width: 400px; display: block; margin: auto;">
# Model Details
This model is the outcome of multiple merges, starting with the base model SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha. The merging process was conducted in several stages:
Merge 1: LLAMA-3_8B_Unaligned_Alpha was SLERP merged with EtherealRainbow-v0.3-8B.
Merge 2: LLAMA-3_8B_Unaligned_Alpha was SLERP merged with TheDrummer/Llama-3SOME-8B-v2.
Soup 1: Merge 1 was combined with Merge 2.
Final Merge: Soup 1 was SLERP merged with Nitral-Archive/Hathor_Enigmatica-L3-8B-v0.4.
The final model is surprisingly coherent (although slightly more censored), which is a bit unexpected, since all the intermediate merge steps were pretty incoherent.
<details>
<summary>Mergekit configs:</summary>
# Merge 1
```yaml
slices:
- sources:
- model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
layer_range: [0, 32]
- model: BeaverAI/Llama-3SOME-8B-v2d
layer_range: [0, 32]
merge_method: slerp
base_model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```
# Merge 2
```yaml
slices:
- sources:
- model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
layer_range: [0, 32]
- model: invisietch/EtherealRainbow-v0.3-8B
layer_range: [0, 32]
merge_method: slerp
base_model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```
# Soup 1
```yaml
slices:
- sources:
- model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
layer_range: [0, 32]
- model: Nitral-Archive/Hathor_Enigmatica-L3-8B-v0.4
layer_range: [0, 32]
merge_method: slerp
base_model: SicariusSicariiStuff/LLAMA-3_8B_Unaligned_Alpha
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```
# Final Merge
```yaml
slices:
- sources:
- model: Soup 1
layer_range: [0, 32]
- model: Nitral-Archive/Hathor_Enigmatica-L3-8B-v0.4
layer_range: [0, 32]
merge_method: slerp
base_model: Soup 1
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
```
</details>
|