---
library_name: peft
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.3
tags:
- axolotl
- generated_from_trainer
datasets:
- json
model-index:
- name: Pretraining-SpongeBoB-7B-Instruct-V1
results: []
---
[
](https://github.com/axolotl-ai-cloud/axolotl)
See axolotl config
axolotl version: `0.8.0.dev0`
```yaml
base_model: mistralai/Mistral-7B-Instruct-v0.3
# optionally might have model_type or tokenizer_type
model_type: MistralForCausalLM
tokenizer_type: LlamaTokenizer
# Automatically upload checkpoint and final model to HF
hub_model_id: AiAF/Pretraining-SpongeBoB-7B-Instruct-V1
load_in_8bit: false
load_in_4bit: true
strict: false
datasets:
- path: json
data_files: [pretraining.jsonl]
type: completion
dataset_prepared_path: last_run_prepared
val_set_size: 0.1
output_dir: ./outputs/qlora-out/Pretraining-SpongeBoB-7B-Instruct-V1
save_total_limit: 10
adapter: qlora
lora_model_dir:
sequence_len: 8192
sample_packing: true
pad_to_sequence_len: true
lora_r: 256
lora_alpha: 64
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:
lora_target_modules:
- gate_proj
- down_proj
- up_proj
- q_proj
- v_proj
- k_proj
- o_proj
wandb_project: "LLM-Pretraining"
wandb_entity:
wandb_watch: "all"
wandb_name: "Pretraining-SpongeBoB-7B-Instruct-V1"
wandb_run_id: "Pretraining-SpongeBoB-7B-Instruct-V1"
wandb_log_model: "false"
gradient_accumulation_steps: 2
micro_batch_size: 9
num_epochs: 10
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.000005
train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false
gradient_checkpointing: true
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1
xformers_attention:
flash_attention: true
loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3
warmup_steps: 10
evals_per_epoch: 5
eval_table_size:
eval_max_new_tokens: 128
saves_per_epoch: 5
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:
```
# Pretraining-SpongeBoB-7B-Instruct-V1
This model is a fine-tuned version of [mistralai/Mistral-7B-Instruct-v0.3](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3) on the json dataset.
It achieves the following results on the evaluation set:
- Loss: 1.6255
## Model description
More information needed
## Intended uses & limitations
More information needed
## Training and evaluation data
More information needed
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 9
- eval_batch_size: 9
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 18
- optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 10.0
### Training results
| Training Loss | Epoch | Step | Validation Loss |
|:-------------:|:------:|:----:|:---------------:|
| 1.7843 | 0.0417 | 1 | 1.7939 |
| 1.8262 | 0.2083 | 5 | 1.7915 |
| 1.839 | 0.4167 | 10 | 1.7733 |
| 1.7503 | 0.625 | 15 | 1.7438 |
| 1.7191 | 0.8333 | 20 | 1.7260 |
| 1.7191 | 1.0417 | 25 | 1.7138 |
| 1.7548 | 1.25 | 30 | 1.7023 |
| 1.6795 | 1.4583 | 35 | 1.6924 |
| 1.6848 | 1.6667 | 40 | 1.6836 |
| 1.6856 | 1.875 | 45 | 1.6770 |
| 1.7155 | 2.0833 | 50 | 1.6715 |
| 1.6901 | 2.2917 | 55 | 1.6665 |
| 1.6797 | 2.5 | 60 | 1.6621 |
| 1.6704 | 2.7083 | 65 | 1.6581 |
| 1.6763 | 2.9167 | 70 | 1.6545 |
| 1.678 | 3.125 | 75 | 1.6516 |
| 1.6271 | 3.3333 | 80 | 1.6490 |
| 1.662 | 3.5417 | 85 | 1.6468 |
| 1.6384 | 3.75 | 90 | 1.6446 |
| 1.6273 | 3.9583 | 95 | 1.6427 |
| 1.5934 | 4.1667 | 100 | 1.6408 |
| 1.6217 | 4.375 | 105 | 1.6393 |
| 1.6383 | 4.5833 | 110 | 1.6378 |
| 1.6244 | 4.7917 | 115 | 1.6365 |
| 1.6238 | 5.0 | 120 | 1.6352 |
| 1.6179 | 5.2083 | 125 | 1.6340 |
| 1.6203 | 5.4167 | 130 | 1.6330 |
| 1.6177 | 5.625 | 135 | 1.6319 |
| 1.6332 | 5.8333 | 140 | 1.6310 |
| 1.6277 | 6.0417 | 145 | 1.6302 |
| 1.6461 | 6.25 | 150 | 1.6296 |
| 1.6668 | 6.4583 | 155 | 1.6290 |
| 1.6249 | 6.6667 | 160 | 1.6284 |
| 1.6013 | 6.875 | 165 | 1.6278 |
| 1.6098 | 7.0833 | 170 | 1.6274 |
| 1.5954 | 7.2917 | 175 | 1.6270 |
| 1.6488 | 7.5 | 180 | 1.6267 |
| 1.6153 | 7.7083 | 185 | 1.6264 |
| 1.6232 | 7.9167 | 190 | 1.6262 |
| 1.6611 | 8.125 | 195 | 1.6260 |
| 1.5997 | 8.3333 | 200 | 1.6258 |
| 1.6166 | 8.5417 | 205 | 1.6258 |
| 1.6427 | 8.75 | 210 | 1.6256 |
| 1.6157 | 8.9583 | 215 | 1.6255 |
| 1.6303 | 9.1667 | 220 | 1.6255 |
| 1.6179 | 9.375 | 225 | 1.6255 |
| 1.6063 | 9.5833 | 230 | 1.6255 |
| 1.6043 | 9.7917 | 235 | 1.6255 |
| 1.5881 | 10.0 | 240 | 1.6255 |
### Framework versions
- PEFT 0.14.0
- Transformers 4.49.0
- Pytorch 2.5.1+cu124
- Datasets 3.2.0
- Tokenizers 0.21.0