SentenceTransformer based on sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2

This is a sentence-transformers model finetuned from sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Sources

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hatemestinbejaia/mmarco-Arabic-mMiniLML-bi-encoder-KD-v1-Nonormalisation")
# Run inference
sentences = [
    'تحديد المسح',
    'المسح أو مسح الأراضي هو تقنية ومهنة وعلم تحديد المواقع الأرضية أو ثلاثية الأبعاد للنقاط والمسافات والزوايا بينها . يطلق على أخصائي مسح الأراضي اسم مساح الأراضي .',
    'إجمالي المحطات . تعد المحطات الإجمالية واحدة من أكثر أدوات المسح شيوعا المستخدمة اليوم . وهي تتألف من جهاز ثيودوليت إلكتروني ومكون إلكتروني لقياس المسافة ( EDM ) . تتوفر أيضا محطات روبوتية كاملة تتيح التشغيل لشخص واحد من خلال التحكم في الجهاز باستخدام جهاز التحكم عن بعد . تاريخ',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Reranking

Metric Value
map 0.5526
mrr@10 0.5566
ndcg@10 0.6272

Training Details

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 128
  • learning_rate: 7e-05
  • warmup_ratio: 0.07
  • fp16: True
  • half_precision_backend: amp
  • load_best_model_at_end: True
  • fp16_backend: amp

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 128
  • per_device_eval_batch_size: 8
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 7e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 3
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.07
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: amp
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: amp
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • eval_use_gather_object: False
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss map
0.0512 2000 60.0645 39.7459 0.4997
0.1024 4000 38.1556 34.3446 0.4994
0.1536 6000 33.7868 32.9171 0.5159
0.2048 8000 31.8491 29.9714 0.5282
0.2560 10000 29.7765 29.9015 0.5078
0.3072 12000 27.5914 26.7202 0.5283
0.3584 14000 25.8129 25.0254 0.5430
0.4096 16000 24.0781 25.0622 0.5207
0.4608 18000 22.9328 23.7991 0.5433
0.5120 20000 21.7429 22.0272 0.5333
0.5632 22000 20.9529 20.9957 0.5485
0.6144 24000 19.9476 19.8111 0.5304
0.6656 26000 19.1556 19.2983 0.5363
0.7168 28000 18.5506 20.4461 0.5421
0.7680 30000 17.8418 19.6846 0.5192
0.8192 32000 17.4182 18.3179 0.5268
0.8704 34000 16.8575 18.5912 0.5401
0.9216 36000 16.4331 17.6217 0.5448
0.9728 38000 15.8319 16.4225 0.5469
1.0240 40000 14.5094 16.8592 0.5283
1.0752 42000 13.2263 15.6646 0.5511
1.1264 44000 12.9718 16.8053 0.5599
1.1776 46000 12.9135 16.9315 0.5557
1.2288 48000 12.6887 16.6569 0.5588
1.2800 50000 12.4705 15.5349 0.5569
1.3312 52000 12.3431 15.9067 0.5597
1.3824 54000 12.0741 15.0079 0.5668
1.4336 56000 11.9194 14.9333 0.5532
1.4848 58000 11.7261 14.3567 0.5598
1.5360 60000 11.5138 14.8380 0.5608
1.5872 62000 11.3494 13.7454 0.5544
1.6384 64000 11.116 14.3529 0.5527
1.6896 66000 11.0054 13.8486 0.5403
1.7408 68000 10.8677 13.8550 0.5598
1.7920 70000 10.6486 15.1113 0.5526
1.8432 72000 10.4977 13.7056 0.5580
1.8944 74000 10.3649 14.4802 0.5526
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.11.9
  • Sentence Transformers: 3.1.1
  • Transformers: 4.45.2
  • PyTorch: 2.4.1+cu121
  • Accelerate: 1.4.0
  • Datasets: 3.2.0
  • Tokenizers: 0.20.3

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MarginMSELoss

@misc{hofstätter2021improving,
    title={Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation},
    author={Sebastian Hofstätter and Sophia Althammer and Michael Schröder and Mete Sertkan and Allan Hanbury},
    year={2021},
    eprint={2010.02666},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}
Downloads last month
7
Safetensors
Model size
118M params
Tensor type
F32
·
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.

Model tree for hatemestinbejaia/mmarco-Arabic-mMiniLML-bi-encoder-KD-v1-Nonormalisation

Evaluation results