Democratizing Medical LLMs For Much More Languages

Covering 12 Major Languages including English, Chinese, French, Hindi, Spanish, Arabic, Russian, Japanese, Korean, German, Italian, Portuguese and 38 Minor Languages So far.

📃 Paper • 🌐 Demo • 🤗 ApolloMoEDataset • 🤗 ApolloMoEBench • 🤗 Models •🌐 Apollo • 🌐 ApolloMoE

Apollo

🌈 Update

  • [2024.10.15] ApolloMoE repo is published!🎉

Languages Coverage

12 Major Languages and 38 Minor Languages

Click to view the Languages Coverage

ApolloMoE

Architecture

Click to view the MoE routing image

ApolloMoE

Results

Dense

🤗 Apollo2-0.5B • 🤗 Apollo2-1.5B • 🤗 Apollo2-2B

🤗 Apollo2-3.8B • 🤗 Apollo2-7B • 🤗 Apollo2-9B

Click to view the Dense Models Results

ApolloMoE

Post-MoE

🤗 Apollo-MoE-0.5B • 🤗 Apollo-MoE-1.5B • 🤗 Apollo-MoE-7B

Click to view the Post-MoE Models Results

ApolloMoE

Usage Format

Apollo2
  • 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>
  • 2B, 9B: User:{query}\nAssistant:{response}<eos>
  • 3.8B: <|user|>\n{query}<|end|><|assisitant|>\n{response}<|end|>
Apollo-MoE
  • 0.5B, 1.5B, 7B: User:{query}\nAssistant:{response}<|endoftext|>

Dataset & Evaluation

  • Dataset 🤗 ApolloMoEDataset

    Click to expand

    ApolloMoE

  • Evaluation 🤗 ApolloMoEBench

    Click to expand
    • EN:

      • MedQA-USMLE
      • MedMCQA
      • PubMedQA: Because the results fluctuated too much, they were not used in the paper.
      • MMLU-Medical
        • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
    • ZH:

      • MedQA-MCMLE
      • CMB-single: Not used in the paper
        • Randomly sample 2,000 multiple-choice questions with single answer.
      • CMMLU-Medical
        • Anatomy, Clinical_knowledge, College_medicine, Genetics, Nutrition, Traditional_chinese_medicine, Virology
      • CExam: Not used in the paper
        • Randomly sample 2,000 multiple-choice questions
    • ES: Head_qa

    • FR:

      • Frenchmedmcqa
      • [MMLU_FR]
        • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
    • HI: MMLU_HI

      • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
    • AR: MMLU_AR

      • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
    • JA: IgakuQA

    • KO: KorMedMCQA

    • IT:

      • MedExpQA
      • [MMLU_IT]
        • Clinical knowledge, Medical genetics, Anatomy, Professional medicine, College biology, College medicine
    • DE: BioInstructQA: German part

    • PT: BioInstructQA: Portuguese part

    • RU: RuMedBench

Results reproduction

Click to expand

We take Apollo2-7B or Apollo-MoE-0.5B as example

  1. Download Dataset for project:

    bash 0.download_data.sh  
    
  2. Prepare test and dev data for specific model:

    • Create test data for with special token
    bash 1.data_process_test&dev.sh
    
  3. Prepare train data for specific model (Create tokenized data in advance):

    • You can adjust data Training order and Training Epoch in this step
    bash 2.data_process_train.sh
    
  4. Train the model

    • If you want to train in Multi Nodes please refer to ./src/sft/training_config/zero_multi.yaml
    bash 3.single_node_train.sh
    
  5. Evaluate your model: Generate score for benchmark

    bash 4.eval.sh
    

Citation

Please use the following citation if you intend to use our dataset for training or evaluation:

@misc{zheng2024efficientlydemocratizingmedicalllms,
      title={Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts}, 
      author={Guorui Zheng and Xidong Wang and Juhao Liang and Nuo Chen and Yuping Zheng and Benyou Wang},
      year={2024},
      eprint={2410.10626},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.10626}, 
}
Downloads last month
14
Safetensors
Model size
10.2B params
Tensor type
BF16
·
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no library tag.

Model tree for nicoboss/Apollo2-9B-llamacppfixed

Base model

google/gemma-2-9b
Finetuned
(227)
this model
Quantizations
2 models

Dataset used to train nicoboss/Apollo2-9B-llamacppfixed