Training DBRX-like model

#13

by nguyenthanhdo - opened Mar 31, 2024

Mar 31, 2024

Hi, i've seen the team mentioned that the code used for training DBRX are optimized versions of Composer, LLM Foundry, MegaBlocks and Streaming but I found it quite challenging to navigate. I want to pretrain a MOE model (DBRX-like architecture), could you please guide me how would I do it with the opensourced versions of those mentioned libs?

GODCRM

Apr 2, 2024

This comment has been hidden

GODCRM

Apr 2, 2024

This comment has been hidden

eitanturok

Databricks org Apr 16, 2024

Try reading this: https://github.com/mosaicml/llm-foundry/tree/main/scripts/train

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment