lora-parameters

Finding the best LoRA parameters

We finetune Mistral-7B using LoRA and DeepSpeed. We ran LoRA on two 40 GB A100 GPUs utilizing DeepSpeed.

See our blog post for our experiment results.

To get started, first install Determined on your local machine:

pip install determined

Then finetune with LoRA:

det e create lora.yaml . 

You can view the actual training code in finetune.py.

Configuration

Change configuration options in lora.yaml. Some important options are:

• slots_per_trial: the number of GPUs to use.
• dataset_subset: the difficulty subset to train on.
• per_device_train_batch_size: the batch size per GPU.

DeepSpeed configuration files are in the ds_configs folder.

Contributors

• By Sze Wai Yuen
• Built on llm-finetuning code by Agnieszka Ciborowska and Kevin Musgrave.