xtuner 0.1.2

A toolkit for efficiently fine-tuning LLM

English | 简体中文

👋 join us on Twitter, Discord and WeChat

🎉 News

• [2023.09.06] Support the training of Baichuan2 models! Try it out by xtuner train baichuan2_7b_base_qlora_oasst1_e3!
• [2023.08.30] XTuner is released, with multiple fine-tuned adapters on HuggingFace.

📖 Introduction

XTuner is a toolkit for efficiently fine-tuning LLM, developed by the MMRazor and MMDeploy teams.

• Efficiency: Support LLM fine-tuning on consumer-grade GPUs. The minimum GPU memory required for 7B LLM fine-tuning is only 8GB, indicating that users can use nearly any GPU (even the free resource, e.g., Colab) to fine-tune custom LLMs.
• Versatile: Support various LLMs (InternLM, Llama2, ChatGLM2, Qwen, Baichuan2, ...), datasets (MOSS_003_SFT, Alpaca, WizardLM, oasst1, Open-Platypus, Code Alpaca, Colorist, ...) and algorithms (QLoRA, LoRA), allowing users to choose the most suitable solution for their requirements.
• Compatibility: Compatible with DeepSpeed 🚀 and HuggingFace 🤗 training pipeline, enabling effortless integration and utilization.

🌟 Demos

• Ready-to-use models and datasets from XTuner API

• QLoRA Fine-tune

• Plugin-based Chat

  Examples of Plugin-based Chat 🔥🔥🔥

🔥 Supports

Models	SFT Datasets	Data Pipelines	Algorithms
InternLM InternLM-Chat Llama Llama2 Llama2-Chat ChatGLM2 Qwen Qwen-Chat Baichuan-7B Baichuan-13B-Base Baichuan-13B-Chat Baichuan2-7B-Base ...	MOSS-003-SFT 🔧 Alpaca en / zh WizardLM oasst1 Open-Platypus Code Alpaca Colorist 🎨 Arxiv GenTitle Chinese Law OpenOrca Medical Dialogue ...	Incremental Pre-training Single-turn Conversation SFT Multi-turn Conversation SFT	QLoRA LoRA Full parameter fine-tune

🛠️ Quick Start

Installation

• Install XTuner via pip

  pip install xtuner
  

  or with DeepSpeed integration

  pip install 'xtuner[deepspeed]'
  

• Install XTuner from source

  git clone https://github.com/InternLM/xtuner.git
  cd xtuner
  pip install -e '.[all]'
  

Fine-tune

XTuner supports the efficient fine-tune (e.g., QLoRA) for LLMs. Dataset prepare guides can be found on dataset_prepare.md.

• Step 0, prepare the config. XTuner provides many ready-to-use configs and we can view all configs by

  xtuner list-cfg
  

  Or, if the provided configs cannot meet the requirements, please copy the provided config to the specified directory and make specific modifications by

  xtuner copy-cfg ${CONFIG_NAME} ${SAVE_PATH}
  

• Step 1, start fine-tuning.

  xtuner train ${CONFIG_NAME_OR_PATH}
  

  For example, we can start the QLoRA fine-tuning of InternLM-7B with oasst1 dataset by

  # On a single GPU
  xtuner train internlm_7b_qlora_oasst1_e3
  # On multiple GPUs
  (DIST) NPROC_PER_NODE=${GPU_NUM} xtuner train internlm_7b_qlora_oasst1_e3
  (SLURM) srun ${SRUN_ARGS} xtuner train internlm_7b_qlora_oasst1_e3 --launcher slurm
  

  For more examples, please see finetune.md.

• Step 2, convert the saved PTH model (if using DeepSpeed, it will be a directory) to HuggingFace model, by

  xtuner convert pth_to_hf ${CONFIG_NAME_OR_PATH} ${PTH} ${SAVE_PATH}
  

Chat

XTuner provides tools to chat with pretrained / fine-tuned LLMs.

xtuner chat ${NAME_OR_PATH_TO_LLM} --adapter {NAME_OR_PATH_TO_ADAPTER} [optional arguments]

For example, we can start the chat with Llama2-7b with adapter trained from MOSS-003-SFT by

xtuner chat meta-llama/Llama-2-7b-hf --adapter xtuner/Llama-2-7b-qlora-moss-003-sft --bot-name Llama2 --prompt-template moss_sft --with-plugins calculate solve search --command-stop-word "<eoc>" --answer-stop-word "<eom>" --no-streamer

For more examples, please see chat.md.

Deployment

• Step 0, merge the HuggingFace adapter to pretrained LLM, by

  xtuner convert merge_adapter \
      ${NAME_OR_PATH_TO_LLM} \
      ${NAME_OR_PATH_TO_ADAPTER} \
      ${SAVE_PATH} \
      --max-shard-size 2GB
  

• Step 1, deploy fine-tuned LLM with any other framework, such as LMDeploy 🚀.

  pip install lmdeploy
  python -m lmdeploy.pytorch.chat ${NAME_OR_PATH_TO_LLM} \
      --max_new_tokens 256 \
      --temperture 0.8 \
      --top_p 0.95 \
      --seed 0
  

  🔥 Seeking efficient inference with less GPU memory? Try 4-bit quantization from LMDeploy! For more details, see here.

  🎯 We are woking closely with LMDeploy, to implement the deployment of plugin-based chat!

Evaluation

• We recommend using OpenCompass, a comprehensive and systematic LLM evaluation library, which currently supports 50+ datasets with about 300,000 questions.

🤝 Contributing

We appreciate all contributions to XTuner. Please refer to CONTRIBUTING.md for the contributing guideline.

🎖️ Acknowledgement

• Llama 2
• QLoRA
• LMDeploy

License

This project is released under the Apache License 2.0. Please also adhere to the Licenses of models and datasets being used.