LLaMA-Factory is an open-source project that provides a comprehensive set of tools and scripts for fine-tuning, serving, and benchmarking LLaMA models. LLaMA (Large Language Model Adaptation) is a collection of foundation language models developed by Meta AI that demonstrate strong performance on various natural language tasks.
The LLaMA-Factory repository makes it easy to get started with LLaMA models by providing:
In this article, we'll walk through the key steps of using LLaMA-Factory to fine-tune and deploy a LLaMA model.
To get started, you'll need to set up your Python environment with the required dependencies. It's recommended to use a virtual environment to isolate the packages.
# Create and activate a virtual environment
python -m venv llama-env
source llama-env/bin/activate
# Install required packages
pip install -r requirements.txt
The requirements.txt file in the LLaMA-Factory repo specifies the necessary Python packages including PyTorch, Transformers, Datasets, and more.
You'll also need to have access to the pretrained LLaMA model weights. The weights are not publicly available, but can be requested from Meta for research purposes. Place the model weights in the llama_checkpoints directory.
The next step is to prepare your dataset for fine-tuning. LLaMA-Factory expects the training data to be in a specific JSON format:
[
{
"instruction": "What is the capital of France?",
"input": "",
"output": "Paris is the capital of France."
},
...
]
Each JSON object represents a training example, with fields for:
instruction: The task instruction or promptinput: Additional context for the task (can be empty)output: The target completion or responseYou can prepare your own dataset in this format, or use one of the example datasets provided in the data directory, such as the Alpaca dataset.
To tokenize and process the dataset, run:
python data_preprocess.py \
--data_path data/alpaca_data.json \
--save_path data/alpaca_data_tokenized.json
This will load the JSON dataset, tokenize the text fields, and save the tokenized data to disk. The tokenizer used is the LlamaTokenizer from the Transformers library.
With the data prepared, you can now launch a fine-tuning run using the finetune.py script:
python finetune.py \
--model_name llama-7b \
--data_path data/alpaca_data_tokenized.json \
--output_dir output/llama-7b-alpaca \
--num_train_epochs 3 \
--batch_size 128 \
--learning_rate 2e-5 \
--fp16
The key arguments are:
model_name: The base LLaMA model to fine-tune, e.g. llama-7bdata_path: Path to the tokenized datasetoutput_dir: Directory to save the fine-tuned modelnum_train_epochs: Number of training epochsbatch_size: Batch size for traininglearning_rate: Learning rate for the optimizerfp16: Use FP16 mixed precision to reduce memory usageThe script will load the pretrained LLaMA model, prepare the dataset for training, and run the fine-tuning process using the specified hyperparameters. The fine-tuned model checkpoints will be saved in the output_dir.
Once you have a fine-tuned LLaMA model, you can use it to generate text completions given a prompt. The generate.py script provides an example of how to load a model and perform inference:
python generate.py \
--model_path output/llama-7b-alpaca \
--prompt "What is the capital of France?"
This will load the fine-tuned model from the model_path, tokenize the provided prompt, and generate a text completion using the model's generate() method. You can customize generation parameters like max_length, num_beams, temperature etc.
For interactive testing and demonstration, LLaMA-Factory also provides a Gradio web UI. To launch the UI, run:
python web_ui.py --model_path output/llama-7b-alpaca
This will start a local web server and open the UI in your browser. You can enter prompts and generate completions from the fine-tuned model in real-time.
The web UI code in web_ui.py shows how to integrate the model with Gradio to build an interactive demo. You can extend and customize the UI for your specific use case.
Finally, LLaMA-Factory includes scripts for benchmarking the performance of fine-tuned models on various evaluation datasets. The benchmark.py script provides an example:
python benchmark.py \
--model_path output/llama-7b-alpaca \
--benchmark_datasets alpaca,hellaswag
This will load the fine-tuned model and evaluate its performance on the specified benchmark_datasets. The script reports metrics like accuracy, perplexity, and F1 score.
You can add your own evaluation datasets by implementing a DatasetBuilder class and registering it with the benchmark script.
LLaMA-Factory provides a powerful toolbox for working with LLaMA language models. With its scripts for data processing, fine-tuning, inference, and benchmarking, you can quickly train and deploy adapted LLaMA models for your specific use case.
The modular design of the codebase also makes it easy to extend and customize the pipeline for advanced workflows. You can experiment with different model architectures, training objectives, and inference strategies.
To learn more and contribute to the project, check out the LLaMA-Factory GitHub repository:
hiyouga
