Skip to Main Content

Fine-Tuning Model Categories

"Full fine-tuning is like a heavy sword, while parameter-efficient fine-tuning is like a small knife. You don't use a heavy sword to cut fruits, and you don't use a small knife for battle." - Ida Alchemy, member of the Alchemy Association, wrote in his notes.

Fine-tuning models can be divided into two categories:

  • Full Fine-Tuning: All parameters of the model are trained, resulting in a new model.
  • Parameter-Efficient Fine-Tuning: Only a small number of parameters are fine-tuned, resulting in a small model that cannot be used alone and needs to be used together with the original model.

These two techniques are suitable for different scenarios.

Examples are given below.

Usage Scenarios for Full Fine-Tuning​

Suppose we have a Stable Diffusion 2.1 model that can generate various scenarios of images.

Now we want to use this model only to generate various real-life scenarios of human photos. In this case, we should use full fine-tuning.

Prepare a large number (10,000 or even more) of real human photos, and fine-tune the model based on these images to train a dedicated model that only generates real human photos.

After obtaining this specialized model for real-life images, we can put the newly trained model into the image generation workstation and start generating real human photos.

Usage Scenarios for Parameter-Efficient Fine-Tuning​

Next, we have another idea. We hope that the people appearing in the photos generated by this model are all a fixed celebrity, that is, we hope that the faces of the people appearing in the photos are fixed.

In this case, we can use parameter-efficient fine-tuning, prepare a small number (10-50) of photos of that celebrity, and fine-tune a parameter-efficient Lora model based on these images.

After obtaining this small model, we need to use the original large model and this parameter-efficient small model together when generating images.

Full Fine-Tuning: Requires more computing resources and is usually more suitable for larger-scale data sets. It is usually suitable for large-scale adjustments, such as overall style adjustments for a model, or a model primarily used for generating two-dimensional images that now needs to be turned into a model for generating real-life images.

Parameter-Efficient Fine-Tuning: More convenient for training and use, such as wanting a model to generate a specific person's face or a specific type of clothing.