Fine-tuning a neural network typically involves retraining every layer on new data. But research shows that networks may perform better when fine-tuning modifies only a subset of layers.
What’s new: Yoonho Lee, Annie S. Chen, and colleagues at Stanford demonstrated surgical fine-tuning, a method that chooses specific layers to modify depending on how the fine-tuning dataset differs from the pretraining data.
Key insight: Earlier layers in a neural network learn to produce representations of fundamental features of the input, such as edges and shapes in an image, while later layers combine these features in a way that contributes to predicting a desired output, such as the image’s label. During fine-tuning, if the new images differ from the pretraining images in appearance, only earlier layers require modification. If the new images resemble the pretraining images but differ in their labels, only later layers require modification. Fine-tuning the appropriate layers updates a network effectively by prioritizing the weights most relevant to the new data.
How it works: The authors fine-tuned a ResNet-26 model pretrained on CIFAR-10 using manual and automated approaches.
- In the manual approach, the authors fine-tuned each layer individually, producing a new network each time. They identified the best layers to fine-tune by comparing the performance of each network.
- In the automated approach, they calculated the gradients for each layer. They divided the gradients by the magnitude of the layer’s weights to obtain relative gradients. They normalized the relative gradients across each layer at the beginning of fine-tuning and periodically throughout. This effectively ranked the layers from lowest to highest relative gradient on a scale from 0 to 1.
- During training, they assigned the learning rate for each layer according to the product of its normalized relative gradient (its score between 0 and 1) and a standard learning rate. This way, layers with the largest relative gradient would have the largest learning rate, while layers with the smallest relative gradient would have an effective learning rate of 0 and remain unchanged by fine-tuning.
Results: Evaluated on CIFAR-C, a version of the CFAR dataset deliberately corrupted by noise, the authors’ manual method classified images with 82.8 percent accuracy, while fine-tuning the whole network achieved 79.9 percent accuracy. The automated approach achieved 81.4 percent.
Why it matters: The authors drew on knowledge of how the neural networks process input to propose an efficient fine-tuning method. Better understanding of how a network extracts features could yield further ways to improve machine learning models.
We’re thinking: On datasets more complex than CIFAR-C, it can be hard to judge the difference between a pretraining dataset and a fine-tuning dataset. This may make the authors’ automated approach more valuable, even though it didn’t yield the best results.