Will fine-tuning affect the ability of the original search?

[Daydaylight]

After fine-tuning on a small dataset containing new categories (e.g., containing 500 black cats and 500 orange cats), the accuracy of retrieving these two color cats goes up, so does the accuracy of the fine-tuned CLIP to retrieve the original categories go down (e.g., to retrieve the other animal, dog)?

[gabrielilharco]

In general yes, fine-tuning typically overspecializes the model. We explore this and solutions in these papers and associated repos: https://arxiv.org/abs/2109.01903 https://arxiv.org/abs/2208.05592 https://github.com/mlfoundations/wise-ft https://github.com/mlfoundations/patching