Predicting Fine-Tuning Performance with Probing

Zining Zhu, Soroosh Shahtalebi, Frank Rudzicz


Abstract
Large NLP models have recently shown impressive performance in language understanding tasks, typically evaluated by their fine-tuned performance. Alternatively, probing has received increasing attention as being a lightweight method for interpreting the intrinsic mechanisms of large NLP models. In probing, post-hoc classifiers are trained on “out-of-domain” datasets that diagnose specific abilities. While probing the language models has led to insightful findings, they appear disjointed from the development of models. This paper explores the utility of probing deep NLP models to extract a proxy signal widely used in model development – the fine-tuning performance. We find that it is possible to use the accuracies of only three probing tests to predict the fine-tuning performance with errors 40% - 80% smaller than baselines. We further discuss possible avenues where probing can empower the development of deep NLP models.
Anthology ID:
2022.emnlp-main.793
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
11534–11547
Language:
URL:
https://aclanthology.org/2022.emnlp-main.793
DOI:
10.18653/v1/2022.emnlp-main.793
Bibkey:
Cite (ACL):
Zining Zhu, Soroosh Shahtalebi, and Frank Rudzicz. 2022. Predicting Fine-Tuning Performance with Probing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11534–11547, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Predicting Fine-Tuning Performance with Probing (Zhu et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.793.pdf
Software:
 2022.emnlp-main.793.software.zip