CELDA: Leveraging Black-box Language Mannequin as Enhanced Classifier with out Labels
Authors: Hyunsoo Cho, Youna Kim, Sang-goo Lee
Summary: Using language fashions (LMs) with out inside entry is changing into a sexy paradigm within the discipline of NLP as many cutting-edge LMs are launched via APIs and boast a large scale. The de-facto methodology in the sort of black-box situation is called prompting, which has proven progressive efficiency enhancements in conditions the place information labels are scarce or unavailable. Regardless of their efficacy, they nonetheless fall brief compared to totally supervised counterparts and are typically brittle to slight modifications. On this paper, we suggest Clustering-enhanced Linear Discriminative Evaluation, a novel strategy that improves the textual content classification accuracy with a really weak-supervision sign (i.e., identify of the labels). Our framework attracts a exact resolution boundary with out accessing weights or gradients of the LM mannequin or information labels. The core concepts of CELDA are twofold: (1) extracting a refined pseudo-labeled dataset from an unlabeled dataset, and (2) coaching a light-weight and sturdy mannequin on the highest of LM, which learns an correct resolution boundary from an extracted noisy dataset. All through in-depth investigations on varied datasets, we demonstrated that CELDA reaches new state-of-the-art in weakly-supervised textual content classification and narrows the hole with a fully-supervised mannequin. Moreover, our proposed methodology could be utilized universally to any LM and has the potential to scale to bigger fashions, making it a extra viable possibility for using giant LMs.
