ATOM: Consideration Mixer for Environment friendly Dataset Distillation
Authors: Samir Khaki, Ahmad Sajedi, Kai Wang, Lucy Z. Liu, Yuri A. Lawryshyn, Konstantinos N. Plataniotis
Summary: Latest works in dataset distillation search to attenuate coaching bills by producing a condensed artificial dataset that encapsulates the knowledge current in a bigger actual dataset. These approaches in the end goal to realize take a look at accuracy ranges akin to these achieved by fashions skilled on the whole thing of the unique dataset. Earlier research in characteristic and distribution matching have achieved important outcomes with out incurring the prices of bi-level optimization within the distillation course of. Regardless of their convincing effectivity, many of those strategies endure from marginal downstream efficiency enhancements, restricted distillation of contextual data, and subpar cross-architecture generalization. To handle these challenges in dataset distillation, we suggest the ATtentiOn Mixer (ATOM) module to effectively distill massive datasets utilizing a combination of channel and spatial-wise consideration within the characteristic matching course of. Spatial-wise consideration helps information the training course of based mostly on constant localization of courses of their respective photos, permitting for distillation from a broader receptive subject. In the meantime, channel-wise consideration captures the contextual data related to the category itself, thus making the artificial picture extra informative for coaching. By integrating each kinds of consideration, our ATOM module demonstrates superior efficiency throughout varied pc imaginative and prescient datasets, together with CIFAR10/100 and TinyImagenet. Notably, our methodology considerably improves efficiency in situations with a low variety of photos per class, thereby enhancing its potential. Moreover, we keep the development in cross-architectures and purposes comparable to neural structure search
