Difficult Gradient Boosted Determination Timber with Tabular Transformers for Fraud Detection at Reserving.com
Authors: Sergei Krutikov, Bulat Khaertdinov, Rodion Kiriukhin, Shubham Agrawal, Kees Jan De Vries
Summary: Transformer-based neural networks, empowered by Self-Supervised Studying (SSL), have demonstrated unprecedented efficiency throughout varied domains. Nonetheless, associated literature means that tabular Transformers could wrestle to outperform classical Machine Studying algorithms, corresponding to Gradient Boosted Determination Timber (GBDT). On this paper, we purpose to problem GBDTs with tabular Transformers on a typical activity confronted in e-commerce, particularly fraud detection. Our research is moreover motivated by the issue of choice bias, usually occurring in real-life fraud detection methods. It’s attributable to the manufacturing system affecting which subset of site visitors turns into labeled. This difficulty is often addressed by sampling randomly a small a part of the entire manufacturing knowledge, known as a Management Group. This subset follows a goal distribution of manufacturing knowledge and due to this fact is normally most popular for coaching classification fashions with commonplace ML algorithms. Our methodology leverages the capabilities of Transformers to study transferable representations utilizing all accessible knowledge by way of SSL, giving it a bonus over classical strategies. Moreover, we conduct large-scale experiments, pre-training tabular Transformers on huge quantities of knowledge situations and fine-tuning them on smaller goal datasets. The proposed method outperforms closely tuned GBDTs by a substantial margin of the Common Precision (AP) rating. Pre-trained fashions present extra constant efficiency than those skilled from scratch when fine-tuning knowledge is proscribed. Furthermore, they require noticeably much less labeled knowledge for reaching efficiency corresponding to their GBDT competitor that makes use of the entire dataset.