Meta-Evaluation with Untrusted Knowledge
Authors: Shiva Kaul, Geoffrey J. Gordon
Summary: [See paper for full abstract] Meta-analysis is a vital device for answering scientific questions. It’s normally performed on a comparatively small quantity of “trusted’’ information — ideally from randomized, managed trials — which permit causal results to be reliably estimated with minimal assumptions. We present tips on how to reply causal questions rather more exactly by making two modifications. First, we incorporate untrusted information drawn from massive observational databases, associated scientific literature and sensible expertise — with out sacrificing rigor or introducing sturdy assumptions. Second, we practice richer fashions able to dealing with heterogeneous trials, addressing a long-standing problem in meta-analysis. Our method relies on conformal prediction, which essentially produces rigorous prediction intervals, however doesn’t deal with oblique observations: in meta-analysis, we observe solely noisy results because of the restricted variety of contributors in every trial. To deal with noise, we develop a easy, environment friendly model of fully-conformal kernel ridge regression, primarily based on a novel situation known as idiocentricity. We introduce noise-correcting phrases within the residuals and analyze their interplay with a “variance shaving’’ method. In a number of experiments on healthcare datasets, our algorithms ship tighter, sounder intervals than conventional ones. This paper charts a brand new course for meta-analysis and evidence-based medication, the place heterogeneity and untrusted information are embraced for extra nuanced and exact predictions.
