Formal definition of the MARS method for quantifying the unique target class discoveries of selected machine classifiers

TR Number
Date
2022-07
Journal Title
Journal ISSN
Volume Title
Publisher
F1000 Research
Abstract

Conventional binary classification performance metrics evaluate either general measures (accuracy, F score) or specific aspects (precision, recall) of a model's classifying ability. As such, these metrics, derived from the model's confusion matrix, provide crucial insight regarding classifier-data interactions. However, modern- day computational capabilities have allowed for the creation of increasingly complex models that share nearly identical classification performance. While traditional performance metrics remain as essential indicators of a classifier's individual capabilities, their ability to differentiate between models is limited. In this paper, we present the methodology for MARS (Method for Assessing Relative Sensitivity/ Specificity) ShineThrough and MARS Occlusion scores, two novel binary classification performance metrics, designed to quantify the distinctiveness of a classifier's predictive successes and failures, relative to alternative classifiers. Being able to quantitatively express classifier uniqueness adds a novel classifier-classifier layer to the process of model evaluation and could improve ensemble model-selection decision making. By calculating both conventional performance measures, and proposed MARS metrics for a simple classifier prediction dataset, we demonstrate that the proposed metrics' informational strengths synergize well with those of traditional metrics, delivering insight complementary to that of conventional metrics.

Description
Keywords
Binary classification, Classifier comparative uniqueness, Classifier performance evaluation, Classifier selection optimization, Machine learning
Citation