The electrochemical oxygen reduction reaction (ORR) is a very important catalytic process that is directly used in carbon-free energy systems like fuel cells. However, the lack of active, stable, and cost-effective ORR cathode materials has been a major impediment to the broad adoption of these technologies. So, the challenge for researchers in catalysis is to find catalysts that are electrochemically efficient to drive the reaction, made of earth-abundant elements to lower material costs and allow scalability, and stable to make them last longer. The majority of commercial catalysts that are now being used have been found through trial and error techniques that rely on the chemical intuition of experts. This method of empirical discovery is, however, very challenging, slow, and complicated because the performance of the catalyst depends on a myriad of factors. Researchers have recently turned to machine learning (ML) to find and design heterogeneous catalysts faster with emerging catalysis databases. Black-box models make up a lot of the ML models that are used in the field to predict the properties of catalysts that are important to their performance, such as their adsorption energies to reaction intermediates. However, as these black-box models are based on very complicated mathematical formulas, it is very hard to figure out how they work and the underlying physics of the desired catalyst properties remains hidden. As a way to open up these black boxes and make them easier to understand, more attention is being paid to interpretable and explainable ML. This work aims to speed up the process of screening and optimizing Pt monolayer alloys for ORR while gaining physical insights. We use a theory-infused machine learning framework in combination with a high-throughput active screening approach to effectively find promising ORR Pt monolayer catalysts. Furthermore, an explainability game-theory approach is employed to find electronic factors that control surface reactivity. The novel insights in this study can provide new design strategies that could shape the paradigm of catalyst discovery.