Understanding the nature of chemical bonding and its variation in strength across physically tunable factors is important for the development of novel catalytic materials. One way to speed up this process is to employ machine learning (ML) algorithms with online data repositories curated from high-throughput experiments or quantum-chemical simulations. Despite the reasonable predictive performance of ML models for predicting reactivity properties of solid surfaces, the ever-growing complexity of modern algorithms, e.g., deep learning, makes them black boxes with little to no explanation. In this Perspective, we discuss recent advances of interpretable ML for opening up these black boxes from the standpoints of feature engineering, algorithm development, and post hoc analysis. We underline the pivotal role of interpretability as the foundation of next-generation ML algorithms and emerging AI platforms for driving discoveries across scientific disciplines.