In machine learning, understanding why a model makes certain decisions is often just as important as whether those decisions are correct. For instance, a machine-learning model might correctly predict that a skin lesion is cancerous, but it could have done so using an unrelated blip on a clinical photo.
source https://www.lifetechnology.com/blogs/life-technology-technology-news/new-method-compares-machine-learning-models-reasoning-to-that-of-a-human
