What is Explainable AI (XAI)?
An introduction to building trust in machine learning through global and local variable importance using post-hoc explanations
There is a lot of buzz around AI these days. Almost every company either has plans to incorporate AI, is actively using it, or is rebranding their old rule-based engines as AI-enabled technologies. As more and more companies embed AI and advanced analytics within a business process and automate decisions, there needs to have transparency into how these models make decisions grows larger and larger.
How do we achieve this transparency while harnessing the efficiencies AI brings. This is where the field of Explainable AI (XAI) can help.
Explainable AI refers to methods and techniques in the application of artificial intelligence technology (AI) such that the results of the solution can be understood by human experts.
It contrasts with the concept of the "black box" in machine learning where even their designers cannot explain why the AI arrived at a specific decision. XAI is an implementation of the social right to explanation [1]
In it’s simplest form, AI takes some inputs to produce an output. When we talk about Explainable AI, we are really talking about the input variables impact on the output.
Why Explainability Matters?
AI has the power to automate decisions and those decisions have business impacts, both positive and negative. Much like hiring decision-makers in the organization, it’s important to understand how AI makes decisions. A lot of organizations want to leverage AI but are not comfortable letting the model or AI make more impactful decisions because they do not yet trust the model. Explainability helps with this as it provides insights into how models make decisions.
One of the biggest challenges I’ve found in my experience isn’t building the right model, it’s getting stakeholder buy-in that the model makes a better decision than a human. A better decision is not the same as higher accuracy scores or lower RMSE. It’s using the right inputs to derive a good answer. Often times the decision-maker needs to understand the decision. They need see how the model reasoned to be comfortable handing it off to the model to make the decision.
How do we make people comfortable with the model?
We build trust in AI and models the same way we do as humans. We show our work. Let’s dive into a practical example of explainability.
What if we were to try and predict the mpg on the number of cylinders a car has using a linear regression model.
This can be easily done in any programming tool or even excel these days. We find the coefficient and the intercept and plot it. The nice thing here is that we can simply see how the overall model makes decisions (global level). At the same time, the individual predictions are consistent with the global level predictions.
import requests
import pandas as pd
from io import StringIO
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model## The URL for the dataset
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data' r = requests.get(url)## Read the file in and store as df
file = r.text.replace("t"," ") # list_labels written manually: list_labels = ['mpg', 'cylinders', 'displacement', 'horsepower', 'weight', 'acceleration', 'model year', 'origin','car name']
df = pd.read_csv(StringIO(file),sep="s+",header = None,names=list_labels)## Modeling
regr = linear_model.LinearRegression()
train_x = np.asanyarray(df[['cylinders']])
train_y = np.asanyarray(df[['mpg']])
regr.fit (train_x, train_y)## Plotting the model
plt.scatter(df.cylinders, df.mpg, color='blue')
plt.plot(train_x, regr.coef_[0][0]*train_x + regr.intercept_[0], '-r')
plt.xlabel("Cylinders")
plt.ylabel("MPG")
There is a negative linear relationship, but as the plot shows our predictions are way off. For example, in the 4 cylinder category, our estimate is about 28–29ish, but the values seem to range from 45+ to 19.
We may need to add more features or try a more complex model to achieve the desired model performance.
Increasing Performance through Complexity
We may want to try a more complex model such as Gradient Boosted Decision Trees. GBMs such as XGBoost, H20, LightGBM, AdaBoost, and Catboost consistently win data science competitions for tabular data.
GBM’s converts weak learners (decision trees) into strong learners. They have lower bias and lower variance by using a sampling technique called boosting.
On the spectrum of complexity, they are not as black-box as neural networks nor as transparent as logistic regression.
When we talk about impact, we differentiate between global and local importance:
- Global importance can be thought of as overall how is the model making decisions.
- Local importance is how is the model making decisions for this one person.
These distinctions may seem small, but they have a significant impact as the right to explanation legislation becomes more prevalent.
Increased Complexity Example
Let’s load up a new data set. In this case, we will use the credit card default prediction dataset from UCI. The dataset is from a Taiwanese bank and includes information
The columns include information on each borrower:
- Demographics
- Historical repayment history
- Historical bill amount
- Historical previous payment amount
We can use this dataset to predict payment defaults using XGBoost. Once we have built the model, we need to evaluate its performance and since this is credit decision, we are interested in how it makes decisions.
import requests
import pandas as pd
from io import StringIO
import matplotlib.pyplot as plt
import numpy as np
from sklearn import linear_model
import shap
import xgboost as xgb
from xgboost import XGBClassifier
from sklearn.model_selection import train_test_split
shap.initjs()
import lime
import lime.lime_tabular
import matplotlib.pylab as pl
from sklearn.metrics import confusion_matrix
from sklearn.metrics import roc_auc_score
from sklearn.metrics import accuracy_score## Read in the file
df = pd.read_excel('https://archive.ics.uci.edu/ml/machine-learning-databases/00350/default%20of%20credit%20card%20clients.xls',header = 1)## Store target in y variable and split to train test
y = df['default payment next month']
df = df.drop(['default payment next month','ID'], axis =1)
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.20, random_state=42)## Train XGBoost Model
xgb_model = XGBClassifier(learning_rate = 0.05, verbosity = 1, n_estimators = 200 ,scale_pos_weight =4, random_state = 2456 )
xgb_model.fit(X_train, y_train)## Predict
y_pred = xgb_model.predict(X_test) roc_auc_score(y_test,y_pred)Feature importance plots are what many data scientist use to evaluate the impact the input variables have on the output variable. Notice that depending on which metric (cover or gain), different feature come to the top. How do you explain to a customer that you declined their application due to their marital status? Is that really how the model made its decision?
## Plot importance (cover)
xgb.plot_importance(xgb_model, importance_type="cover")
pl.title('xgboost.plot_importance(model, importance_type="cover")')
pl.show()## Plot importance (gain)
xgb.plot_importance(xgb_model, importance_type="gain")
pl.title('xgboost.plot_importance(model, importance_type="gain")')
pl.show()
The truth is that while these feature importance plots show the overall impact of the features, they are not able to explain how the model makes decisions for individual people.
Global interpretability of models entails seeking to understand the overall structure of the model. What if I want to understand how a model made a prediction for a specific person?
Local Interpretability
Local interpretability of models consists of providing detailed explanations for why an individual prediction was made.
Let’s talk about the first local interpretability framework, often referred to as LIME.
LIME randomly samples from the prediction and uses simpler models to explain the prediction. Since it is not tied to a specific model, it is considered model agnostic.
xgb_model.fit(X_train.as_matrix(), y_train)
predict_fn_xgb = lambda x: xgb_model.predict_proba(x).astype(float)
explainer = lime.lime_tabular.LimeTabularExplainer(X_train.to_numpy(), feature_names = X_train.columns)
exp = explainer.explain_instance(X_test.to_numpy()[observation_1], predict_fn_xgb, num_features = 9)
exp.show_in_notebook()
If you look closely you can see that the top feature is Pay > 0 meaning that the person is behind on their payments.
LIME uses a decision tree to explain the more complex models. Because LIME estimates the prediction using local approximations it can sometimes be risky to use as a ground truth explanation.
Shapley Values (SHAP)
SHAP is another novel approach to explainability developed by Scott Lundberg at Microsoft and eventually opened sourced.
SHAP has a strong mathematical foundation based on Shapley values in game theory where each player in the cooperation is rewarded based on how important they are to cooperation.
Here we see the same most likely defaulting borrower and the impact each variable has on the output. These local explanations are accurate and can be used to explain how XGBoost makes its decision.
## Calculate SHAP Values
explainer = shap.TreeExplainer(xgb_model)
expected_value = explainer.expected_value## Generate forceplot
shap.force_plot(expected_value,shap_values[observation_1],features_display.iloc[observation_1] ,X_test.columns,link='logit', matplotlib=True)
SHAP can illustrate local importance and the really nice thing about SHAP is that it is globally consistent. Several model frameworks can spit out SHAP values (XGBoost for example) directly from the code.
shap.summary_plot(shap_values, X_test)
The last framework I’ll mention t is Explainable Boosting Machines.
Explainable Boosting Machines or EBMs is another Microsoft open source library. Instead of being a post-hoc explanation framework, they are considered more white-box than XGBoost, while reaching similar performance.
EBM uses a framework called GA2M, which is a generalized additive model with an interaction function added to it.
The models have been proven to be as accurate as XGBoost and a GA2M won a data science contest hosted by FICO a year or so ago.
from interpret.glassbox import ExplainableBoostingClassifier
from interpret import show## Train Model
ebm = ExplainableBoostingClassifier(scoring = 'auc')
ebm.fit(X_train, y_train)## Calculate Global Explaination
ebm_global = ebm.explain_global()## Plot Global Variable Importance
show(ebm_global)
InterpretML is still in alpha release right now, but for tabular data GA2M models they are starting to become the de-facto standard for white-box models.
There is a very nice user interface with a dashboard that explains the predictions from the global level down to the local level, much like SHAP.
## Local explainations
ebm_local = ebm.explain_local(X_test, y_test)
show(ebm_local)
When should you use what?
You should use explainability techniques such as LIME and SHAP when you need to build trust with your stakeholders by being transparent in modeling. If you are a decision-maker, always ask your data scientist or vendor for explanations of how the model makes decisions. As with almost everything in life, the best model and explanation option usually depends on the situation.
[1] Explainable Artificial Intelligence, Wikipedia
[2] Lundberg, Scott M and Lee, Su-In, A Unified Approach to Interpreting Model Predictions (2017), Advances in Neural Information Processing Systems 30
[3] H. Nori, S. Jenkins, P. Koch, and R. Caruana, InterpretML: A Unified Framework for Machine Learning Interpretability (2019)
Share This Article
Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program.
Write for TDS
