LIME to Inspect Image & Text Classification¶

This tutorial focuses on showing how to use Captum's implementation of Local Interpretable Model-agnostic Explanations (LIME) to understand neural models. The following content is divided into an image classification section to present our high-level interface Lime class and a text classification section for the more customizable low-level interface LimeBase.

1. Image Classification¶

In this section, we will learn applying Lime to analyze a Resnet trained on ImageNet-1k. For testing data, we use samples from PASCAL VOC 2012 since its segmentation masks can directly serve as semantic "super-pixels" for images.

1.1 Load the model and dataset¶

We can directly load the pretrained Resnet from torchvision and set it to evaluation mode as our target image classifier to inspect.

This model predicts ImageNet-1k labels for given sample images. To better present the results, we also load the mapping of label index and text.

As mentioned before, we will use PASCAL VOC 2012 as the test data, which is available in torchvision as well. We will load it with torchvision transforms which convert both the images and targets, i.e., segmentation masks, to tensors.

This dataset provides an addional segmentation mask along with every image. Compared with inspecting each pixel, the segments (or "super-pixels") are semantically more intuitive for human to perceive. We will discuss more in section 1.3.

Let's pick one example to see how the image and corresponding mask look like. Here we choose an image with more than one segments besides background so that we can compare each segment's impact on the classification.

1.2 Baseline classification¶

We can check how well our model works with the above example. The original Resnet only gives the logits of labels, so we will add a softmax layer to normalize them into probabilities.

Then we present the top 5 predicted labels to verify the result.

As we can see, the result is pretty reasonable.

1.3 Inspect the model prediction with Lime¶

In this section, we will bring in LIME from Captum to analyze how the Resnet made above prediction based on the sample image.

Like many other Captum algorithms, Lime also support analyzing a number of input features together as a group. This is very useful when dealing with images, where each color channel in each pixel is an input feature. Such group is also refered as "super-pixel". To define our desired groups over input features, all we need is to provide a feature mask.

In case of an image input, the feature mask is a 2D image of the same size, where each pixel in the mask indicates the feature group it belongs to via an integer value. Pixels of the same value define a group.

This means we can readily use VOC's segmentation masks as feature masks for Captum! However, while segmentaion numbers range from 0 to 255, Captum prefers consecutive group IDs for efficiency. Therefore, we will also include extra steps to convert mask IDs.

It is time to configure our Lime algorithm now. Essentially, Lime trains an interpretable surrogate model to simulate the target model's predictions. So, building an appropriate interpretable model is the most critical step in Lime. Fortunately, Captum has provided many most common interpretable models to save the efforts. We will demonstrate the usages of Linear Regression and Linear Lasso. Another important factor is the similarity function. Because Lime aims to explain the local behavior of an example, it will reweight the training samples according to their similarity distances. By default, Captum's Lime uses the exponential kernel on top of the consine distance. We will change to euclidean distance instead which is more popular in vision.

Next, we will analyze these groups' influence on the most confident prediction television. Every time we call Lime's attribute function, an interpretable model is trained around the given input, so unlike many other Captum's attribution algorithms, it is strongly recommended to only provide a single example as input (tensors with first dimension or batch size = 1). There are advanced use cases of passing batched inputs. Interested readers can check the documentation for details.

In order to train the interpretable model, we need to specify enough training data through the argument n_samples. Lime creates the perturbed samples in the form of interpretable representation, i.e., a binary vector indicating the “presence” or “absence” of features. Lime needs to keep calling the target model to get the labels/values for all perturbed samples. This process can be quite time-consuming depending on the complexity of the target model and the number of samples. Setting the perturbations_per_eval can batch multiple samples in one forward pass to shorten the process as long as your machine still has capacity. You may also consider turning on the flag show_progress to display a progess bar showing how many forward calls are left.

Now, let us use Captum's visualization tool to view the attribution heat map.

The result looks decent: the television segment does demonstrate strongest positive correlation with the prediction, while the chairs has relatively trivial impact and the border slightly shows negative contribution.

However, we can further improve this result. One desired characteristic of interpretability is the ease for human to comprehend. We should help reduce the noisy interference and emphisze the real influential features. In our case, all features more or less show some influences. Adding lasso regularization to the interpretable model can effectively help us filter them. Therefore, let us try Linear Lasso with a fit coefficient alpha. For all built-in sklearn wrapper model, you can directly pass any sklearn supported arguments.

Moreover, since our example only has 4 segments, there are just 16 possible combinations of interpretable representations in total. So we can exhaust them instead random sampling. The Lime class's argument perturb_func allows us to pass a generator function yielding samples. We will create the generator function iterating the combinations and set the n_samples to its exact length.

As we can see, the new attribution result removes the chairs and border with the help of Lasso.

Another interesting question to explore is if the model also recognize the chairs in the image. To answer it, we will use the most related label rocking_chair from ImageNet as the target, whose label index is 765. We can check how confident the model feels about the alternative object.

Then, we will redo the attribution with our Lasso Lime.

As shown in the heat map, our ResNet does present right belief about the chair segment. However, it gets hindered by the television segment in the foreground. This may also explain why the model feels less confident about the chairs than the television.

1.4 Understand the sampling process¶

We have already learned how to use Captum's Lime. This section will additionally dive into the internal sampling process to give interested readers an overview of what happens underneath. The goal of the sampling process is to collect a set of training data for the surrogate model. Every data point consists of three parts: interpretable input, model predicted label, and similarity weight. We will roughly illustrate how Lime achieve each of them behind the scene.

As we mentioned before, Lime samples data from the interpretable space. By default, Lime uses the presense or absense of the given mask groups as interpretable features. In our example, facing the above image of 4 segments, the interpretable representation is therefore a binary vector of 4 values indicating if each segment is present or absent. This is why we know there are only 16 possible interpretable representations and can exhaust them with our iter_combinations. Lime will keep calling its perturb_func to get the sample interpretable inputs. Let us simulate this step and give us a such interpretable input.

Our input sample [1, 1, 0, 1] means the third segment (television) is absent while other three segments stay.

In order to find out what the target ImageNet's prediction is for this sample, Lime needs to convert it from interpretable space back to the original example space, i.e., the image space. The transformation takes the original example input and modify it by setting the features of the absent groups to a baseline value which is 0 by default. The transformation function is called from_interp_rep_transform under Lime. We will run it manually here to get the pertubed image input and then visualize what it looks like.

As shown above, compared with the original image, the absent feature, i.e., the television segment, gets masked in the perturbed image, while the rest present features stay unchanged. With the perturbed image, Lime is able to find out the model's prediction. Let us still use "television" as our attribution target, so the label of perturbed sample is the value of the model's prediction on "television". Just for curiosity, we can also check how the model's prediction changes with the perturbation.

Reasonably, our ImageNet no longer feel confident about classifying the image as television.

At last, because Lime focuses on the local interpretability, it will calculate the similarity between the perturbed and original images to reweight the loss of this data point. Note the calculation is based on the input space instead of the interpretable space. This step is simply passing the two image tensors into the given similarity_func argument which is the exponential kernel of euclidean distance in our case.

This is basically how Lime create a single training data point of sample_interp_inp, sample_label, and sample_similarity. By repeating this process n_samples times, it collects a dataset to train the interpretable model.

Worth noting that the steps we showed in this section is an example based on our Lime instance configured above. The logic of each step can be customized, especially with LimeBase class which will be demonstrated in Section 2.

2. Text Classification¶

In this section, we will take use of a news subject classification example to demonstrate more customizable functions in Lime. We will train a simple embedding-bag classifier on AG_NEWS dataset and analyze its understanding of words.

2.1 Load the data and define the model¶

torchtext has included the AG_NEWS dataset but since it is only split into train & test, we need to further cut a validation set from the original train split. Then we build the vocabulary of the frequent words based on our train split.

The model we use is composed of an embedding-bag, which averages the word embeddings as the latent text representation, and a final linear layer, which maps the latent vector to the logits. Unconventially, pytorch's embedding-bag does not assume the first dimension is batch. Instead, it requires a flattened vector of indices with an additional offset tensor to mark the starting position of each example. You can refer to its documentation for details.

2.2 Training and Baseline Classification¶

In order to train our classifier, we need to define a collate function to batch the samples into the tensor fomat required by the embedding-bag and create the interable dataloaders.

We will then train our embedding-bag model with the common cross-entropy loss and Adam optimizer. Due to the simplicity of this task, 5 epochs should be enough to give us a stable 90% validation accuracy.

Now, let us take the following sports news and test how our model performs.

Our embedding-bag does successfully identify the above news as sports with pretty high confidence.

2.3 Inspect the model prediction with Lime¶

Finally, it is time to bring back Lime to inspect how the model makes the prediction. However, we will use the more customizable LimeBase class this time which is also the low-level implementation powering the Lime class we used before. The Lime class is opinionated when creating features from perturbed binary interpretable representations. It can only set the "absense" features to some baseline values while keeping other "presense" features. This is not what we want in this case. For text, the interpretable representation is a binary vector indicating if the word of each position is present or not. The corresponding text input should literally remove the absent words so our embedding-bag can calculate the average embeddings of the left words. Setting them to any baselines will pollute the calculation and moreover, our embedding-bag does not have common baseline tokens like <padding> at all. Therefore, we have to use LimeBase to customize the conversion logic through the from_interp_rep_transform argument.

LimeBase is not opinionated at all so we have to define every piece manually. Let us talk about them in order:

• forward_func, the forward function of the model. Notice we cannot pass our model directly since Captum always assumes the first dimension is batch while our embedding-bag requires flattened indices. So we will add the dummy dimension later when calling attribute and make a wrapper here to remove the dummy dimension before giving to our model.
• interpretable_model, the surrogate model. This works the same as we demonstrated in the above image classification example. We also use sklearn linear lasso here.
• similarity_func, the function calculating the weights for training samples. The most common distance used for texts is the cosine similarity in their latent embedding space. The text inputs are just sequences of token indices, so we have to leverage the trained embedding layer from the model to encode them to their latent vectors. Due to this extra encoding step, we cannot use the util get_exp_kernel_similarity_function('cosine') like in the image classification example, which directly calculate the cosine similarity of the given inputs.
• perturb_func, the function to sample interpretable representations. We present another way to define this argument other than using generator as shown in the above image classification example. Here we directly define a function returning a randomized sample every call. It outputs a binary vector where each token is selected independently and uniformly at random.
• perturb_interpretable_space, whether perturbed samples are in interpretable space. LimeBase also supports sampling in the original input space, but we do not need it in our case.
• from_interp_rep_transform, the function transforming the perturbed interpretable samples back to the original input space. As explained above, this argument is the main reason for us to use LimeBase. We pick the subset of the present tokens from the original text input according to the interpretable representation.
• to_interp_rep_transform, the opposite of from_interp_rep_transform. It is needed only when perturb_interpretable_space is set to false.

The attribution call is the same as the Lime class. Just remember to add the dummy batch dimension to the text input and put the offsets in the additional_forward_args because it is not a feature for the classification but a metadata for the text input.

At last, let us create a simple visualization to highlight the influential words where green stands for positive correlation and red for negative.

The above visulization should render something like the image below where the model links the "Sports" subject to many reasonable words, like "match" and "medals".