Peering into the Black Box: Interpretable Text Classification via Probing LLM Embeddings

In the rapidly evolving landscape of natural language processing (NLP), Large Language Models (LLMs) have emerged as the dominant architects of textual representation. By transforming raw, unstructured text into high-dimensional numerical vectors—known as embeddings—these models capture the nuanced semantic relationships between words, sentences, and paragraphs. However, this transformative power comes with a significant trade-off: the "black-box" nature of deep neural networks. As developers increasingly rely on these embeddings for downstream tasks like sentiment analysis, topic classification, and intent detection, the demand for transparency regarding how these models "think" has never been higher.

This article explores the methodology of "probing," a technique designed to illuminate the internal mechanics of LLM-generated embeddings. By leveraging the Scikit-LLM framework, UMAP visualization, and SHAP (SHapley Additive exPlanations) values, we can move beyond the surface-level performance of these models and gain a rigorous understanding of the semantic information encoded within their vector spaces.

The Evolution of Text Classification

Historically, text classification was a domain dominated by statistical machine learning models—Naive Bayes, Support Vector Machines, and Logistic Regression. These models were inherently interpretable; one could inspect the weights assigned to specific n-grams to understand why a document was categorized as "spam" or "ham."

The transition to deep learning and, more recently, to transformer-based LLMs, has significantly improved accuracy while simultaneously obscuring the decision-making process. Today, a typical pipeline involves passing raw text through an LLM to generate an embedding, which is then fed into a classifier. While this pipeline is undeniably efficient, it leaves practitioners with a haunting question: What exactly is the model learning, and which latent dimensions are driving the final prediction?

Chronology: From Raw Text to Interpretable Insight

To demystify this process, we utilize a structured approach. The following workflow demonstrates how to transition from raw data to a fully interpretable classification model using free, local resources.

1. Environment Setup

Modern interpretability research should not be gated by expensive API costs. By utilizing Ollama to host local LLMs and the Scikit-LLM library to provide a scikit-learn compatible wrapper, developers can build robust pipelines on local hardware or accessible cloud environments like Google Colab.

The installation process requires setting up the necessary dependencies: scikit-llm for LLM integration, umap-learn for manifold visualization, and shap for feature importance analysis. Once the environment is configured, we connect our local server to the GPTVectorizer, allowing us to treat an LLM as a feature engineering component in a traditional machine learning pipeline.

2. Data Preparation and Embedding Generation

For this demonstration, we utilize the classic IMDB movie review dataset. By selecting 500 positive and 500 negative reviews, we establish a balanced binary classification task. Stratified sampling ensures that our training and testing sets maintain this parity, preventing class imbalance from skewing our interpretability results.

Generating embeddings is the most computationally intensive phase. Using the all-minilm model via the GPTVectorizer, we transform 1,000 text samples into dense numerical vectors. This step is critical; it is here that the LLM condenses complex semantic information into a fixed-length vector space, which we will later probe.

Supporting Data: Assessing Embedding Quality

The effectiveness of a downstream classifier is fundamentally tied to the quality of the input embeddings. To validate that our LLM has successfully captured the semantic essence of the movie reviews, we employ a probing classifier.

The Logic of Probing

A probing classifier is a simple, high-bias model—typically a linear regression—trained on the embeddings. The intuition is simple: if a simple linear model can achieve high accuracy on these embeddings, the embeddings themselves must contain highly separable, linearly organized semantic information.

In our analysis, the logistic regression model achieved an accuracy of 77%. While not state-of-the-art for a deep transformer, it is a significant result for a linear model on such a small, complex dataset. This confirms that the LLM has successfully mapped "positive" and "negative" sentiment into distinct regions of the vector space, making them easily retrievable by subsequent algorithms.

Visualizing Manifolds with UMAP

While accuracy scores tell us that the model is working, they do not show us how. To visualize the structure of these high-dimensional embeddings, we use UMAP (Uniform Manifold Approximation and Projection). UMAP reduces the dimensionality of our embeddings from hundreds of dimensions down to two, allowing for human-readable scatterplots.

The resulting visualization shows a distinct, albeit overlapping, density of positive and negative reviews. The "southern" cluster contains a higher concentration of negative reviews, while the "northern" region leans toward positive. This visual confirmation serves as a sanity check: the LLM is not just generating noise; it is clustering semantically similar documents, which is the prerequisite for effective classification.

Official Interpretability: The SHAP Analysis

To conclude our inquiry, we turn to SHAP values, the gold standard for model explainability. SHAP treats the prediction as a cooperative game where each feature (or, in our case, each dimension of the embedding) contributes to the final output.

Understanding Latent Dimensions

In our analysis, we identified that specific latent dimensions carry vastly different weights.

• Dimension 208 and 317: These dimensions emerged as primary signals for negative sentiment. When these values are high, the probability of the review being categorized as "negative" increases significantly.
• Dimension 139: This acts as a catalyst for positive sentiment. Higher values in this dimension push the classification toward the positive class.

This level of granularity is transformative. Instead of treating the LLM as a monolithic "black box," we can now identify which latent features are most sensitive to sentiment, providing a diagnostic path for fine-tuning or prompt engineering.

Implications for Future Development

The implications of this research are twofold. First, for developers, this approach provides a roadmap for debugging LLM pipelines. If a model fails in production, practitioners can now probe the embeddings to see if the LLM is failing to capture the relevant semantic signal, or if the downstream classifier is failing to utilize that signal effectively.

Second, from a regulatory and ethical standpoint, interpretability is a prerequisite for the deployment of AI in high-stakes environments. If we cannot explain why an LLM classifies a piece of text as "harmful" or "toxic," we cannot trust it in clinical, legal, or financial applications.

By integrating probing, UMAP visualization, and SHAP analysis into the development cycle, the industry can move toward a "Glass Box" paradigm. We are no longer limited to merely observing the outputs of LLMs; we can now peer into their internal representations, validating their reasoning and ensuring that the models align with our intended goals. As LLMs become more integrated into the fabric of digital infrastructure, this commitment to transparency will define the next generation of responsible and effective AI development.