保留正文与补采附件。源页面当前未重新核验；具体限制见交付清单。

Revisit consent button We value your privacy We use cookies to enhance your browsing experience, serve personalised ads or content, and analyse our traffic. By clicking "Accept All", you consent to our use of cookies. Customise Reject All Accept All Customise Consent Preferences We use cookies to help you navigate efficiently and perform certain functions. You will find detailed information about all cookies under each consent category below. The cookies that are categorised as "Necessary" are stored on your browser as they are essential for enabling the basic functionalities of the site. ... Show more NecessaryAlways Active Necessary cookies are required to enable the basic features of this site, such as providing secure log-in or adjusting your consent preferences. These cookies do not store any personally identifiable data.

Functional Functional cookies help perform certain functionalities like sharing the content of the website on social media platforms, collecting feedback, and other third-party features.

Analytics Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics such as the number of visitors, bounce rate, traffic source, etc.

Performance Performance cookies are used to understand and analyse the key performance indexes of the website which helps in delivering a better user experience for the visitors.

Advertisement Advertisement cookies are used to provide visitors with customised advertisements based on the pages you visited previously and to analyse the effectiveness of the ad campaigns.

Uncategorised Other uncategorised cookies are those that are being analysed and have not been classified into a category as yet. No cookies to display. Reject All Save My Preferences Accept All Toggle SearchToggle Mobile Navigation Publish AI, ML & data-science insights to a global community of data professionals. Sign inSubmit an Article LatestEditor's PicksNewsletterShipAI

Write For TDS LinkedInXToggle Search Artificial Intelligence

Explainable AI and the Focus on Human Curiosity

and concentrating on the top few factors and counterfactuals Tirthajyoti Sarkar January 24, 20228 min read

Explainable AI is hard

Explainable AI - or "xAI" as it is called sometimes - is hard - both figuratively and computationally. Figuratively because the scope, purpose, and effectiveness of explainability are often hard to anticipate precisely and formalize in a problem definition or a user case study. These attributes take on a fluid form and spread out rather quickly and unpredictably. The computational challenge of xAI comes from the mechanics of calculating metrics like Shapley values (for a large number of possible combinations of input features even if not for the entire power set). This is primarily a side effect of the fact that the most successful - and, therefore, important to offer explainability - machine learning models tend to have a large parameter space and deal with a bagful of features (or high-dimensional data type like images or natural language). There are a good number of high-quality articles written about these. Exciting startups are working hard to address many of these problems. In this article, we just want to focus on one aspect - how anticipating the user curiosity may help guide the xAI system design effort. Learn this step by step with the interactive AI Engineer, AI and Data Scientist, Computer Science and Machine Learning roadmaps.

Two definitions - one to avoid

There is a landmark paper in this domain by Tim Miller: "Explanation in Artificial Intelligence: Insights from the Social Sciences" (2017). It has, sort of, two (non-mathematical) definitions for "Machine Learning Interpretability". One definition is - Interpretability is the degree to which a human can understand the cause of a decision. Another goes by - Interpretability is the degree to which a human can consistently predict the model's result.

Depending on human prediction is tricky

I would argue that the second definition should be avoided, even if it is straightforward and tempting to follow. Why tempting to follow? Because between the two, this one has clearer mechanics for measurement. And, measurement is important for designing engineering systems. One can test the interpretability of an ML system by employing human evaluators and measure how consistently they can predict the model's results. This also parallels the famous "Human-level performance " benchmark that all deep learning models and their designers talk about. Please accept cookies to access this content But, let's not overlook the prickly fact that depending on the human mind for consistently predicting structured results - essentially mimicking a sophisticated computer program - is not always a wise choice. That's why this definition of ML interpretability should be avoided. Let me show you an example - right here in this article.

A demonstration

I wrote an article about building a single utility function to train a complete convolutional neural net model from just one input of the training images storage location.

A single function to streamline image classification with Keras Here is the notebook. One of the examples was about classification between "cup" or "crab" from the famous Caltech-101 dataset. Let me ask you - the reader - to classify the test images. What is it? A cup or a crab? How much time did it take you to classify? Less than 100 milliseconds, maybe? What about this one? Again, instant recognition, isn't it? Now, what about this one? You had a chuckle, didn't you? "A tricky one for the machine, " you thought silently. So, as a human evaluator, you did pretty well. But, comparing your performance to an ML algorithm - purely from an interpretability point of view - will be very hard in this exercise. Because a 5-layer convolutional neural net is sort of a black box whose inner working will be hard to explain to anyone. You will never be satisfied with the comparison and reconcile with the fact that your thinking is in the same line as that of the ML model. You just won't know what that 'line of thinking for the ML model is! Now, consider this example. Let's say you have built a logistic regression model of a chemical process for your manufacturing plant to predict the quality - good or bad - based on input process parameters (measured by sophisticated sensors). Here is a snapshot of the test data (think of it as parallel to the test images you saw above). I have hidden the "Target" column partially because that's what you will not see but will be evaluated on. What's your chance of predicting the right targets on this one looking at those maze of numbers? What's the chance that you can rattle off correct predictions under 100 milliseconds? The grand scheme of using human evaluators as the metric of ML interpretability will fail spectacularly for this model. The irony, however, is that the choice of a simple binary logistic classification is perfect from an interpretability point of view. It is, indeed, an interpretable model. Just not the way we pitted a human evaluator against it.

Human brains have not evolved to deal with tabular data

Tabular data, in a nicely written form, is a recent development in human history. Sumerian or Assyrian cuneiforms were some of the earliest examples of written scripts and they were developed out of the need for record-keeping - land sales, agriculture output, tax revenue. A bronze age version of Microsoft Excel! But, it's only the bronze age. Just around the agricultural revolution, whose impact on food production, population density, and urbanization, led to this kind of tabular dataset. Human cognitive revolution has been taking place for a long time - much before that. Here is an approximate timeline. And, this chart above does not even cover the timeline of the evolution of the most amazing visual cortex and the human eye. That biology was shaped by the need to survive in the African savannah - classifying a predator from prey, a safe berry to eat from a poisonous one. Hans Moravec, an Austria-born American roboticist and computer scientist, and presently, an adjunct faculty member of the Robotics Institute of Carnegie Mellon University, put it in a slightly different way - "it is comparatively easy to make computers exhibit adult level performance on intelligence tests or playing checkers, and difficult or impossible to give them the skills of a one-year-old when it comes to perception and mobility ".

Why math is easy for AI but gardening is not: Moravec's paradox All this is to say - human evaluators will do fine with an object classification work but fail miserably with a binary classification using a tabular dataset. Although the common ML models used for these tasks will be exactly opposite from the interpretability point of view - a hard-to-interpret neural net and a relatively easier logistic regression.

Go with the curiosity

For interpretable AI/ML systems, therefore, it makes sense, not to pit humans against the machine to see if the system predictions align with the human-generated ones. Instead, those systems need to be designed - from the ground up - by looking at the variety and nature of the natural curiosity of human users.

The Why's and the "What-if's

Human thinking needs to be respected. This include - "why " and "what if ".

Top-3 or Top-4 is sufficient and advisable

The top-3 (or 4) factor is important to note. A well-designed interpretable ML system should not try to overwhelm the user with all the strong patterns in the data that led to the model prediction. This is, again, intimately related to our biology. We are good only with a small set of numbers. The short-term working memory of the human brain is around 7–8 items/numbers. Therefore, a good system will focus on showing only a few visual concepts or charts for explaining an ML model. UX/UI design assumes a critical role. This goes for general data analytics problems too. Here is an example from my article. For a correlation problem involving multiple similar machines, build a simple bar chart clearly highlighting the outlier machines instead of showing the actual correlation plots.

Counterfactuals

The what-if questions are often non-trivial and involve uncovering scenarios that had not happened but could have had. If we have a given model, you may think this is easy to do - just throw an exhaustive combination of input to the model, note down all the responses, and present them nicely. But who decides that combination? How to assign priority to the list of stimuli that the model will be tested with for the first time? Here again, user (human) curiosity will be the key to deciding the priority and filtering the list. Additionally, we have to be careful about not testing the model with scenarios that are not too wildly different than the distribution of the training cases it has seen.

Curiosity changes with the domain

It is critical to remember that curiosity changes with the domain and level of the user within that domain. For a manufacturing organization, various stakeholders will be curious about various aspects of the ML model predictions, Therefore, a good interpretable ML system must strive for covering as much base as possible with well-proven common design patterns e.g. visual analytics that has been proven to be effective (for their explanation power) for a large cross-section of users.

The explainability may come from a different layer

Human curiosity knows no bounds and it is prone to cross boundaries in a fluid manner. Most ML-powered systems are not just a single ML box. More often than not, they have other layers wrapping around the ML model and external users only see the recommendation or suggestions from this outer layer. The human curiosity will be drawn to ask questions more pertaining to the structure of this outer layer. Interpretable ML systems should be ready to tackle this as they have to correctly identify the sub-system that the user's curiosity has tapped into.

Summary

Explainable AI is a great field that will drive the adoption of AI/ML systems as true aids to human endeavors. To succeed, apart from the design and computational challenges, anticipating the variety and subtlety of human curiosity will be key. Domain knowledge and an experimental attitude (to design the most appealing UI for example) will also be key ingredients for success.

Citation for Caltech-101 dataset

L. Fei-Fei, R. Fergus and P. Perona. "Learning generative visual models from few training examples: an incremental Bayesian approach tested on 101 object categories" IEEE. CVPR 2004, Workshop on Generative-Model Based Vision. 2004.

Loved the article? Become a Medium member to continue learning without limits . I'll receive a portion of your membership fee if you use the following link,__with no extra cost to you .

Join Medium with my referral link - Tirthajyoti Sarkar

Register Now to Continue Reading

Unlock Towards Data Science with a Free Account

All articles on Towards Data Science are available to registered users. Create a free account today to access this article and thousands more. Register for freeSign in to your account

Written By

Tirthajyoti Sarkar

See all from Tirthajyoti Sarkar Artificial IntelligenceData ScienceExplainable AiMachine LearningTechnology Share This Article

Towards Data Science is a community publication. Submit your insights to reach our global audience and earn through the TDS Author Payment Program. Write for TDS

Related Articles

What Is the Minimum Sample Size Required to Perform a Meaningful Linear Regression?

Data Science Thoughts and Theory, Machine Learning A quantitative approach based on estimation accuracy What is the... Gabriel de Longeaux June 4, 20216 min read

Conducting Federated Learning Research with FedJAX

Machine Learning Ten or fewer takeaways on using FedJAX for federated learning research Andrei November 6, 20216 min read

Matrix calculus for data scientists

Machine Learning Take the red pill and learn about matrix calculus! Vincent Mueller September 6, 20216 min read

Model Overload - Which NLP Model Should I Choose?

Artificial Intelligence How to evaluate and compare Huggingface NLP models with Sagemaker Experiments Heiko Hotz July 7, 20213 min read

Evolutionary Algorithm - Mutations Explained

Machine Learning Illustrations with code implementation, applied to TSP as an example James Koh, PhD November 7, 20238 min read

Convert a Tensorflow2 model to OpenVINO®

Machine Learning A brief guide to run your model everywhere Application of machine learning are, nowadays, endless.... Filippo Valle May 5, 20214 min read

How to Transform Time Series for Deep Learning

Artificial Intelligence Forecasting with deep neural networks Vitor Cerqueira February 14, 20236 min read

The Marketing Reporting Gap

Data Science Why do marketers turn to spreadsheets and how to close this gap João António Sousa January 20, 20245 min read

A supervised or semi-supervised ULMFit model to Twitter US Airlines Sentiment Dataset

Machine Learning Our task is to apply a supervised/semi-supervised technique like ULMFit (Ruder et al, 2018) to the Twitter US Airlines sentiment analysis... Aadit Kapoor August 15, 20209 min read

Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals. Subscribe to Our NewsletterWrite For TDS•About•Advertise•Privacy Policy•Terms of Use © Insight Media Group, LLC 2026•The New Stack•roadmap.sh•Creator Network