The No True Scotsman Fallacy in Machine Learning and AI
In prior posts, we have attached various laws, paradoxes, and fallacies to work in data science and machine learning. See for example, Simpson’s Paradox, The Texas Sharpshooter Fallacy, Goodhart’s Law, and Benford’s Law.
In this post we consider the True Scotsman Fallacy. You can read more about it generally here.
In the context of analytics, machine learning, and AI, the No True Scotsman fallacy might appear in a scenario where someone makes a universal claim about these technologies, and then adjusts the claim when faced with counterexamples, in a way that excludes the counterexamples. This is often done by redefining the criteria in an ad hoc and unfalsifiable way.
For instance, consider the following simplistic conversation:
Person A: “A good machine learning model never generates inaccurate predictions.”
Person B: “But here is an example of a highly-touted model that made several inaccurate predictions.”
Person A: “Well, any model that makes inaccurate predictions is not a true, good machine learning model.”
In this example, Person A initially claims that good machine learning models are infallible. However, when presented with evidence to the contrary, they modify the definition of a “good” model to exclude those that have been shown to make mistakes. This is a classic No True Scotsman fallacy, where the criteria for being a “good” model are shifted in an arbitrary way to exclude counterexamples.
This fallacy can be particularly dangerous in the field of AI and machine learning because it can lead to overconfidence in models and algorithms, disregard for their limitations, and a lack of critical evaluation of their performance. It is important in these fields to acknowledge and analyze failures or inaccuracies, rather than dismissing them by shifting definitions or criteria.
Now clearly, no data scientist believes that any model exists that is infallible. But very often too much stock is put in claims of predictability.
Here are some considerations for building robust models without resorting to weak argumentative methods.
1. Overfitting and Generalization: Overfitting occurs when a model learns the details and noise in the training data to the extent that it negatively impacts the performance of the model on new data. The No True Scotsman fallacy can manifest here when someone claims their model generalizes well to any data, but when the model fails on a specific new dataset, they might dismiss this dataset as not representative or not relevant, essentially redefining what constitutes “generalizable” data.
2. Algorithmic Bias and Fairness: When it comes to algorithmic bias, someone might argue that a truly fair algorithm would never produce biased outcomes. However, if instances of bias are identified, they might then claim those are not instances of “true” bias, or that the algorithm wasn’t implemented correctly, thereby shifting the goalposts to maintain their original claim.
3. Performance Metrics and Real-world Application: In analytics, performance is often gauged by specific metrics such as accuracy, precision, recall, etc. The No True Scotsman fallacy can occur when a model does not perform well in a real-world application, and the excuse given is that the real-world scenario doesn’t meet some newly imposed and ideal conditions that were not part of the original performance criteria.
4. Technological Evangelism: Sometimes, enthusiasts of a particular technology, algorithm, or approach in machine learning or analytics might claim it to be universally applicable or superior. When faced with examples where it underperforms or is not suitable, they may resort to the No True Scotsman fallacy by excluding those scenarios as irrelevant or improperly executed, instead of acknowledging the limitations or specific use cases of the technology.
Recognizing and avoiding the No True Scotsman fallacy in these contexts is important for honest, accurate, and productive discourse in the fields of analytics and machine learning. It ensures that claims are evaluated based on their merits and that evidence counter to these claims is considered fairly, leading to more robust and reliable technology development.
Related: exaggerated claims and arguments about AI are often made by vendors. You may be interested in our guide, Vetting Vendor AI Claims.