Barassi, V. (2024).
Harvard Data Science Review, Special Issue 5.
Abstract
The rise of generative AI confronts us with new and key questions about AI failure, and how we make sense of and learn how to coexist with it. While computer scientists understand AI failure as something that we can learn from and predict, in this article I argue that we need to understand AI failure as a complex social reality that is defined by the interconnection between our data, technological design, and structural inequalities by processes of commodification and by everyday political and social conflicts. Yet I also show that to make sense of the complexity of AI failure we need a theory of AI errors. Bringing philosophical approaches to error theory together with anthropological perspectives, I argue that a theory of error is essential because it sheds light on the fact that the failures in our systems derive from processes of erroneous knowledge production, from mischaracterizations and flawed cognitive relations. A theory of AI errors, therefore, ultimately confronts us with the question about what types of cognitive relations and judgments define our AI systems, and sheds light on their deep-seeded limitations when it comes to making sense of our social worlds and human life.
Here are some thoughts:
As generative AI technologies continue to advance, they bring remarkable capabilities alongside significant challenges. Veronica Barassi’s work delves into these challenges by critically examining AI "hallucinations" and errors. Her analysis emphasizes the need for a foundational theory to understand and address the societal implications of these phenomena.
AI systems often generate outputs that are factually incorrect or nonsensical, commonly referred to as "hallucinations." These failures arise from the design of AI models, which prioritize persuasive, realistic outputs over factual accuracy. Barassi contends that these errors are not merely isolated technical glitches but the product of erroneous knowledge production influenced by biases and flawed cognitive patterns inherent in the systems. Furthermore, describing these inaccuracies as hallucinations risks anthropomorphizing AI, attributing human-like cognitive capabilities to machines that operate solely on probabilistic algorithms. This misrepresentation can distort ethical evaluations and public understanding of AI's true capabilities and limitations.
One major concern highlighted is the structural homogenization of AI systems. Foundation models, which underpin many generative AI technologies, rely on vast datasets and self-supervised learning. While this approach enables scalability and adaptability, it also amplifies systemic flaws, making AI vulnerable to cascading failures. These failures, when integrated into critical systems such as infrastructure or healthcare, could have catastrophic societal consequences.
Barassi also underscores the social dimensions of AI errors, which often reflect deeper societal issues. Embedded biases in data and technology perpetuate structural inequalities, disproportionately impacting marginalized groups. She argues that AI failures cannot be seen as mere technical bugs but must be understood as sociotechnical constructs shaped by interactions between humans and machines. Addressing these challenges requires a robust theoretical framework combining philosophical error theories and anthropological insights. This perspective shifts the focus from technical fixes to understanding AI errors as reflections of our cultural, social, and epistemological landscapes.
Another critical issue is the cultural and linguistic bias inherent in many AI systems. Language models primarily reflect English-speaking, Western contexts, marginalizing the diversity of human experiences. This lack of inclusivity highlights the need for broader, more representative datasets and culturally sensitive approaches to AI development.
Barassi’s analysis calls for a paradigm shift in how we approach AI failures. Rather than viewing errors as problems to be eliminated, she advocates for accepting their inevitability and critically engaging with them. Policymakers, developers, and users must prioritize transparency, accountability, and inclusivity to mitigate AI's societal risks. By fostering interdisciplinary collaboration and embracing the complexity of AI systems, society can better navigate the challenges and opportunities presented by this rapidly evolving technology. This approach emphasizes coexistence with AI’s limitations while fostering ethical development and deployment practices.
