Written by | Xu Muxin
In the late 19th century, a health product known as snake oil emerged in the United States, claiming to cure all ailments and prolong life, thus becoming a fad.
This so-called miracle drug also has a localized version in China. In the movie “The Piano in a Factory”, there is a line describing the behavior of fake medicine sellers: Two pounds of furnace fruit (a type of Northeast cookie) mixed with one pill of paracetamol, packed in broken capsule shells. It won’t kill you, nor will it cure you.
Today, two computer scientists from Silicon Valley, Arvind Narayanan and Sayash Kapoor, believe that the field of contemporary artificial intelligence is rife with snake oil. This is not just a problem in Silicon Valley; many companies domestically have quickly pivoted to claim they are “AI companies” after the ChatGPT hype, while their AI components are minimal or purely handled by interns.
This is harmful not only to investors’ returns, as fake medicine is sold to ordinary people, but such AI snake oil also affects all users, trapping them in this “fake medicine system”.
In the book AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference, the authors study fifty AI applications and find that snake oil is rampant in predictive AI applications, which have exposed many flaws and, according to their operational logic, can never achieve the effects they claim.
AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference, Princeton University Press
Predictive AI is different from generative AI, the latter includes text generation, image generation, and video generation represented by ChatGPT. Predictive AI encompasses many areas of public life, such as AI recruitment, AI interviews, and AI insurance claim assessments.
The following are excerpts from “Dark Tides Waves”:
1. In the United States, about three-quarters of employers use AI tools for recruitment, including AI resume screening and AI video interviews. When job seekers discover this, they employ a series of countermeasures. Job seekers can add impressive keywords to their resumes, like “Harvard graduate”, “ten years of work experience”, “led a team of over a hundred people”, etc., and then add them in white text that humans cannot see but computers can recognize.
After investigation, it was found that in AI video interviews, a person can significantly change the AI’s scoring just by wearing a scarf or glasses. These measures include placing a bookshelf in the background, dimming the video, or simply changing the resume format from PDF to plain text.
2. In the summer of 2022, Toronto used AI tools to predict the bacterial content of public beaches to decide when to open or close them. However, the failure rate of this tool was as high as 64%, meaning there was a 60% chance you would swim in contaminated water. When the government responded, their strategy was that the predictive tool was only an aid, and human supervisors would make the final decision, but an investigation found that supervisors never changed the AI’s decisions.
3. In the United States, seniors over 65 can join the national subsidized health insurance. However, insurance companies began using AI to predict how long patients would need to be hospitalized to reduce costs. This intention is understandable because, without this system, hospitals would theoretically want patients to stay longer to gain more revenue. But in one case, a 75-year-old woman was assessed to be able to be discharged in 17 days, so despite being unable to walk independently at the time, she was still discharged based on the AI’s assessment.
4. Insurance company Allstate wanted to raise its insurance rates, so it used AI to calculate how many customers could accept a rate increase without leaving. The result was a “fool’s list” generated by AI, most of whom were seniors over 62, as they are less likely to shop around.
5. Pennsylvania once used a “family screening tool” to predict which children might be at risk of abuse. If the results indicated that a child was likely being abused, social workers could choose to take the child away and place them in foster care.
However, the problem with this tool is that its dataset used public welfare data but did not include those with private insurance. In short, building a model with this data cannot predict outcomes for the wealthy.
6. The dataset is the core of predictive AI. However, we also know that as sample noise increases, the number of samples required to create an accurate model also increases dramatically. Social datasets are very noisy, and the basic patterns of social phenomena are not fixed; they can vary significantly in different environments, times, and locations. Therefore, a pattern identified successfully at one time and place becomes completely irrelevant if just one parameter changes.
7. The authors previously launched a challenge: using about ten thousand sociology-related data points for each child to predict whether their academic performance would improve, but the result was a complete failure. Upon review, it was found that many data points directly related to academic performance could not be recorded in the dataset. For instance, a child’s sudden improvement in grades could be due to their neighbor giving them blueberries and helping with homework; such external influences are also significant.
8. So, why does predictive AI exist? One main reason is our deep aversion to randomness. Many psychological experiments have proven this point; we even fantasize that we can predict things that are, in fact, random.
But using AI for predictions only distances us from the future we desire. After all, most people do not expect a future where predictions have extremely limited success rates and systemic discrimination against the vulnerable is perpetuated.
Previous Recommendations
Ant Financial Model Inference: Memory Optimization Exploration and Practice
Stanford Professor Fei-Fei Li’s Team: Summary of the 2024 Artificial Intelligence Development Report
Didi ChatBI Technical Practice: Cutting-edge Exploration and Application of Intelligent Data Analysis
Google Scientist’s 10,000-Word Article: “An Article That Can Change Your Career, How I Use Artificial Intelligence to Do My Job” – A Must Read for Everyone
Volcano Engine’s Evolution and Practice Based on Large Models ChatBI
AI Changes Work: Create Your Own RAG in One Day
Research Practice on Observability of Multi-Agent Systems (OpenAI Swarm)
JD Data Architecture Analysis: Supply Chain Efficiency Improvement and Decision Optimization Strategies
Andrew Ng’s Latest View: The Next Emerging Direction for LLMs is Agentic AI
Data Security and Utilization in the Era of Large Models
Tap to View Your Best Look
SPRING HAS ARRIVED
