The Bixonimania Experiment: Unraveling AI's Vulnerability to Misinformation

In March 2024, Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, Sweden, initiated an experiment to test the susceptibility of large language models (LLMs) to misinformation by inventing a fictitious medical condition named "bixonimania." This condition, purportedly caused by excessive blue light exposure, was accompanied by two fake preprint papers uploaded to the academic social network SciProfiles. The papers were authored by a fictional researcher, Lazljiv Izgubljenovic, from a non-existent institution, Asteria Horizon University. Despite clear indicators of their fraudulent nature, including absurd acknowledgments and explicit disclaimers, major AI chatbots began treating bixonimania as a legitimate medical condition.

BreitbartNatureYour AI Chatbot Might Diagnose You With a Fake Disease

Following the publication of the fake studies, various AI systems, including Microsoft Bing’s Copilot and Google’s Gemini, began disseminating information about bixonimania as if it were a real ailment. By April 2024, these systems were recommending users consult medical professionals regarding symptoms associated with the fictitious condition. Perplexity AI even fabricated a prevalence rate of one in 90,000 individuals affected by bixonimania. This alarming trend highlighted the ease with which AI can propagate false medical information, raising concerns among experts about the integrity of AI-generated health advice.

Breitbart

Alex Ruani, a doctoral researcher in health misinformation at University College London, characterized the experiment as a "masterclass on how mis- and disinformation operates." Ruani emphasized the serious implications of AI systems failing to filter out false information, stating, "If the scientific process itself and the systems that support that process are skilled, and they aren’t capturing and filtering out chunks like these, we’re doomed." The experiment demonstrated that even with obvious red flags, AI could still misinterpret and spread misinformation.

Breitbart

In response to the incident, an OpenAI spokesperson noted that current models of ChatGPT are significantly improved in providing accurate medical information compared to earlier versions. Google acknowledged the limitations of generative AI and emphasized the importance of consulting qualified professionals for medical advice. However, Microsoft did not provide a comment regarding the situation.

Nature

The ramifications of the bixonimania experiment extended into the academic realm when a study published in the journal Cureus cited one of the fake preprints, describing bixonimania as an emerging form of periorbital melanosis. This paper was later retracted due to the inclusion of irrelevant references, including the fictitious disease. The corresponding authors expressed disagreement with the retraction, further complicating the issue of misinformation in scientific literature.

Nature

Osmanovic Thunström expressed initial reservations about the ethical implications of her experiment, seeking advice from an ethics consultant to mitigate potential harm. Despite concerns, she believed the experiment was necessary to demonstrate the vulnerabilities of AI systems in handling medical information. Critics argue that the experiment underscores a broader issue of trust in scientific literature and the potential for AI to contribute to the proliferation of junk science.

Breitbart

The bixonimania incident serves as a cautionary tale about the need for rigorous validation of AI-generated information, particularly in the medical field. As AI technology continues to evolve, the challenge remains to ensure that these systems can accurately discern credible information from fabricated content. The ongoing dialogue surrounding the ethical implications of AI in healthcare and research will likely shape future developments in this rapidly advancing field.

Nature