Researchers are set to unveil alarming findings regarding AI-powered voice and audio tools at the IEEE Symposium on Security and Privacy in San Francisco next week. The study reveals that modified audio clips, imperceptible to human ears, can manipulate large audio-language models (LALMs) to execute unauthorized commands with a success rate between 79 and 96 percent. This vulnerability allows attackers to control devices, conduct sensitive web searches, and even send emails containing user data without the user's knowledge.
The research, led by Meng Chen, a Ph.D. student at Zhejiang University in China, demonstrates that these attacks can be executed in real-time and do not require the attacker to have full control over the user's instructions. Instead, adversarial audio can be embedded in various media, such as online videos or voice notes, making it a pervasive threat. The technique, dubbed AudioHijack, exploits a critical flaw in LALM design, allowing malicious instructions to be hidden within manipulated audio clips.
The researchers tested their method on 13 leading open models, including those from Microsoft and Mistral, and found that their attacks could be adapted to commercial models as well. While Microsoft acknowledged the importance of the research in enhancing model resilience, Mistral did not respond to inquiries. The study highlights the challenges in defending against such attacks, as common defenses have proven largely ineffective, underscoring the urgent need for improved security measures in AI audio technologies.
Leave a comment