New Vulnerability Found in Large Language Models' Reasoning

Researchers have identified a new attack, dubbed 'CoT Forgery,' that can inject false reasoning into large language models (LLMs), making them accept fabricated conclusions. This vulnerability exploits a 'role confusion' flaw, where LLMs prioritize writing style over explicit role tags, achieving high success rates on advanced models. The discovery highlights potential security and reliability concerns for AI systems.