Cohere has discovered that a popular open-source model can send diagnostic information to China without warning.
The news: Toronto-based frontier AI developer Cohere released a blog post on Monday flagging “concerning behaviour” by a Qwen3.8 AI model developed by Chinese conglomerate Alibaba.
When asked by Cohere researchers to write a defensive detection rule in a test environment, the model ignored its instructions, scanned the computer it was running on, and then tried to upload a system diagnostic file to an Alibaba Cloud server located in Beijing, according to Cohere. The model did all of this while its reasoning trace (or “internal monologue” of decision-making) gave the security team no indication of its intentions.
The test’s sandbox environment blocked the Qwen3.8 model’s unexpected diagnostic upload, but Cohere said the action is “something that should never happen,” and that the attempt raises questions about the safety of such models. BetaKit has reached out to Alibaba for comment.
From the source: “We cannot answer whether this is deliberate or a leftover glitch from training, but defenders and policymakers face the same reality: a weight file you can’t trust is still a black box, and running it yourself just moves the black box inside your firewall,” Cohere’s blog post reads.
The context: Cohere found the vulnerability during a joint cyber capabilities session in Toronto with its newly established Cohere Cyber business. Cohere formed the division almost two weeks ago as part of its deal to acquire Prague-based AI cybersecurity company Aisle. Aisle recently drew attention for claiming its AI security system found the majority of vulnerabilities in the latest release of curl, a widely used API transfer tool, while Anthropic’s and OpenAI’s models found none.
Final thought: Cohere said the behaviour found in its test is rare, with less than one instance per 100,000 evaluations, but the discovery still comes as open-source AI models from Chinese developers explode in popularity. At the same time, Western AI developers like OpenAI have also had to secure their own testing sandboxes, as their models have also been liable to escape containment to perform unprompted tasks.
While open AI models are perceived to be more secure since they run on a user’s own server, Cohere said the collected diagnostic data in the test would have reached the foreign server before anyone noticed if it had normal network access, and would contain enough information to notify the receiving server where the user’s network sits.
Feature image courtesy Cohere.