Kimi K3.1 Pro shows high results in benchmarks, but its real-world performance splits observers. In an episode of the Interconnects podcast, Nathan Lambert and Florian Brand revisit this launch, the cybersecurity risks associated with AI models and the arrival of agents capable of handling personal tasks, notably Muse.
Kimi K3.1 Pro: promising benchmarks, but reservations
On Artificial Analysis, Kimi K3.1 Pro ranks among the best‑evaluated models, ahead of GPT‑6 Astra according to Nathan Lambert. Other tests, however, paint a different picture. On Terminal‑Bench Science, it notes a 20‑point gap in favor of GPT‑6 Astra.
Florian Brand offers an explanation: the Artificial Analysis evaluations cover relatively short tasks, unlike ProgramBench, which demands more exploration. According to him, Kimi’s reinforcement training may favor exploiting known solutions rather than seeking new approaches.
The model size also raises questions. With its three trillion parameters, Kimi K3.1 Pro requires, according to Florian Brand, B300 GPUs to host a replica on a single node. Nathan Lambert notes that the publisher announced the model weights would be released by 4 November.
Cybersecurity: are open models really more dangerous?
Kimi states it has deliberately limited K3.1 Pro’s offensive capabilities in cybersecurity, notably by adjusting its training data. The publisher aims to preserve defensive uses without pushing the boundaries of attack capabilities available with open‑weight models.
This decision comes amid a series of vulnerability disclosures. Florian Brand cites, among others, a PHP flaw that enables remote code execution, reported as part of the Patch the Planet by OpenAI program, alongside other security researchers.
Both speakers, however, stress that a rise in discovered vulnerabilities does not necessarily translate into a surge of attacks. They also mention the compromise of LiteLLM, whose preliminary report attributes the incident to an API key left exposed on GitHub for five months.
Muse: an AI agent that organizes procedures and purchases
Florian Brand finally shares his experience with Muse since its deployment in the United States. The agent helps him prepare administrative procedures, locate documents and organize his purchases. It can build a cart and then offer payment via a Stripe link using a single‑use credit card.
The experience, however, has limitations. Florian Brand reports slowdowns, browser crashes and blocks imposed by certain sites. He also explains that he moved his emails to Gmail to make them easier for Muse to access, while preferring to take back control personally when the agent needs to communicate with a human interlocutor.
