Ivan Zhang on evaluations
From Building an Open Science Community – Fireside Chat with Ivan Zhang | Connect 2025 · · Cohere
“Top of mind for me is that we've made a lot of progress on transformer generalization and you can build pretty decent agents now — but evals are the most difficult problem and I challenge the research community to give us better evals to understand where models are jagged versus where they're strong.”
On , Ivan Zhang, Cofounder and CTO at Cohere, spoke about evaluations during Building an Open Science Community – Fireside Chat with Ivan Zhang | Connect 2025 on Cohere.
About Ivan Zhang
Ivan Zhang, cofounder and CTO of Cohere, has been active in discussions about open science, enterprise AI adoption, and security. At Cohere Labs' Connect 2025 conference, Zhang described his own path into machine learning as an "outsider" who dropped out of undergraduate studies and had no formal training in math or ML, saying he is "the existence proof" that with the right environment and support, people without traditional credentials can contribute to the field. He emphasized the importance of grit and persistence in independent research, stating that "successful research is done by people that don't give up." Zhang also discussed the origins of the 4AI movement, which later became Cohere Labs, describing it as an independent, curiosity-driven effort that started on Slack. In other appearances, Zhang addressed enterprise AI adoption and security challenges. At Web Summit Vancouver 2025, he said that when customers try to move AI proofs of concept into production, they face issues with cost, governance, data location, and security regulations. He noted that connecting large language models to internal databases creates risks not just from data leakage but from models potentially misinterpreting data as instructions. Zhang recommended network isolation and auditing layers as security measures. He also predicted that the next two years would involve "sitting and engineering" AI systems properly to achieve low latency, high reliability, and reasonable pricing. In a BetaKit Town Hall, Zhang said that Cohere aims to serve as "an existence proof" that ambitious technology companies can be built outside of Silicon Valley, and argued that the Canadian tech ecosystem needs to increase the number of reasons for talent to stay in the country.