Protege, the data and evaluation platform for AI in healthcare, today announced the launch of independent testing to measure how well healthcare AI models perform on real patient cases they have never seen before. The company is introducing its first benchmark, Disease Odyssey, alongside work with healthcare AI companies Xlue AI and Standard Model Biomedicine to test models that could help doctors identify serious illnesses, predict disease risks and better understand cancer outcomes.

AI has the potential to transform healthcare, but as more hospitals and doctors use AI tools, an important question remains: How do we know they actually work? Today, many AI companies test their own models, sometimes using information the models may have already seen during training. As a result, test scores may not reflect how well AI performs with real patients, making it harder for healthcare organizations to know which tools to trust.

The stakes are high and the need for reliable testing is growing. 81% of physicians now use AI in their daily work, yet testing has not kept pace with its growing use. Meanwhile, 91% of healthcare organizations report that they cannot verify how AI is being managed and monitored within their organizations. Independent testing can help hospitals and doctors better understand how AI performs and make more informed decisions about which tools to use.

“Healthcare AI is moving quickly, but the way we evaluate it hasn’t kept pace,” said Bobby Samuels, co-founder and CEO of Protege. “When AI builders test their own models on data they control, healthcare organizations are being asked to trust performance without an independent measure of whether it will hold up against real-world patient scenarios. We think healthcare AI needs a credible third party that can provide that evidence.”

Protege aims to close that gap by testing AI models with evaluations that are built using de-identified and privacy-protected real-world patient records and are also “double-blind:” model builders do not see the de-identified patient data or test questions in advance, and Protege evaluates the model without knowing how it was built. Unlike traditional evaluations, Protege independently designs, runs, and scores the evaluations, giving healthcare AI builders a clearer picture of where their models work and where they need improvement.

The company’s first benchmark, Disease Odyssey, tests whether AI can recognize signs of a serious illness before a patient is diagnosed. For example, a patient may have visited multiple doctors over several months without getting a clear answer. Disease Odyssey tests whether AI can connect clues across those visits to identify the underlying condition. In initial testing, leading AI models correctly identified conditions such as sleep apnea and anemia in approximately 51% to 67% of cases, highlighting how much room for improvement remains.

“Healthcare AI models should be independently measured before it enters the care room or admin office,” said Engy Ziedan, Ph.D., co-founder and Chief Scientific Officer at Protege. “That means continuously testing models on questions that reflect real clinical workflows, with clinician-verified answer keys, using net-new patient data that accurately reflects what happens in real healthcare settings and can be refreshed with each model update.”

Protege is also working with healthcare AI companies to put independent testing into practice:

  • Standard Model Biomedicine: Protege partnered with the company to test its AI model’s ability to predict how long patients with breast or lung cancer might survive and when they may need another treatment. The model performed competitively with leading AI systems at a fraction of the cost — between 36 and 170 times less per prediction — demonstrating the potential for more affordable tools to support cancer research and clinical trial planning.

  • Xlue AI: Protege independently tested HealthSAW, Xlue’s AI model designed to predict how a patient’s health may change over time. Evaluated on unseen patients, HealthSAW outperformed frontier models on 14 of 16 disease modeling tasks – predicting new onsets of cancer, heart disease and neurological conditions – by at least 10% absolute as measured by sensitivity at 95% specificity. Additionally, HealthSAW is able to screen these patients at less than 0.1% of the cost per prediction. These capabilities could eventually help doctors identify at-risk patients earlier and help researchers better select participants for clinical trials.

“We were pleased to see the Standard Model outcompete frontier model performance, with an order of magnitude decrease in operational costs,” said Kevin Brown, co-founder and CEO of Standard Model Biomedicine. “Protege’s double-blind evaluation shows that a compact, specialized model can beat the world’s largest frontier models on real patient histories at significantly lower costs. We’re excited to have early access to Protege’s benchmarking platform, as its double-blind evaluation offers the kind of rigorous comparison the industry has been waiting for.”

“Our ambition is to build models that understand how a patient’s health evolves and help anticipate what comes next and how patients react to treatments,” said Chenyan Xiong, co-founder and CEO of Xlue AI. “Protege gives us something essential to that work: an independent test on patient histories our model has never seen.”

Protege plans to expand its evaluations into additional areas, including pathology, dermatology, oncology and healthcare administration. Healthcare AI builders interested in independently evaluating their models or applications, as well as enterprises looking for independent measurement of healthcare AI before deployment can contact Protege here.

About Protege:

Protege is the data and evaluation platform for AI in healthcare. Protege combines high-quality healthcare data access across modalities with the scientific research expertise an rigor required to make raw data ready for AI development.

Founded in 2024, Protege has raised $65 million from investors including Andreessen Horowitz, Footwork, and CRV and works with the majority of the Magnificent Seven and leading AI labs. Learn more at www.withprotege.ai.

Media gallery

About The Author
Business Wire