Newshot

OpenAI Launches MedChemBench Benchmark to Evaluate AI-Assisted Drug Discovery Capabilities

AI summary

OpenAI released the MedChemBench benchmark on November 20 to assess AI models' reasoning in medicinal chemistry research, including identifying molecular structures, interpreting experimental results, comparing compounds, and predicting their properties. OpenAI said models perform strongly in reconstructing molecular structures and explaining experimental evidence, but still have clear room for improvement in compound screening and numerical property prediction.

Translation(model translation, not the source's own text)

OpenAI Launches MedChemBench Benchmark to Evaluate AI-Assisted Drug Discovery Capabilities

IT Home, November 21 — On November 20 local time, OpenAI released the MedChemBench benchmark to evaluate the reasoning capabilities of AI models in medicinal chemistry research, including identifying molecular structures, interpreting experimental results, comparing compounds, and predicting their properties. OpenAI said models perform relatively well at reconstructing molecular structures and explaining experimental evidence, but there is still clear room for improvement in compound screening and numerical property prediction.

According to the official introduction obtained by IT Home, the benchmark contains 99 questions drawn from 117 papers and patents, involving 761 distinct molecular structures. Most questions provide the model with context at a certain point in the research, including the biological target, the researchers' goals, previously studied compounds, and some experimental results.

Officials divide the evaluation into four parts, with details as follows from IT Home:

  • Molecular structure: assembling molecules based on the structure diagrams and chemical groups given in the question, testing whether atom connectivity and stereochemistry are correct;
  • Scientific explanation: explanations must be supported by the given evidence, for example why one compound was chosen among several;
  • Ranking and selection: whether compound ranking and final selection meet the requirements of the question;
  • Property prediction: whether predictions match the numerical values and categories given in the original material, covering both measured and calculated molecular properties.

According to officially published information, at the xhigh reasoning level, GPT-6.2 Astra scored 84.5% usable score, GPT-6 Astra scored 83.3%, GPT-6.1 Sol scored 82.0%, and GPT-5.6 Sol scored 58.2%. Officials said raising the reasoning level brings score improvements, but gaps between models remain pronounced.

Molecular reconstruction comparison: GPT-6 Astra preserved the correct 18-atom ring, GPT-5.6 Sol generated a smaller ring, and GPT-5.6 Luna generated disconnected fragments.
Molecular reconstruction comparison: GPT-6 Astra preserved the correct 18-atom ring, GPT-5.6 Sol generated a smaller ring, and GPT-5.6 Luna generated disconnected fragments.

In terms of sub-item performance, Astra achieved a 95.4% usable score on the molecular structure section at the xhigh level, while scoring only about half on compound selection and numerical property prediction.

Officials cited an example in which a question asked the model to assemble a molecule from a given central structure and two side groups while preserving an 18-atom ring.

At the xhigh level shared by 11 models, all ten of GPT-6 Astra's responses were correct; GPT-5.6 Sol and GPT-5.6 Luna were wrong on all ten, with the former assembling a ring of only 15 atoms and the latter splitting the molecule into two disconnected fragments.

Another question was drawn from research on PRMT5 inhibitors; PRMT5 is a protein of interest in oncology drug development. The model needed to predict how strongly three related compounds block PRMT5 inside cells, how easily they cross cell barriers, and how quickly they are degraded by liver enzymes in laboratory assays.

Follow-up experiments omitted from the question by officials revealed trade-offs: compound 23 could block PRMT5 inside cells at lower concentrations, while compound 24 crossed cell barriers more easily and was degraded more slowly in the liver enzyme assay.

Officials said these results cannot determine whether models can discover new drugs or improve laboratory decisions. The possibility of prior exposure to published materials cannot be ruled out, and each response is a single attempt with no tools, no feedback, or new experiments. Although the benchmark is based on historical evidence rather than real drug discovery, its results help identify where models are making progress and where challenges remain. The team believes MedChemBench will support continued progress and looks forward to sharing updates as models advance and the application field develops.

Related Articles

  • OpenAI announces Spaces collaborative workspace open to ChatGPT Plus users, supporting multi-user co-editing
  • OpenAI invites some Plus users to try the dots AI agent, with mobile ChatGPT to give it a dedicated tab
  • OpenAI opens GPT-Live-1.1 API: supports real-time two-way conversation, with agents able to chat while handling complex tasks
  • Apple is developing its own AI agent: able to control devices, browse the web, and write code, aiming to compete with OpenAI and Anthropic
  • ChatGPT adds "real-time dictation": speak into the microphone and speech turns into text instantly
  • Due to an agent breaching a banking system, Arizona launches a criminal investigation into OpenAI and other AI companies

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by IT之家
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗