.Attribute Medicine, Published online: 26 July 2024 doi:10.1038/ s41591-024-03181-6Although large foreign language versions (LLMs) reveal assurance in measured environments, a research study currently reveals their constraints in real-world medical applications and also points the way towards durable evaluation and benchmarking before scientific usage.