A mental-health assistant can avoid an obviously dangerous sentence and still respond badly. It can give generic advice without asking what happened, over-medicalise ordinary distress, remove the user’s sense of choice, miss a change from high distress to immediate danger or suggest a resource that does not exist in the user’s country. MentalHealthBench is designed to evaluate that wider response, not only whether a model refuses prohibited content.[1]
OpenAI built the open benchmark with more than eighty licensed psychologists and psychiatrists from twenty-two countries, speaking nineteen languages and representing nearly twenty subspecialties. The conversations are synthetic and cover non-acute situations, high-acuity concerns and emergencies. OpenAI explicitly says their mix is designed for testing and does not represent how frequently these topics appear in ChatGPT.[1]
How the benchmark was constructed
| Measure | Value | Context |
|---|---|---|
| Expert cohort | More than 80 licensed experts | Psychologists and psychiatrists across 22 countries.Source 1 |
| Languages spoken by the expert cohort | 19 languages | The expert group spoke 19 languages; this count is not evidence of equal benchmark or product performance in 19 languages.Source 1 |
| Clinical breadth | Nearly 20 subspecialties | Designed to bring varied professional perspectives into the rubrics.Source 1 |
| Review per conversation | At least three experts | Each synthetic conversation received independent rubric work.Source 1 |
| Criterion retention | Agreement by at least two, no contradiction from a third | OpenAI’s rule for keeping an expert-written criterion.Source 1 |
| Criterion weight | −10 to +10 | Positive values reward helpful behaviour; negative values penalise harmful behaviour.Source 1 |
| Automated grader | GPT-5.6 Sol | Model responses are compared with expert-written criteria by an automated grader.Source 1 |
The benchmark asks whether the response preserves agency
OpenAI identifies four central behaviours: safety, seeking context, preserving user agency and providing actionable guidance when appropriate. Experts read a synthetic conversation and write criteria for the model’s response to its final user message. This tests whether a reply fits the supplied context; it does not follow a real person through an ongoing relationship with the system.[1]
In an everyday situation, a helpful response may reflect feelings, clarify the user’s goal and offer several manageable next steps. In high-acuity distress, it may need to ask more direct questions and encourage timely professional support. In an emergency, the priority shifts toward immediate real-world help, staying connected and using local crisis or emergency services. A universal comforting template would fail these different jobs.
What a strong response must balance
| Behaviour | Useful expression | Common failure |
|---|---|---|
| Safety | Recognise immediate risk and connect the person to real-world support when needed. | Either underreacting to danger or treating every distress signal as an emergency.Source 1Source 2 |
| Context | Ask the smallest important questions before giving consequential guidance. | Confident advice based on a thin or mistaken picture.Source 1 |
| Agency | Offer choices, respect the person’s values and avoid coercive certainty. | Turning the model’s preferred path into an instruction.Source 1Source 2 |
| Actionability | Suggest concrete, feasible next steps appropriate to urgency and location. | Vague encouragement or resources that the user cannot access.Source 1 |
| Boundaries | State what the system cannot assess and when professional care matters. | Imitating a diagnosis or therapeutic relationship beyond the evidence.Source 1Source 2 |
Expert rubrics are richer than one gold answer
A sensitive conversation rarely has one perfect sentence. Rubric criteria can reward several necessary properties while penalising specific harms. One response may be warm but omit an urgent question. Another may identify risk yet strip the user of agency. A weighted rubric can expose both, provided the grader actually recognises the criteria reliably.
The automated grader creates a second evaluation problem. It should be tested against held-out expert judgements, across languages, writing styles and high-stakes criteria. Large average agreement can conceal failure on rare but severe cases. Public use of the benchmark should report per-category performance, disagreement and confidence rather than only one leaderboard number.
A benchmark is not a deployment policy
A consumer product must add controls the benchmark does not establish: age-appropriate experiences, location-aware resources, privacy and retention choices, escalation pathways, rate limits for unhealthy dependency, clear identity and a way to recover after an interrupted crisis interaction. Product monitoring should also distinguish a technically successful response from whether the person reached support.
- Evaluate non-acute, high-acuity and emergency scenarios separately.
- Measure context seeking, agency and actionability alongside safety refusal behaviour.
- Audit automated grading against multiple licensed experts, especially for the largest positive and negative weights.
- Test localisation: language, available services, emergency numbers and culturally appropriate options.
- Track harmful over-escalation as well as missed escalation.
- Never convert a benchmark score into a claim of clinical effectiveness without a clinical study.
MentalHealthBench advances the standard because it asks what a good response should do across the full continuum, not merely what it must avoid saying in an emergency. Its most important idea is that support is relational and contextual: understand enough, protect safety, keep the person’s agency and make the next step more reachable. The next advance is equally important—show that the product carrying those responses behaves safely over time in the lives of real people.
Sources & further reading
- Introducing MentalHealthBench OpenAI · Accessed 10 Oct 2026
- Health Advisory on the Use of Generative AI Chatbots for Wellness and Mental Health American Psychological Association · Accessed 10 Oct 2026