BSI’s new AI assessments make the scope of a claim matter

BSI announced new AI assessment services on 14 October covering large language model performance and product risk, alongside a biometric readiness pilot. The services extend an existing offer that includes certification of organisations’ AI management systems. Each examines a different object: a performance claim, a product’s risks or the way an organisation manages AI.[1]

Start with the claim

BSI’s general AI Performance service begins by agreeing the model, dataset and metrics. Clients provide the model and data, reference answers and claimed results. BSI then compares its evaluation with those claims. Where sharing the model or dataset is impractical, the service can assess supplied model outputs against reference answers instead.[2]

For language models, its published metrics include factual accuracy, consistency, summarisation quality and toxicity. Subgroup comparisons require additional demographic metadata; robustness can be examined under changed data conditions. Those choices establish which aspects of a system an assessment can speak about.[2]

Consider an illustrative customer-service assistant that explains a retailer’s returns policy. A test could examine whether its answers correctly state the deadline in each country. Another could ask whether the tone remains appropriate when a customer is angry. Both are useful questions. A result about one tells the buyer little about the other unless the test explicitly covers both.

Follow the product

The new product-risk assessment references the NIST AI Risk Management Framework and ISO/IEC 23894. BSI describes it as examining how risks are identified, controlled and managed at the product and application layer.[1]

NIST’s framework organises the work into governance, understanding the context, measurement and management. These functions interact throughout a system’s life. Its measurement guidance includes testing before deployment and during operation, reporting uncertainty, and comparing performance with benchmarks. Independent review can help expose internal biases in that evaluation.[3]

In the returns example, the operational question might be what happens after an incorrect answer: whether it reaches a customer, whether staff can correct it, and whether the error changes a later release. That connects a model’s behaviour to a service people actually use. A single accuracy percentage cannot describe the whole sequence.

Examine the organisation

ISO/IEC 42001 concerns an AI management system: the organisation’s policies, objectives and processes for developing, providing or using AI responsibly. Its requirements cover establishing, maintaining and improving that system. The object of this standard is the organisation’s management arrangements. A particular answer from a particular model still needs its own evidence.[4]

BSI says successful performance assessment produces a verification letter and access to its Mark of Trust against defined criteria. It also recommends reassessment around major releases, component changes and ongoing monitoring. The letter’s scope therefore matters as much as the visible mark.[2]

For the hypothetical retailer, an assurance record could name the assistant version, countries covered and outcomes assessed. A buyer could then connect the tested claim with the service being purchased.[2]

Report an error or suggest a correction