C. Safety

Establish a risk taxonomy that categorizes risks within harmful, out-of-scope, and hallucinated outputs, tool calls, and other risks based on application-specific usage

Conduct internal testing of AI systems prior to deployment across risk categories (including high-risk, harmful, hallucinated, and out-of-scope outputs and tool calls) for system changes requiring formal review or approval

Implement safeguards or technical controls to prevent harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception

Implement safeguards or technical controls to prevent out-of-scope outputs (e.g. political discussion, healthcare advice)

Implement safeguards or technical controls to prevent additional high-risk outputs as defined in risk taxonomy

Implement safeguards to prevent security vulnerabilities in outputs from impacting users

Implement an alerting system that flags high-risk recommendations for human review

Implement monitoring of AI systems across risk categories

Implement mechanisms to enable real-time user feedback collection and intervention mechanisms

Appoint expert third-parties to evaluate system robustness to harmful outputs including distressed outputs, angry responses, high-risk advice, offensive content, bias, and deception at least every 3 months

Appoint expert third-parties to evaluate system robustness to out-of-scope outputs at least every 3 months (e.g. political discussion, healthcare advice)

Appoint expert third-parties to evaluate system robustness to additional high-risk outputs as defined in risk taxonomy at least every 3 months