AI Evaluation Metrics - Precision

Definition:
Precision is the proportion of true positive results among all positive predictions made by the AI model. In simpler terms, it measures how often the AI's positive advice or recommendations are actually relevant and correct.

Guide for Compliance Team and Engineers:

Purpose:
Reduce false positives—e.g., avoid recommending irrelevant or potentially harmful products or advice, which can lead to user mistrust or safety issues.

For Compliance Team:

  • Set Precision Thresholds: Define acceptable precision levels (e.g., ≥ 80% for wellness, ≥ 93% for pharmacies).

  • Review False Positives: Regularly audit AI outputs to identify and document cases where the AI gave incorrect positive recommendations.

  • User Feedback Loop: Implement mechanisms for users to flag irrelevant or inappropriate advice to feed into precision improvement.

  • Compliance Reporting: Report precision metrics in compliance documentation and track improvements over time.

For Engineers:

  • Balanced Training Data: Ensure training datasets minimize noise and false labels to improve precision.

  • Fine-Tuning Thresholds: Adjust classification thresholds and confidence scores to balance precision vs. recall per use case.

  • Error Analysis: Analyze false positives systematically to identify common causes and update model or data accordingly.

  • Model Calibration: Use calibration techniques (e.g., Platt scaling, isotonic regression) to align confidence scores with true likelihoods.

  • Implement Filters: Add rule-based or post-processing filters to catch obvious false positives in critical contexts.

Compliance Engineering, AI Evaluation MetricsFrancesca Tabor