The AI You Approved in March Is Not the AI Running in July
Dr. Jonah Tebaa argues that traditional software governance is no longer effective for AI systems, as they continue to change after initial approval. This is not a problem unique to any particular organization, but rather a default condition of running AI today. The assumption that a system only changes when intentionally modified is no longer true, as AI models can be updated by vendors, and the data they retrieve can drift over time.
A hospital system's experience with a triage-support model illustrates this issue. After eight months of clinical review, security review, and board-level sign-off, the model was finally approved and went live. However, the vendor had already pushed two silent version updates during those eight months, making the approved system different from the one handling patient intake. This scenario is not unique to the hospital, but rather a common issue in organizations running AI today.
Dr. Tebaa emphasizes that approved is not a permanent state, but rather a snapshot with an expiration date that is often not explicitly stated. The instinct to tighten the front door by adding more checks and signatures before launch is misguided, as it does not address the drift that occurs after the initial approval. Instead, Dr. Tebaa recommends a set of ongoing habits to catch potential issues.
This isn't unique to healthcare. A survey of real-world machine learning deployments found that detection of concept drift becomes a growing concern for teams that maintain ML models in production, which is the same after-approval shift Dr. Tebaa is describing under a different name.
Ongoing Habits for AI Governance
Dr. Tebaa's framework consists of five practices that help mitigate the risks associated with AI system drift. These practices include:
- Version Pinning Where You Can: Pinning the version of the underlying model or API instead of accepting the latest version. This ensures that the system behaves consistently, and changes are made intentionally.
- The Update Trigger: Implementing a standing rule that automatically triggers a re-evaluation before a new version touches production traffic. This mechanism ensures that changes are properly assessed and approved.
- A Living Evaluation Set: Maintaining a small set of real cases that are re-run on a fixed schedule and after any material change. This provides a consistent and comparable read on the system's behavior over time.
- Re-approval on a Calendar: Scheduling re-approval on a regular basis, with the frequency depending on the system's consequences. For example, a system touching money, safety, or legal exposure may require quarterly re-approval, while a low-stakes internal tool may only need annual re-approval.
- Material Change Management: Establishing a process to manage material changes to the system, including updates to the model, API, or configuration. This ensures that changes are properly assessed, approved, and implemented.
Dr. Tebaa's framework provides a structured approach to AI governance, acknowledging that AI systems are inherently dynamic and require ongoing monitoring and maintenance. By implementing these practices, organizations can better manage the risks associated with AI system drift and ensure that their AI systems operate within approved parameters.
The importance of ongoing habits in AI governance cannot be overstated. As AI systems continue to evolve and improve, it is essential to have a framework in place that can adapt to these changes. Dr. Tebaa's framework provides a foundation for organizations to build upon, ensuring that their AI systems are governed effectively and operate within approved parameters.
Conclusion
In conclusion, Dr. Tebaa's argument highlights the need for a new approach to AI governance, one that acknowledges the dynamic nature of AI systems. By recognizing that approved is not a permanent state, organizations can take steps to implement ongoing habits that mitigate the risks associated with AI system drift. Dr. Tebaa's framework provides a valuable resource for organizations seeking to improve their AI governance practices and ensure that their AI systems operate effectively and within approved parameters.
