AI as an Accountable Entity: Governing Risk When Machines Make Decisions

AI as an Accountable Entity: Governing Risk When Machines Make Decisions

Speakers: Pavithra Pradip

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=jqV61NQOCLw

Overview

Pavithra Pradip’s talk, "AI as an Accountable Entity: Governing Risk When Machines Make Decisions," addresses a critical and rapidly escalating challenge in modern technology: the governance of Artificial Intelligence systems that make autonomous decisions. As AI transitions from being merely a tool or feature to becoming a distributed decision engine, traditional oversight mechanisms designed for human-centric processes or predictable software fall short. The core problem highlighted is the subtle, often unalarming failure modes of AI—such as drift, bias, and unfairness—which can have significant real-world impacts without clear accountability.

Pradip, a Staff Technical Compliance Manager known for bridging business, engineering, and security, argues that the existing frameworks of Governance, Risk, and Compliance (GRC) are not obsolete but require a fundamental shift in mindset and application to effectively manage AI risks. Her presentation reframes AI systems as risk-bearing assets, necessitating a design-centric approach to governance rather than treating it as an afterthought. The talk provides a pragmatic framework for establishing accountability, transparency, and continuous assurance for AI systems, ensuring that even as machines make decisions at unprecedented speeds, human oversight and ethical considerations remain paramount.

The significance of this topic cannot be overstated in an era where AI is increasingly integrated into critical functions, from financial services to healthcare and autonomous vehicles. Without robust governance, the potential for unintended negative consequences, legal liabilities, and erosion of public trust is substantial. Pradip's work emphasizes that rather than inventing entirely new bureaucratic processes, organizations should adapt and extend existing engineering principles and GRC practices to address the unique characteristics of AI, thereby making these powerful systems explainable, auditable, and ultimately, accountable.

Background

▶ Watch: Governing AI decisions: The core challenge (0:54)

Traditionally, software development and deployment have operated on explicit rules, predictable behavior, and human-understandable logic. When a piece of software malfunctions, it typically "fails loudly," producing errors that are traceable to specific lines of code or design flaws. Oversight and accountability in these systems are relatively straightforward: a human loan officer reviews an application, and the rationale for approval or rejection is documented and auditable. The intent is clear, and processes are stable and well-documented.

However, the advent of AI systems introduces a paradigm shift. Unlike traditional software, AI and Machine Learning (ML) models learn patterns from data, making decisions based on probabilistic behavior that is constantly evolving through environmental changes, model retraining, and new data inputs. This dynamic nature makes their behavior inherently harder to predict, explain, or even trace in a fixed manner. Pradip highlights that AI systems rarely fail loudly; instead, they exhibit subtle failures like model drift, bias, or unfairness that gradually lead to bad outcomes without triggering alarms. Teams working with these systems might sense something is wrong, but pinpointing a single point of failure becomes exceedingly difficult.

The existing GRC frameworks, often perceived as bureaucratic processes involving audits and paperwork, were not designed for systems that learn and adapt. While engineers already practice risk management through activities like design reviews, incident response, and reliability engineering, these practices need expansion. Pradip argues that GRC, when correctly applied, builds on this engineering foundation by extending into areas like compliance, accountability, and privacy for AI. The challenge is that AI now makes decisions faster than traditional governance was ever intended to supervise, creating a gap in oversight that needs urgent attention. The central question posed is: if an autonomous AI system makes a wrong decision, such as rejecting a loan application that should have been approved, who is accountable—the bank, the engineers, or the GRC team? This problem underscores the necessity of rethinking accountability in the age of autonomous AI.

Key Findings

▶ Watch: Loan application example: Autonomous AI decisions (3:22)

Pradip's talk introduces several pivotal concepts for governing AI effectively. The first fundamental mind shift is to recognize that AI systems are not just a feature; they are a system that carries risk, akin to a service or a data pipeline. By viewing AI as a risk-bearing asset, governance transforms from a feature problem into a design problem, integrated from the outset of development.

To manage this risk, organizations must first inventory their AI systems, understanding their purpose, autonomy level, data types, and real-world impact. However, not all AI systems require the same level of rigorous oversight. Pradip proposes a materiality scoring approach based on three primary factors to determine which AI systems warrant governance:

1. Impact: Does the AI system influence critical business or customer outcomes? Does it have a direct and significant effect on customers or core business goals?
2. Autonomy: How much freedom does the AI system have in making decisions? Is it fully autonomous, semi-autonomous with human review, or merely advisory?
3. Sensitivity: How sensitive is the data the AI system processes? Does it involve Personally Identifiable Information (PII), HIPAA-protected data, or other regulated and sensitive information?

These factors, possibly combined with others like regulatory exposure, can be used to calculate a materiality score. For instance, using a basic summation where each factor is scored from one to three (three being high), an AI loan approval system would likely score high (e.g., three for impact, three for autonomy, three for sensitivity), resulting in a high materiality score (e.g., nine out of nine), indicating a strong need for governance.

The core of AI accountability, according to Pradip, rests on answering three critical questions, forming an accountability triangle:

1. What it did: Understanding the decision made by the AI.
2. Why it did it: Explaining the rationale or interpretability behind the decision.
3. Who owns it: Identifying the human parties responsible for the system's actions.
4. Can you prove it: Demonstrating the decision path through structured evidence.

If any of these pillars—ownership, explainability, or auditability—are missing, true accountability becomes impossible. AI does not remove accountability; it merely redistributes it among various human stakeholders, making it imperative to clearly define these roles and mechanisms for proof.

Technical Deep Dive

▶ Watch: Redefining GRC for engineers: Ownership, risk, compliance (4:22)

The technical deep dive into AI governance, as presented by Pradip, centers on operationalizing the accountability triangle and applying established engineering principles of reliability and observability to the dynamic nature of AI. This involves a shift from static, human-centric oversight to a more continuous, system-level approach.

Central to this framework is the concept of ownership. In traditional software, ownership might reside with a single team or individual. For AI, Pradip emphasizes redistributed ownership. This means accountability is shared across multiple roles:

• Model Owner: Responsible for the integrity of the AI model, its performance, and retraining processes. This role ensures the model functions as intended and is regularly updated.
• Business Owner: Accountable for the ethical outcomes of the AI system, ensuring its decisions align with business values, fairness, and legal requirements. They define the desired behavior and impact.
• GRC/Control Owners: Responsible for the overall oversight and compliance, setting the guardrails within which the AI system must operate. This includes defining policies, conducting reviews, and ensuring adherence to regulations.

The collective responsibility of these owners ensures that when decisions go wrong, there is a clear escalation path and designated parties "on call" to address the issues.

Explainability addresses the "what" and "why" of an AI's decision. For an AI system to be accountable, it cannot be a black box. Explainability is about the interpretability of a decision, allowing stakeholders—be they auditors, regulators, or customers—to understand how and why a particular outcome was reached. This is crucial for justifying decisions, debugging unexpected behaviors, and building trust. Without explainability, an AI system cannot be defended or improved effectively. It involves understanding the criteria the model used, the data it processed, and the logic it applied, even if that logic is learned rather than explicitly programmed.

Finally, auditability provides the "proof." It's not merely about collecting logs of outcomes but about gathering structured evidence over time. This evidence should encompass the entire lifecycle of the AI system:

• Details of the teams involved and their accountable roles.
• The explanations behind decisions.
• Information about retraining data and processes.
• Parameters and assumptions used at the time of decision-making.

The goal of auditability is to enable the reconstruction of history. Given the exact same data, parameters, and assumptions, an auditor should be able to reproduce the decision path and outcome. This allows internal and external auditors to evaluate the effectiveness of controls, identify drift, bias, or unexplained changes, and verify the system's fairness and stability. Pradip terms this combination of ownership, explainability, and auditability as the minimum viable reliability model for AI governance, effectively applying the engineering concepts of observability, ownership, and historical data to AI decisions.

Demo / Proof of Concept

▶ Watch: AI breaks traditional human-centric oversight and accountability (6:08)

The talk did not feature a live technical demonstration or a specific proof of concept tool. Instead, Pavithra Pradip used a consistent, illustrative example throughout her presentation: an AI-powered loan application system that autonomously approves or rejects loans. This example served as a conceptual proof of value, demonstrating the real-world implications of AI decision-making and the challenges it poses for accountability.

By consistently referring back to the loan system, Pradip effectively illustrated how the proposed governance framework—including materiality scoring, distributed ownership, explainability requirements, and auditability—would apply in a practical, high-stakes scenario. For instance, she detailed how such a system would score high on impact, autonomy, and sensitivity, thus requiring significant oversight. She also explained how specific governance, risk, and assurance controls (e.g., business owner sign-off, quarterly bias testing, semi-annual audits) would be implemented to ensure the loan AI system operates fairly, ethically, and accountably. While not a code-level demo, this continuous, concrete example successfully contextualized the abstract concepts for the audience.

Defensive Implications

▶ Watch: AI's subtle failures: Drift, bias, and gradual bad outcomes (7:08)

The defensive implications of Pradip's framework are centered on adapting and extending existing GRC controls to function as guardrails for AI systems. She reframes controls not as bureaucracy but as boundaries that define acceptable system behavior, translating abstract values like fairness and stability into testable conditions. These controls are categorized into three types:

1. Governance Controls: These act as the "steering wheel," setting the overall direction and policies before the AI system is even deployed. For the loan AI example, governance controls would include:

• Requiring business owner sign-off before deployment.
• Mandating an ethics review for the model.
• Establishing a policy for annual recertification of the system, including regular testing.

These controls ensure the system is on the right path from the outset.

1. Risk Controls: Functioning like the "brakes" of a car, these controls ensure fairness and stability throughout the AI system's operational life, allowing for intervention if conditions change. Examples for the loan AI system include:

• Implementing quarterly bias testing to detect and mitigate discriminatory outcomes.
• Establishing fairness monitoring to track for model drift or unexpected changes in decision-making patterns.
• Generating explainability logs for every decision, detailing the criteria and factors that led to a specific outcome.

These proactive measures help contain the blast radius of failures and ensure the system remains within acceptable parameters.

1. Assurance Controls: These are the "dashboard," providing performance monitoring and verification that the governance and risk controls are working as intended. For the loan AI system, assurance controls would involve:

• Semi-annual audits conducted by internal or external auditors.
• Performance testing to verify consistent and fair outcomes, even with demographic changes or new features.
• Reproducibility checks to confirm that the system yields the same outcomes given identical parameters and data.

Crucially, Pradip emphasizes a shift from static, one-time audits to continuous assurance for AI systems. Because AI models are constantly learning and evolving, continuous monitoring and auditing are essential to maintain trust over time and detect subtle failures like drift or bias as they emerge. This approach ensures that oversight is as dynamic as the AI systems themselves, allowing organizations to maintain accountability and mitigate risks effectively.

Key Takeaways

• AI as a Risk-Bearing Asset: Organizations must shift their mindset from viewing AI as merely a feature to recognizing it as a critical, risk-bearing system akin to a service or data pipeline.
• Risk-Based Oversight: Not all AI systems require the same level of governance. Prioritize oversight based on a materiality score derived from impact (on business/customers), autonomy (level of independent decision-making), and sensitivity (of data processed).
• The Accountability Triangle: True AI accountability relies on three interconnected pillars: clear ownership (redistributed among model, business, and GRC owners), robust explainability (understanding what and why a decision was made), and comprehensive auditability (structured, reconstructible evidence of decisions over time).
• Adapt Existing Frameworks: Instead of creating entirely new, bureaucratic processes, leverage and adapt existing engineering practices (like reliability engineering, incident response) and GRC frameworks to address the unique challenges of AI.
• Continuous Assurance is Key: Traditional static audits are insufficient for dynamic AI systems. Implement continuous assurance mechanisms, including ongoing bias testing, fairness monitoring, and regular performance evaluations, to maintain trust and detect subtle failures over time.
• Treat Retraining as Deployment: Any retraining of an AI model should be treated with the same rigor as deploying a new feature or product, ensuring thorough testing, compliance checks, and adherence to governance policies.

About the Speaker(s)

Pavithra Pradip is a Staff Technical Compliance Manager at a large company, which she humorously alludes to as having connections to April 15th (tax day). She is particularly recognized for her expertise in integrating business, engineering, and security functions, fostering collaboration across these critical domains. Her background uniquely positions her to address the complex intersection of technical innovation, regulatory compliance, and risk management in the context of emerging technologies like Artificial Intelligence. Pradip's work focuses on developing practical, actionable frameworks that extend existing organizational principles to ensure accountability and ethical operation of advanced AI systems.