Open Problems in Technical AI Governance

Open Problems in Technical AI Governance

Author's Note: This post summarizes a collaborative research paper titled "Open Problems in Technical AI Governance." This project was a joint effort involving over 30 leading experts in AI governance, led by Anka Reuel and Ben Bucknall. Below, we provide an executive summary of the paper. The full paper and an online repository listing open research questions and resources can be found here. 

Introduction

As AI systems become more integrated into our daily lives, there is a growing recognition of the need for effective governance mechanisms to manage their development, deployment, and impact on society. However, current governance mechanisms are hampered by technical limitations. In particular, policymakers and stakeholders face gaps in:

Technical Information: There is a lack of comprehensive technical understanding to identify where interventions are needed and how to assess the efficacy of different policy options.
Technical Tools: The necessary technical tools to successfully implement policy proposals are often underdeveloped or nonexistent.

To address these challenges, we introduce the emerging field of Technical AI Governance—a discipline focused on bridging these gaps by developing technical analyses and tools that support the effective governance of AI systems.

We refer to Technical AI governance as technical analysis and tools for supporting the governance of AI. We argue that technical AI governance can:

Identify areas where policy intervention is needed through mapping technical aspects of systems to risks and opportunities associated with their application;
Inform policy decisions by assessing the effectiveness and feasibility of different policy options; and
Enhance policy options by enabling mechanisms for enforcing, incentivizing, or complying with norms and requirements.

By integrating technical insights into the policy-making process, Technical AI Governance aims to bridge the gap between the capabilities of AI systems and the societal, ethical, and legal considerations they raise.

A Taxonomy of Technical AI Governance

We categorize Technical AI Governance along two dimensions: technical targets across the AI value chain—from inputs to deployment—and governance capacities that can be applied to those targets. Key targets include:

Data: The datasets used for training AI models.
Compute: The computational resources required for AI development.
Algorithms and Models: The software and mathematical frameworks that underpin AI systems.
Deployment: The real-world application and integration of AI systems.

Across these targets, we identify key governance capacities:

Assessment: Evaluating AI systems for safety, fairness, robustness, and effectiveness.
Access: Controlling and facilitating appropriate access to AI systems and data, balancing openness with security and privacy considerations.
Verification: Ensuring the integrity, compliance, and accountability of AI systems through mechanisms like audits and certifications.
Security: Protecting AI systems from unauthorized access, manipulation, and other security threats.
Operationalization: Translating governance objectives into practical implementation strategies, standards, and best practices.
Ecosystem Monitoring: Observing and analyzing trends in AI development, deployment, and impact to inform proactive and future-proof governance.

Alt text: An infographic titled 'Overview of Technical AI Governance Capacities,' featuring six sections, each with an icon and a brief description corresponding to a capacity within AI governance: Assessment, Access, Verification, Security, Operationalization, and Ecosystem Monitoring.

Examples of open problems across key governance capacities

The ability to evaluate AI systems, involving both technical analyses and consideration of broader societal impacts.

Data. What license and meta-data reporting requirements could assist in responsible data practices? What infrastructure is needed to enable researchers to audit large datasets? How can problematic data be identified without full/direct access to the dataset?

Compute. What hardware properties or chip specifications are most indicative of suitability for AI training and/or inference? How does this differ from other scientific, business, or casual uses of high-end hardware? 

Model How can the thoroughness of evaluations be measured? How can potential blind spots of evaluations be identified? How can potential data contamination be accounted for when conducting evaluations? 

Deployment How can downstream impact evaluations be scaled across languages and modalities? How can benchmarks be designed in a way that ensures construct validity and/or ecological validity? 

The ability to interact with AI systems, including model internals, as well as obtain relevant data and information while avoiding unacceptable privacy costs.

Data. How can data access be structured so as to preserve privacy while enabling meaningful auditing? How can openly hosted datasets be prevented from contaminating training data?

Compute. How can public compute resources be allocated fairly and equitably between users? 

Model. What research and auditing methodologies are possible given a range of forms of access on the continuum between black- and white-box access?

Deployment. How can model access requirements for research and auditing be reconciled with commercial and/or safety concerns?

The ability of developers or third parties to verify claims made about AI systems' development, behaviours, capabilities, and safety.

Data. How can it be verified that a model was (not) trained on a given dataset?

Model. How can the risk associated with a given context, query, and AI response be assessed in order to obtain assurances about the system’s compliance with safety requirements?

Deployment. How can metadata watermarking be applied to AI-generated content? How should verification information from model registries be presented to users? 

The development and implementation of measures to protect AI system components from unauthorised access, use, or tampering.

Data. How can AI systems be made robust to data extraction attacks?

Compute. How can hardware-enabled governance methods be implemented at scale to ensure the security of a compute cluster? 

Model.  How effective are model unlearning and model editing techniques when applied to multi-lingual or multi-modal models? What cybersecurity measures can be taken at the infrastructure level to protect model weights from theft by an adversary?

Deployment. How can the robustness of methods for detecting adversarial attacks be improved?

The translation of ethical principles, legal requirements, and policy objectives into concrete technical strategies, procedures, or standards.

What system properties (if any) are the most reliable indicators of risk, and thus candidates for serving as regulatory targets? How can AI safety, reliability, and other technical requirements be standardized given an insufficient explanatory understanding of model behavior? What are general intervention and correction options if flaws with a model are identified post deployment?

Understanding and studying the evolving landscape of AI development and application, and associated impacts.

What risks, whether from intended or unintended harm, are associated with different (types of) systems? How can trends and/or properties observed in current systems be extrapolated to make predictions about future systems?  How could developments in AI-specific hardware impact the governability of compute? What information about systems is needed to accurately assess the environmental impact of their development and deployment? 

Key Takeaways

We highlight several key takeaways from the emerging field of technical AI governance:

Evaluations of systems and their downstream impacts on users and society have been proposed in many governance regimes. However, current evaluations lack robustness, reliability, and validity, especially for foundation models.
Hardware mechanisms could potentially enable actions including facilitating privacy-preserving access to datasets and models, verifying the use of computational resources, or attesting to the results of audits and evaluations. However, the use of such mechanisms for these purposes is largely unproven.
The development of AI research infrastructure, such as resources for conducting analyses of large training datasets or for providing privacy-preserving access to models for evaluation and auditing, could facilitate the scientific understanding of AI systems and external oversight into developers’ activities.
Research that aims to monitor the AI ecosystem by collecting and analyzing data on trends and advances in AI has already proven crucial for providing policymakers with the information needed to ensure that policy is forward-looking and future-proof.

We note that technical AI governance is merely one component of a comprehensive AI governance portfolio, and should therefore be seen in service of broader sociotechnical and political solutions. A purely “techno-solutionist” approach to AI governance and policy is unlikely to succeed.

Recommendations

Based on the above takeaways, we recommend:

Allocating funding and resources to technical AI governance research through open calls and funding bodies, drawing on established expertise in adjacent fields;
That policymakers collaborate closely with technical experts to define feasible objectives and identify viable pathways to implementation;
That government bodies, such as AI Safety Institutes and the EU AI Office, conduct in-house research on technical AI governance topics, beyond their current focus on performing evaluations;
That the future summits on AI, other fora such as the G7, the UN AI advisory body, and reports such as the International Scientific Report on the Safety of Advanced AI, focus effort and attention towards technical AI governance.

Open Problems in Technical AI Governance

Author

Lisa Soder

Become a Subscriber for ...