AI Procurement Is Moving Faster Than Public-Sector AI Governance
Artificial intelligence is entering government through ordinary purchasing channels.
A municipality may acquire an AI-enabled contact center, fraud detection tool, permitting platform, meeting transcription service, or cybersecurity product without classifying it as an AI deployment. A university may add generative features to software already used for admissions, advising, research, or student services. Vendors can also introduce AI capabilities after the original contract is signed.
This creates a structural problem for government AI procurement. Technology can move from demonstration to operational use before the public entity has defined who owns the risk, what to test, or how to monitor performance.
The solution is not to stop every AI purchase until a perfect policy exists. Public institutions need a small set of enforceable controls inside the procurement lifecycle.
Why Standard Technology Procurement Falls Short
Traditional software reviews focus on functionality, cybersecurity, privacy, accessibility, cost, and support. Those controls remain necessary, but an AI-enabled system may:
- Produce different outputs from similar inputs.
- Change after a model or configuration update
- Generate convincing but inaccurate information.
- Depend on training data the buyer cannot inspect
- Send agency data to external models or subprocessors.
- Influence decisions without being identified as a decision-making system
These characteristics make AI difficult to evaluate as a static product. Procurement teams must examine the intended use, data flow, model behavior, and consequences of failure.
The NIST AI Risk Management Framework addresses AI risk across the design, development, use, and evaluation lifecycle. Although voluntary, its approach provides municipalities and universities with a practical framework for evaluating systems based on their actual risks.
Build a Public-Sector Algorithm Inventory First
An agency cannot govern systems it has not identified. A public-sector algorithm inventory should therefore be one of the first elements of an AI governance program.
The inventory should include more than products marketed as artificial intelligence. It should cover machine learning, automated scoring, prediction, classification, recommendation, facial or voice analysis, natural language generation, and other automated decision functions.
Each inventory record should document:
- Product, vendor, department, and internal owner
- Business process and intended use.
- Users and affected population
- AI function, model provider, and relevant subprocessors
- Data inputs, sources, and sensitivity
- Whether the output affects eligibility, enforcement, safety, employment, education, or another consequential decision
- Location and authority of human review
- Latest evaluation date and known limitations
- Contract renewal and termination dates
The document must include embedded AI features. Otherwise, an agency may track new AI purchases but miss capabilities enabled by routine SaaS updates.
Use Risk Tiers in Municipal AI Procurement Policy
Not every system needs the same level of review. A tool that summarizes internal notes does not create the same exposure as software that flags benefit applications for investigation.
A practical municipal AI procurement policy can separate proposed uses into three tiers:
Lower risk
Drafting, summarization, coding assistance, or internal search where a qualified person reviews the output and the system cannot directly affect a person’s rights or access to services.
Moderate risk
Resident-facing chatbots, document classification, service routing, forecasting, or employee monitoring. Errors may affect privacy, service quality, staff workload, or public trust.
Higher risk
Systems that influence eligibility, enforcement, public safety, employment, education, housing, health, or other consequential outcomes. These uses require stronger testing, documentation, oversight, and appeal procedures.
Risk tiering should determine the depth of review, not whether a review happens. Higher-risk systems may require coordinated legal, privacy, security, procurement, program, records, accessibility, and civil rights analysis.
Require Evidence in the AI Vendor Risk Assessment
An AI vendor risk assessment should not rely on a general responsible AI statement. Procurement teams need evidence about the configured product and proposed use.
The review should answer five questions:
1. What does the system do?
The vendor should define which outputs should not be trusted, the intended use, prohibited uses, known limitations, and the conditions under which it operates.
2. What happens to public data?
The assessment should identify what agency data is processed, where it is stored, and how long it is retained. It should also detail whether prompts, files, outputs, or metadata can be used to train a vendor or third-party model.
3. How was performance tested?
Testing should match the proposed workflow and population. A single accuracy score may hide different error rates across languages, document types, demographic groups, or operating conditions. Generative AI evaluations may also need to measure factuality, privacy leakage, harmful content, and prompt-injection resistance.
4. Where does human review occur?
The vendor and agency should document who reviews the output, what evidence the reviewer receives, whether staff can override the system, and how an affected person can challenge a result.
5. How are changes and incidents managed?
The review should identify which model and configuration were tested, how the vendor monitors performance, and what happens when a model update or security incident changes the system’s risk.
The U.S. Government Accountability Office AI Accountability Framework organizes oversight around governance, data, performance, and monitoring. These principles provide a useful structure for vendor evaluation even when state and local entities are not subject to federal agency requirements.
Convert Governance Into Generative AI Contract Clauses
Procurement controls have limited value unless they become enforceable obligations. Effective generative AI contract clauses should address:
- Authorized use: Approved users, environments, purposes, and prohibited applications
- Agency data: Rules for retention, disclosure, reuse, and model training
- Third parties: Disclosure of material model providers and subprocessors
- Documentation: System descriptions, known limitations, evaluation results, and data flows
- Testing rights: Authority to test security, privacy, accessibility, accuracy, and use-specific risks
- Material changes: Advance notice of changes that may alter performance or risk
- Incident reporting: Reportable events, notification deadlines, investigation duties, and corrective action
- Human review: Required oversight and escalation for disputed outputs
- Audit access: Records needed for contract management, investigations, and public accountability
- Exit rights: Feature suspension, termination, data export, deletion verification, and transition assistance
Contract requirements should align with the deployment’s risk and the applicable law. Generic SaaS terms may be inadequate for consequential AI, while controls designed for a high-risk system may be excessive for a limited internal tool.
Continue Governance After the Contract Is Signed
AI approval captures one point in time. Models, prompts, knowledge sources, integrations, and uses can change after deployment.
Post-award governance should include a named system owner, an approved baseline configuration, periodic performance reviews, incident reporting, material-change triggers, and a retirement plan.
NIST’s Generative AI Profile identifies risks specific to generative systems and actions organizations can use to manage them. The procurement implication is clear: testing cannot end with vendor selection.
A Seven-Step Government AI Procurement Gate
Public institutions can create a repeatable governance process without waiting for every policy question to be resolved:
- Identify whether the product uses AI or automated decision logic.
- Inventory its purpose, owner, data, model, and affected population.
- Classify the use by impact and risk.
- Assess the vendor and configured system using relevant evidence.
- Contract for data controls, testing, transparency, change notices, incidents, and exit rights.
- Approve the specific use, model, configuration, and oversight plan.
- Monitor performance, changes, incidents, and expanded uses throughout the contract.
This approach supports responsible experimentation while reducing the chance that pilots and embedded features become unreviewed public infrastructure.
What GovTech Vendors Should Expect
GovTech vendors should expect more detailed questions about data use, model dependencies, testing methods, human oversight, updates, and incident response.
The strongest response is not a promise that an AI system is risk-free. It is a clear explanation of how the system works, where it can fail, how failures are detected, and what controls the public entity can enforce.
Government AI procurement is becoming a governance function. A public-sector algorithm inventory, risk-tiered review, evidence-based vendor assessment, and enforceable contract terms give agencies a practical foundation for evaluating AI before it becomes an operational dependency.
Frequently Asked Questions
What is government AI procurement?
Government AI procurement is the process that public entities use to evaluate, acquire, contract for, implement, and oversee products that contain artificial intelligence or automated decision-making capabilities.
What should a municipal AI procurement policy include?
It should define covered technologies, require an inventory, classify uses by risk, assign review responsibilities, and establish requirements for vendor evidence, approval, monitoring, incidents, and retirement.
What belongs in an AI vendor risk assessment?
The assessment should cover intended use, data sources, model providers, privacy and security controls, performance testing, human oversight, limitations, changes, incidents, and exit requirements.
Which generative AI contract clauses are most important?
Key clauses address authorized use, agency data, model training, third parties, testing rights, documentation, material changes, incident reporting, human review, audit access, termination, and deletion.
Evidence links included in the article:
