10 questions every general counsel should ask before signing an AI vendor contract
AI vendor contracts in 2026 have a problem: the technology evolves faster than the contracts can be drafted. Procurement teams are signing six- and seven-figure agreements based on demo videos and marketing pages. General counsel teams reviewing these agreements often have no way to verify whether the AI system being purchased actually does what the contract says it does.
These ten questions are the front line. They distinguish vendors who can defend their claims with documentation and evidence from vendors who can’t. A vendor who answers them clearly is probably one you can trust. A vendor who deflects, hedges, or asks for time is a vendor whose system you should audit before signing — and ideally before paying.
1. What specifically does your AI do, and how do you verify it does that thing?
What you’re looking for: a written capability claim that maps to a documented test. “Our AI summarizes customer support transcripts” is a vague claim. “Our AI produces summaries that human reviewers rate ≥4/5 for accuracy on 95% of inputs from our internal benchmark set” is a defensible claim.
Red flag: The answer relies on marketing language (“transforms,” “intelligent,” “industry-leading”) without a measurable outcome you could verify.
What good looks like: A model card, technical specification, or capability document that names the task, the metric, the benchmark, and the result.
2. What is the failure rate, and what scenarios cause it?
Every AI system fails. The question is whether the vendor knows where and why. A vendor who says “our system has a 5% error rate on edge cases involving X” has actually tested their system. A vendor who says “our system is highly accurate” has marketing copy, not a system.
Red flag: The vendor cannot describe a specific failure mode or refuses to share their internal error analysis.
What good looks like: A documented list of known failure modes, edge cases the system shouldn’t be used for, and guardrails the vendor has implemented.
3. Has the system been independently audited?
Internal audits matter. External audits matter more. The AI ecosystem in 2026 has a small but growing number of independent auditors who can verify a vendor’s claims against actual system behavior. If your vendor has been audited externally, ask to see the report. If they haven’t, that’s a data point.
Red flag: “We don’t share our audit results” is a common dodge. Either the audit doesn’t exist, or it’s bad.
What good looks like: A summary or executive overview of an independent audit, with the auditing party named. Not the full audit (that’s often confidential), but at minimum confirmation an audit happened and a high-level finding.
4. What data was the model trained on, and what’s in it?
This question separates serious AI vendors from black boxes. You don’t need every detail — you need to know:
- Was the training data lawfully obtained? (No scraped copyrighted content, no unauthorized PII)
- Is there documentation of training data sources?
- For specialized models: was the model trained on data representative of your domain?
Red flag: The vendor cannot or will not describe training data sources. This is the AI equivalent of a software vendor refusing to tell you what programming language they use.
What good looks like: A training data datasheet, a sourcing methodology document, or at minimum a written affirmation of lawful and documented data sources.
5. What happens when the AI is wrong?
Look for: monitoring, alerting, human review paths, rollback mechanisms, customer-facing communication. An AI deployed without these is an AI deployed without a safety net.
Red flag: The vendor’s answer focuses entirely on prevention (“our system is highly accurate”) with no recovery plan when it fails.
What good looks like: A documented error-handling pathway. Monitoring metrics, alert thresholds, who gets notified, what happens to affected customers, how the model is retrained.
6. What contractual rights do I have to audit your system?
Most standard AI vendor contracts give you zero audit rights. This is a problem. If a regulator (EU AI Act, state-level US disclosure, sector regulator) comes to you asking what your AI does, “the vendor told us it does X” is not a defensible answer. You need contractual right to inspect.
Red flag: “Audit rights are subject to our standard terms” usually means no, you don’t have them.
What good looks like: Explicit audit clauses that permit periodic review, allow independent third-party auditors, require cooperation with regulatory inquiries.
7. Will the system change without my knowledge?
AI systems update. Models retrain. Vendor APIs evolve. The question is whether you find out about it.
Red flag: “We continuously improve our model” with no notification commitment.
What good looks like: Contractual notice requirements for material model changes (the definition of “material” is the negotiation). Versioning. Rollback access for at least 30-90 days after a material change.
8. What’s the data residency, and what data crosses borders?
For European customers (and increasingly for healthcare, finance, and education in the US), this is mandatory due diligence. AI vendors often route inference through multiple regions for cost optimization. That means your data may be processed in countries you didn’t agree to.
Red flag: The vendor cannot tell you with certainty where your data is processed.
What good looks like: A written data residency commitment, with specific countries listed, sub-processor disclosure, and a process for requesting changes.
9. What’s your model risk management framework?
This is the question that separates we built an AI from we have an AI practice. Mature AI vendors have a documented framework for managing model risk: governance, lifecycle management, bias testing, performance monitoring, deprecation criteria.
Red flag: Blank stare or marketing-speak (“we follow industry best practices”) with no specifics.
What good looks like: A model risk management policy that resembles, in structure, what you’d see in financial services or pharmaceutical product development. NIST AI Risk Management Framework alignment is a positive signal.
10. Who is accountable when something goes wrong?
Not “what’s covered by your indemnification clause” — that’s the legal question. The other question is: organizationally, inside the vendor, who is the human accountable for AI risk? Many vendors don’t have anyone with this specific responsibility.
Red flag: The vendor cannot name a specific role (Chief AI Officer, Head of Responsible AI, AI Risk Lead, etc.).
What good looks like: A named role with reporting line. Bonus: a published code of AI ethics or a public commitment that names this accountability.
Putting it together
None of these questions individually are dealbreakers. A vendor failing one might just be a vendor whose paperwork hasn’t caught up to their engineering. But a vendor failing three or four of them is a vendor whose AI you don’t actually understand.
Most general counsel teams reviewing AI contracts today don’t have a structured way to assess this. The questions get asked piecemeal, the answers come back in marketing language, and the contract gets signed because the procurement clock is ticking.
That’s why we built the Diagnostic. Ten questions, ten minutes, written read of where your organization stands on AI governance maturity. If the questions above feel like a checklist you don’t currently have, the Diagnostic is the cheapest read on what to address first.
If you’ve already worked through them and have systematic answers, you’re further along than most. We work with companies on both sides of that line.