25-question AI Vendor Assessment Checklist
A practical AI Vendor Assessment Checklist for evaluating AI suppliers.
The demo was impressive. The AI answered questions, summarised documents and promised to save everyone hours of work. Someone asked about pricing. Someone else suggested a pilot.
But here's a less exciting question: what exactly are we agreeing to?
Buying AI software still requires the usual checks on security, privacy and reliability. Yet AI introduces a few extra complications: your data might pass through a third-party model, the system might confidently produce incorrect answers, and some AI agents can take actions rather than merely make suggestions.
Before signing anything, I'd want clear answers in six areas.
Six things to ask your AI supplier
What powers it? Which models and subcontractors are involved? Can the supplier change them without notice?
What happens to our data? Are prompts, files or outputs retained, shared or used to train models? Where are they processed?
How is it secured? Beyond standard cyber controls, how does the supplier test for prompt injection and accidental data disclosure?
How reliable is it? What has been tested, against which use cases, and what are the known failure modes?
Who stays in control? Can people review or override decisions? What permissions and safeguards constrain autonomous agents?
What happens later? How are changes, incidents, data deletion and contract exit handled?
A reassuring "yes" is not evidence. Ask for documentation, test results, contractual commitments or a demonstration of the control itself.
Download the Free AI Vendor Checklist
I've put these checks into a 25-question AI Vendor Assessment Checklist, with space for supplier responses, supporting evidence, reviewer findings, risks and a final decision.
Download the Free AI Vendor Checklist
It's a starting point, not a compliance certificate. Adapt the depth of review to what the AI will do and the consequences if it gets something wrong. A drafting assistant and an agent authorised to change customer records should not face the same scrutiny.
AI may have become rather good at answering questions. That's no reason to stop asking them.
For further guidance, see the ICO's AI and data protection resources and the NCSC's supplier assurance questions.
