Every two weeks a client asks us to review a proposed AI vendor. The volume is not slowing down. This is the checklist we run on each one. It is deliberately opinionated — some items reflect specific regulatory postures (UK, EU, US) and some reflect scars from prior engagements.
Section 1 — Company and product
- How long has the vendor existed? Last funding round; runway; ownership structure. For start-ups, we are looking at whether the product will exist in three years.
- What is the core product, and what is adjacent? Many vendors have a strong core and weak adjacencies. Scrutinise what you actually need.
- How many paying enterprise customers? Reference calls with at least two of them, selected by you, not by the vendor.
- What is the roadmap for the next twelve months? Is it coherent, or a wishlist?
- Is there a UK entity, or only US/EU? Matters for contracting and some regulated sectors.
Section 2 — Data handling
- Where is data stored? Specific regions; not "our cloud". UK, EU, US each have different implications.
- Who has logical access, and how is it audited? SOC 2 Type II or equivalent; if not, why not.
- Does the vendor train on your data? If so, under what conditions, and can it be turned off? "Enterprise" tier should default to no.
- What is the retention policy for logs, prompts, outputs? Can you set it?
- Encryption at rest, in transit, and — if relevant — in use (confidential compute). Key management: vendor-managed or customer-managed?
- Sub-processor list, with change notification. If the vendor is opaque here, stop.
Section 3 — Model provenance
- Which base models are used? Proprietary, OpenAI, Anthropic, Mistral, open-weight? The mix matters for risk exposure and regional compliance.
- Can the model set be restricted to specific jurisdictions or providers on request?
- How is the base model hosted — native vendor, pass-through, self-hosted? Pass-through means vendor terms stack on top of model provider terms.
- What happens when the base model is deprecated? Who carries the switching cost?
Section 4 — Safety, bias and evaluation
- What evaluation do they run on their own product? Ask for a recent eval report.
- What safety filters are in place, and can you inspect the filter taxonomy?
- How are hallucinations surfaced? Is there a confidence score?
- Has the vendor published red-team results? Serious vendors do.
Section 5 — Integration and operability
- API stability: how long do they keep APIs stable? What is the deprecation policy?
- Authentication: OIDC, SAML, SCIM? Role-based access control?
- Observability: can you export logs, traces, usage? Is there an audit log?
- Rate limits and quotas: what is the enterprise ceiling?
- SDK availability and maturity.
Section 6 — Commercial
- Pricing model: seat, token, flat, or hybrid? For AI products, token-based pricing can be deeply volatile; flat-rate is preferable for budgeting.
- Price lock: how long is the price fixed, and what is the escalator?
- Minimum commitment: annual, multi-year, true-up provisions?
- Volume discount structure.
- Termination clauses: can you exit early, and at what cost?
Section 7 — Contractual
- DPA that is actually negotiated, not a pre-signed PDF.
- Liability cap: insist on multiple of fees, not a flat cap.
- Indemnities: IP indemnity on model outputs (the major providers now offer this; ask for it).
- Audit rights: can you or your regulator audit?
- Notification obligations for breaches, model changes, acquisition, sub-processor change.
Section 8 — Exit
- What data can you extract, and in what format? Prompts, outputs, fine-tuning data, evaluation data?
- How long does exit take, and who does the work?
- Is your fine-tuned model portable, or locked to the vendor?
- What happens to logs and backups after exit, and how is deletion evidenced?
Section 9 — Regulatory posture
- Does the vendor understand and accommodate UK/EU AI Act obligations where relevant?
- Sector-specific certifications — HIPAA, ISO 27001, Cyber Essentials Plus, NHS DSPT, ISO 42001?
- Support for client-side DPIAs: do they provide a pre-populated DPIA annex? Serious vendors now do.
How to run the assessment
Two hours with the vendor's solution engineer, one hour with their security team, one hour with their legal team, two reference calls. Document in a single spreadsheet. A red, amber, green per item, with notes. The goal is not "pass"; it is an accurate picture of what you are signing up for.
Common disqualifiers
- "Your data might be used to improve our models" with no opt-out.
- No SOC 2 Type II and no clear path to one.
- Pricing based on tokens with no cap, no price lock, no usage transparency.
- DPA non-negotiable.
- Sub-processor list opaque or unshareable.
- Exit clause makes re-use of fine-tuning work impossible.
Any one of these should trigger a hard conversation. Two should probably stop the engagement.