How Non-Technical Founders Can Evaluate an ML Vendor

Frequently Asked Questions

01.How can a non-technical founder test an ML vendor's model?

Hold back a few hundred labelled examples from your own records, ask each vendor to score them without seeing the labels, and compare the results with today's error rate. Public benchmarks can overstate accuracy: on a fresh test built to mirror a public one, researchers observed drops of up to eight percent, though many models, especially frontier ones, showed minimal overfitting (arXiv, November 2024).

02.Does an ML vendor train its models on my data?

It depends on the contract and often on the plan. Google's Gemini API terms, effective March 2026, let Google use content from unpaid services to improve its products but not prompts or responses on paid services, and apply the paid data-use terms to all services in the European Economic Area, Switzerland and the United Kingdom (Google, March 2026). Get the answer in writing for the exact plan you will buy.

03.Who is the provider under the EU AI Act when I buy an ML system?

Usually the vendor that develops the system and places it on the market under its own name. Under Article 25 of Regulation (EU) 2024/1689, you become the provider of a high-risk system if you put your name or trademark on it, subject to contracts allocating the obligations otherwise, substantially modify it, or repurpose a system so it becomes high-risk. For Annex III systems these duties apply from 2 December 2027, and to systems already in service before then only if their design changes significantly (EUR-Lex, July 2026).

04.How do I avoid lock-in with an ML vendor?

Own the artefacts you pay to create: prompts, labelled data, evaluation sets and any fine-tuned weights, with export formats written into the contract. In a 2025 survey of 100 CIOs, 37 in every 100 used five or more models, up from 29 in every 100 a year earlier, and a16z found agentic workflows make switching harder (a16z, June 2025). Rehearse the export before you sign.