The AI Vendor Evaluation Framework: How to Score AI Products Before You Buy

Evaluating an AI vendor comes down to scoring six dimensions: Capability Fit (does the product do your specific job, measured on your data, not the vendor's demo), Evaluation Evidence (can the vendor show systematic quality measurement, error rates, and regression history for the claims it makes), Data and Deployment Posture (where your data goes, what trains on it, and what the deployment surface exposes), Cost Structure Transparency (can you compute the cost of one unit of work from the contract, and does that cost survive volume), Vendor Durability (will this company, product, and underlying model path exist in three years), and Exit Cost (what leaving would take in data, workflow, and re-tuning terms, priced before signing rather than after). Each dimension is scored zero to four against deterministic gate questions, then weighted: Evaluation Evidence carries 25 points, Capability Fit 20, and the remaining four dimensions 15, 15, 10, and 15 respectively, for a 100-point scale. The weighting rule is deterministic and unforgiving: a zero on any dimension disqualifies the vendor regardless of total, because a weighted average must never launder a fatal flaw, and totals only rank vendors that pass every gate. One mental model governs the entire exercise: demos are ground the vendor picked, evals are ground you picked, and a vendor is never scored on ground it picked. Everything below expands that paragraph: the gate questions per dimension an executive can put directly into an RFP, the scoring matrix, and the red flags that end conversations early.

Key Takeaways

• Score AI vendors on six named dimensions: Capability Fit, Evaluation Evidence, Data and Deployment Posture, Cost Structure Transparency, Vendor Durability, and Exit Cost. Weight them 20, 25, 15, 15, 10, and 15 on a 100-point scale.
• The scoring rule is deterministic: zero to four per dimension against pass/fail gate questions, weighted total for ranking, and any single zero disqualifies outright. Averages exist to compare qualified vendors, never to rescue disqualified ones.
• The central trap is scoring the demo. A demo is a performance on inputs the vendor chose; an evaluation is a measurement on inputs you chose. Every dimension below converts vendor claims into questions answerable on your ground.
• The timing matters: Gartner projects that over 40 percent of agentic AI projects will be canceled by the end of 2027 even as agentic capability reaches a third of enterprise software by 2028, and platform vendors are bundling AI into consolidation plays. Buying discipline is scarcest exactly when the selling pressure is highest.
• The framework runs in a live vendor meeting in about ten minutes using the scoring matrix, and in full as an RFP section. Either way, the questions are designed so that a vendor's refusal to answer is itself a score.

Why AI Procurement Broke the Old Playbook

Enterprise software procurement has a mature playbook: reference calls, feature checklists, security questionnaires, a pilot, a negotiation. That playbook assumed the product behaves deterministically, so a feature either exists or does not, and a pilot that works will keep working. AI products violate the assumption at the root. Their output quality is probabilistic, varies with the input distribution, drifts as models are swapped under the hood, and degrades in ways a feature checklist cannot express. A vendor can pass every line of a traditional RFP and still fail catastrophically on your actual workload, because the traditional RFP never measured quality on your distribution.

The market context, as of the week of July 13, makes the discipline urgent rather than academic. Agent features are entering production faster than the governance around them: Gartner projected in mid 2025 that more than 40 percent of agentic AI projects will be canceled by the end of 2027 on cost and risk grounds, while the same firm expects agentic capability embedded in a third of enterprise software by 2028, up from under 1 percent in 2024. Both numbers describe the same buying environment, one where nearly every renewal now arrives with an AI addendum, priced with new meters the buyer has never forecast against. At the same time, the large platform vendors are bundling AI features into consolidation plays, offering to collapse point solutions into a suite where the AI is "included," which is precisely the moment disciplined per-dimension scoring matters most: a bundle is a mechanism for making individually unjustifiable products impossible to score individually.

The framework below is the buying-side counterpart to two other FinTekCafe frameworks: the technical due diligence guide covers evaluating a company's technology in an acquisition, and the AI agent readiness maturity model covers whether your own organization is ready to deploy agents at all. This one answers the question in between: a specific AI product is on the table, and someone has to decide.

The Central Mental Model: Demos Are Their Ground, Evals Are Yours

Before the dimensions, the one idea that governs all of them. Every AI sales motion is built around the demo, and the demo is a performance: inputs selected by the vendor, a scenario rehearsed by the vendor, edge cases absent by design. None of this is dishonest. It is what a demo is. But a probabilistic product evaluated on chosen inputs tells you the ceiling of its behavior, and a buyer needs the floor.

The corrective is to move every consequential claim onto ground you picked: your documents, your tickets, your transactions, your edge cases, scored against your definition of acceptable. That is an evaluation, and building even a small one changes the power dynamics of the purchase. Organizations that maintain their own evaluation sets, along the lines described in the executive guide to AI evals, can put fifty representative cases in front of any candidate product in a day and read the failure modes directly. Organizations that cannot are, structurally, buying the demo. The rule that falls out of this, and that the rest of the framework enforces dimension by dimension, is simple: never score a vendor on ground the vendor picked.

The Six Dimensions

Dimension 1: Capability Fit (weight 20)

What it measures: whether the product performs your specific job on your data distribution at a quality bar you defined, as opposed to performing an adjacent job on clean data at a bar the vendor defined.

Gate questions for the RFP or the live meeting:

• Will you run against a blind sample of our data before commercial commitment, with results we score, not you? (Pass requires yes.)
• What is the product's measured error rate on the task category, on data the vendor did not curate? A vendor that has no number fails; a vendor whose number comes only from a public benchmark scores one at best, because public benchmarks are chosen ground.
• What happens on the inputs the product cannot handle: does it fail loudly with an abstention, or produce a confident wrong answer? Confident wrongness on your workload is the single most expensive behavior in applied AI.
• Which parts of the demo were the model and which were scaffolding, retrieval, or hand-built rules? The answer reveals how much of the magic generalizes.

What good looks like: the vendor volunteers a blind pilot on your data, hands over an error-rate table by input category, and describes failure modes unprompted. Red flag: any variant of "our accuracy is over 95 percent" with no denominator, no dataset description, and no error taxonomy.

Dimension 2: Evaluation Evidence (weight 25)

What it measures: whether the vendor itself measures quality systematically, because a vendor that cannot show its own evaluation machinery is asking you to be its evaluation machinery, in production, at your expense. This dimension carries the largest weight for a structural reason: it is the best single predictor of every other quality claim being true. The argument that evaluation assets are the durable moat of applied AI is made in full in the analysis of evals as the competitive moat; this dimension is that argument turned into procurement.

Gate questions:

• Show the regression suite: how many graded cases, covering what input categories, refreshed how often? A real answer has numbers and a cadence. (Pass requires specifics; "we test extensively" is a zero.)
• When you swap or upgrade the underlying model, what is re-verified before the change reaches customers, and what is the notification policy? Vendors ride on frontier models and swap them for margin; an unevaluated silent swap is a quality event you inherit without warning.
• What is the escape rate: how many production defects per period reach customers, and what was the last one? Every real vendor has one. A vendor with no defect to describe is not defect-free; it is measurement-free.
• Can we see the eval results for our specific pilot, in the same harness the vendor uses internally, as described in the enterprise LLM evaluation harness pattern?

What good looks like: a named eval suite with case counts, model-swap regression gates, and a willingness to share category-level scores under NDA. Red flag: quality claims sourced entirely to customer logos and benchmark leaderboards.

Dimension 3: Data and Deployment Posture (weight 15)

What it measures: where your data goes, what is retained, what trains on it, and how much attack and failure surface the deployment adds. This is the dimension where traditional security review still applies, extended by the questions unique to AI systems.

Gate questions:

• Trace one request end to end: which subprocessors touch it, in which regions, with what retention? (Pass requires a complete, written data-flow answer; a diagram is better.)
• Is customer data used to train or fine-tune any model, including "de-identified" or "aggregate" use, and is the exclusion contractual or a settings toggle? Toggles get reset; contracts do not.
• What sits between the model and your systems: what input filtering, output validation, and action authorization exist, in the sense described in the guide to AI guardrails in production? For agentic products that take actions rather than draft text, this question is the whole dimension: an agent with tool access is a new privileged principal inside your perimeter and must be scoped like one.
• Where does the product sit in your architecture, and does the vendor's deployment model (SaaS only, VPC, on-premise) match the sensitivity of the workload? The layered view in the enterprise AI stack reference architecture is the map to score against.

What good looks like: written data-flow documentation, contractual training exclusion by default, guardrails the vendor can describe at the component level, and deployment options that match data sensitivity. Red flag: "your data is safe with us" delivered as a sentence rather than a document.

Dimension 4: Cost Structure Transparency (weight 15)

What it measures: whether you can compute, from the contract alone, the cost of one unit of your work, and whether that unit cost survives your realistic volume range. AI pricing in the current market is deliberately hard to forecast: seat prices with AI surcharges, credit systems with opaque burn rates, usage meters denominated in tokens the buyer cannot predict. Industry reporting through 2025 and 2026 has repeatedly described enterprises imposing per-user spend caps after agentic workloads overran inference budgets by multiples of the seat price.

Gate questions:

• Denominate the price in our unit of work: what does one processed claim, one resolved ticket, one drafted contract cost at our volumes? (Pass requires the vendor to do the arithmetic with you, in the meeting.)
• If usage doubles, what happens to unit cost, and where are the cliffs? Credits that expire, tiers that reset, and overage multipliers are the gaming surface; make the vendor walk each one.
• What percentage of the price is passthrough model cost, and when frontier token prices fall, as they have repeatedly, who captures the decline? A vendor whose margin grows silently as its input costs fall is charging you for a market trend.
• Is there a hard cap or kill switch on metered spend, controllable by you? A meter without a buyer-controlled cap is an unbounded liability, whatever the rate card says.

What good looks like: a worked unit-cost model at three volume scenarios, disclosed passthrough economics, buyer-controlled caps. Red flag: pricing explicable only by the vendor's own sales engineer, or credits whose conversion rate to work the vendor cannot state.

Dimension 5: Vendor Durability (weight 10)

What it measures: the probability that the company, the product, and its model supply chain all still exist, on acceptable terms, in three years. AI vendors carry two extra fragilities on top of ordinary startup risk: dependence on upstream model providers whose pricing and access terms can change abruptly, and exposure to the platform vendors bundling competing features into suites, the consolidation dynamic under way across enterprise software in the current cycle.

Gate questions:

• What is the model supply chain, and what happens if the primary provider raises prices, deprecates the model version, or becomes a competitor? A credible answer names a tested fallback, not a theoretical one.
• Runway and revenue durability: for a private vendor, months of runway and net revenue retention; for a feature-stage product inside a larger suite, the roadmap commitment in writing. (Pass requires numbers under NDA, or a parent-company commitment.)
• What is the single most concentrated dependency in the business: one model provider, one cloud, one anchor customer, one distribution partner? Every vendor has one; a vendor that claims none has not looked.
• If the company is acquired, what contractual protections travel with the contract: price locks, data return, continuity of the deployment model?

What good looks like: a tested multi-model fallback, disclosed financial durability, acquisition protections in the paper. Red flag: a single-model wrapper priced as a platform, or a roadmap that is one hyperscaler keynote away from being a feature of something you already own.

Dimension 6: Exit Cost (weight 15)

What it measures: the fully loaded cost of leaving, priced before signing. Exit cost in AI products is systematically underestimated because the visible artifacts, prompts and configurations, look portable, while the real accumulation is behavioral: workflows tuned to one product's failure modes, staff trained on its quirks, and downstream systems consuming its specific output format. The dynamics mirror frontier-model lock-in, where the binding constraint is operational rather than contractual.

Gate questions:

• What leaves with us: raw data, enriched data, fine-tuned artifacts, evaluation results, audit logs, in what format, at what cost, in what timeframe? (Pass requires a written data-egress commitment with a timeline.)
• What would a competitor need to reproduce today's quality on our workload, and how much of that is ours contractually? If the answer is "the accumulated tuning," that tuning is your asset being held on their platform.
• Can the product's outputs be consumed through an abstraction you control (an API layer, a schema you defined), so that a replacement slots in behind it?
• Termination mechanics: notice period, data destruction certification, transition assistance obligations, and price protection during any wind-down.

What good looks like: contractual egress with formats and timelines, outputs consumable through buyer-controlled interfaces, and a vendor untroubled by the questions. Red flag: exit terms that require "a conversation with your account team," which is a price quote you have agreed to receive later, under duress.

The Scoring Matrix

Score each dimension zero to four: zero, a failed gate (disqualifying); one, claims without evidence; two, evidence on vendor-chosen ground; three, evidence on buyer-chosen ground; four, evidence on buyer-chosen ground plus contractual commitment. Multiply by the weight, divide by four, sum to 100.

Reading the total: 80 and above, buy with confidence and negotiate from strength; 60 to 79, pilot with the specific weak dimensions as written conditions; 40 to 59, revisit in two quarters, because the product or the vendor is earlier than the pitch; below 40, decline. And the override rule once more, because it is the framework's spine: any zero disqualifies at any total. A 92-point vendor with a zero on Exit Cost is not a 92. It is a trap with excellent references.

In a live meeting, the matrix runs in about ten minutes: one gate question per dimension, chosen from the lists above, scored on the spot. The point of the exercise is not precision. It is that vendors sort themselves fast when the questions are deterministic, because the strong ones have answers ready and treat the questions as a signal of a serious buyer, while the weak ones negotiate the questions instead of answering them. How a vendor responds to being scored is itself evaluation evidence, on ground you picked.

Common Failure Modes in AI Buying

Four patterns account for most regretted purchases, and each maps to a skipped dimension.

Buying the demo is a Capability Fit failure: the product was scored on vendor ground, and the input distribution in production turned out to be the one the demo avoided. Buying the roadmap is a Vendor Durability failure: the contract paid current prices for future features, and the features arrived late, differently, or as a bullet in someone's acquisition announcement. Buying the bundle is a Cost Structure failure: the AI was "included" in a consolidation deal, which meant its cost was unmeasurable, which meant its value was never tested against an alternative; suites should be scored per-capability with this same matrix, and a capability that cannot be scored separately should be valued at zero in the negotiation. Buying the incumbent's toggle, accepting the AI addendum on a renewal because declining felt like friction, is an Evaluation Evidence failure: the addendum's quality was never measured at all, and the meter it introduced was never forecast.

The through-line is that none of these failures announce themselves at signing. They surface at month six, in the error queue, the invoice, or the deprecation notice. The framework's six dimensions are simply the six month-six surprises, asked about at month zero, on ground you picked.

Key Takeaways (Applied)

• Put the gate questions into the RFP verbatim, one block per dimension, and state the scoring rule in the RFP itself, including disqualification on any zero. Vendors calibrate honesty to what they know is measured.
• Build even a fifty-case evaluation set before the next AI purchase. It converts every future demo into a measurement and every future renewal into a negotiation.
• Score bundles per capability. A suite that cannot show a capability's standalone quality and cost has priced that capability for you already: zero.
• Re-score annually. AI vendors change models, pricing, and owners faster than any software category before them; a score is a photograph, not a portrait.
• Weight evaluation evidence above everything, because it is the one dimension that predicts the others. A vendor that measures itself rigorously tends to survive the other five dimensions; a vendor that does not has asked you to run its QA in production, and the invoice for that arrives with your logo on it.