The Vendor Verification Playbook

Start here

The Gap Between the Claim and the Reality

At a regulated organization, that gap is where confidentiality gets breached without anyone deciding to breach it.

A true story from the field. In one professional-services vendor pilot, anything not automatically redacted reached the vendor's servers and stayed there until deletion was requested. Default redaction caught roughly a third of sensitive content: a 67% leakage rate, discovered after security sign-off.

Not because anyone was careless. The information simply was never surfaced by the process, and what a process doesn't surface, sign-off can't catch. Closing the gap took sixty-plus custom redaction terms, built in-house, to bring leakage down to about 2%.

The diligence happened. It just happened after engagement instead of before. This playbook exists to reverse that order, permanently, with a standard that applies to every vendor identically.

The most dangerous phrase in AI procurement is "we reviewed it." The question is always: what did the review actually surface?

Part I · The Standard

Every Vendor, Every Time, No Exceptions

Not for familiarity. Not for relationships. Not for how well anyone knows someone there.

The standard has four legs, and a vendor engagement stands only when all four hold:

Independent risk and compliance assessment before pilot data ever flows. Independent means conducted by someone without a stake in the engagement proceeding. If the person championing the vendor also runs the assessment, you don't have an assessment. You have an endorsement with paperwork.
Data-handling claims verified in contract language, not conversation. Retention, residency, training use, subprocessors, deletion. If it isn't in the agreement, it isn't a commitment. It's marketing.
Terms of art keep their industry meaning. Zero data retention means zero data retention. When a claimed capability turns out to mean something else, that is not a misunderstanding to be explained. It is a red flag to be escalated.
Your own technical validation. Test, don't take the tour. Send controlled test data through the system and observe what actually happens to it, where it lands, and what the defaults do.

The relationship clause:

The process does not change based on who the vendor is or who they know. A vendor with a personal connection inside your organization gets the same assessment, the same verification, the same testing. Connections are how vendors arrive. They are never how vendors get verified.

Part II · The Checklist

The Questions to Ask Every AI Vendor

In writing, before the pilot, with the answers bound into the agreement.

Data handling

What exactly do you retain, for how long, and where? Break the answer out by prompts, outputs, files, embeddings, and logs. "We don't retain data" without that breakdown is not an answer.
Where does our data physically reside, and can residency be guaranteed by contract?
Is our data, or anything derived from it, used to train or improve any model, yours or a third party's? Is that commitment in the agreement, and does it survive your acquisition?
Who are your subprocessors, and what happens to our data on their infrastructure? Every hop counts.
What is your deletion process, what is the SLA, and how is deletion evidenced? "Deleted upon request" means retained until we remember to ask.

Security posture

Which certifications do you hold, at what level, and how current are they? A SOC 2 Type I says the controls were designed. A Type II says they operated over time. The difference matters.
When was your last independent penetration test, and can we see the summary?
What is your breach history, and what is your contractual notification window if our data is involved?
What do your default settings do? Not the configured ideal. The defaults. Most leakage lives in the gap between the two.

Architecture

Where does inference actually happen: your cloud, a hyperscaler, or on-premises options?
What is logged, at what verbosity, retained how long, and visible to whom, including your own support staff?
If you claim zero data retention: what is its exact scope? Prompts? Outputs? Attachments? Metadata? Telemetry? ZDR that covers prompts but not logs is not ZDR.
What happens to in-flight data if the service fails mid-request?

Legal and relationship

Does the contract language match the sales deck? Wherever they differ, the contract is the truth and the deck is the aspiration.
Will you execute a data processing agreement with our required terms, or only your paper?
What indemnification do you offer if your handling of our data causes a breach of our obligations to our clients?
Are there any existing relationships between your company and any of our personnel? Disclosed conflicts are manageable. Discovered conflicts are incidents.

Definitions Are Not Negotiable

The glossary that keeps vendor conversations honest.

Term	What it must mean
Zero data retention	Nothing persists after the request completes. Not prompts, not outputs, not embeddings, not "temporarily for quality purposes." Zero means zero, and the scope is stated in writing.
Data residency	A contractual guarantee about where data lives, including backups and subprocessors. A preference or a default region setting is not residency.
"We don't train on your data"	Covers the model, fine-tunes, evaluations, and derived artifacts, for the vendor and every subprocessor, in perpetuity. Anything narrower gets named for what it actually covers.
Anonymized	Cannot be re-identified by any party with any auxiliary data. If it can be reversed with a lookup table, it is pseudonymized, which is a different promise.
Encrypted	Specify at rest, in transit, and, where claimed, in use, with key ownership stated. "Encrypted" with vendor-held keys answers a different question than customer-managed keys.
SOC 2	Type II, current period, relevant trust criteria, and you have read the exceptions section. The logo on the website is not the report.

When a vendor redefines a term of art under questioning, the redefinition is the finding.

Part III · The Red Flags

A Field Taxonomy

Each of these has been observed in the wild. Each one is an escalation, not a conversation.

The redefinition

A claimed capability turns out to mean something other than the industry term, and the response to being caught is a long explanation of why the term "really" means the narrower thing. The explanation is the confession.

The tour

Every question is answered with a demo, a diagram, or a call with a solutions engineer, and never with contract language. Tours are choreography. Test data is truth.

The familiarity lever

Pressure to shortcut the process because someone inside your organization vouches for the vendor. The stronger the personal connection, the more the standard process protects everyone, including the person vouching.

The default gap

The secure configuration exists but is not the default, and nobody mentions that until you find it. Vendors are responsible for their defaults. You are responsible for testing them.

The retention shrug

"Data is deleted upon request." Translation: your data persists indefinitely on their infrastructure until you affirmatively remember to ask, with no SLA and no evidence of deletion.

The paper mismatch

The sales materials promise what the contract declines to say. If the vendor won't write the claim down, the vendor doesn't believe the claim.

The acquisition silence

No answer for what happens to your data and your terms if the vendor is acquired. In AI, assume acquisition. Your protections should survive it by contract.

Part IV · The Workflow

Verification as a Lifecycle

Before the pilot, during the pilot, and for as long as the vendor holds your data.

Before pilot data flows

Independent risk and compliance assessment completed and documented.
Question checklist answered in writing; answers attached to the agreement.
Definitions confirmed against the glossary; any redefinition escalated.
Conflicts of interest disclosed and recorded.
Technical validation plan written: what test data, what observations, what pass criteria.

During the pilot

Controlled test data only until validation passes. Never production client data in a pilot's first phase.
Defaults tested as shipped, then hardened, then re-tested.
Redaction, filtering, and logging observed empirically, with rates measured, not assumed.
Findings documented while fresh, including what the process failed to surface and why.

Ongoing

Annual re-verification: certifications current, subprocessor list current, terms unchanged.
Vendor changelogs and policy updates monitored. A quiet terms-of-service change is a new engagement decision.
Deletion exercised and evidenced at least once. A deletion process that has never been tested is a hope.
Exit plan on file: how your data comes back, how it dies on their side, and how long that takes.

Language That Binds

Plain-language starting points for the clauses that matter. Adapt with counsel; the point is that each claim becomes an obligation.

Retention: "Vendor shall not retain Customer Data, including prompts, outputs, files, embeddings, and logs, beyond completion of each processing request, except as expressly listed in Schedule X with stated durations."
Training: "Neither Vendor nor any subprocessor shall use Customer Data, or any data derived from it, to train, fine-tune, evaluate, or improve any model or system. This obligation survives termination and any change of control."
Residency: "Customer Data shall be processed and stored exclusively within [region], including backups and subprocessor infrastructure."
Deletion: "Upon request or termination, Vendor shall delete all Customer Data within [X] days and provide written certification of deletion, including from backups within [Y] days."
Notification: "Vendor shall notify Customer of any incident affecting Customer Data within [X] hours of discovery."
Audit: "Customer may verify Vendor's compliance through [mechanism] no more than [frequency], and Vendor shall provide current certifications and subprocessor lists upon request."
Conflicts: "Vendor represents that it has disclosed all existing relationships between Vendor personnel and Customer personnel, and shall disclose any that arise."

None of this is exotic. All of it is routinely signed by vendors who mean what their marketing says. Reluctance to sign is information.

The One-Page Gate

The five questions that decide whether a pilot proceeds. If any answer is no, the pilot waits.

Has an independent assessment been completed by someone with no stake in the outcome?
Are all data-handling claims written into the agreement, in the glossary's definitions?
Have the defaults been tested with controlled data, and did the observed behavior match the claims?
Have all personal relationships between vendor and organization been disclosed and recorded?
Does anyone involved have an unanswered question they've been reluctant to ask out loud?

That last question catches more risk than the other four combined. Build a process where asking it is safe, and the process will protect you. Build one where it isn't, and no checklist will.

Notes & Provenance

Field case (vendor pilot redaction findings, 67% to ~2%): BlockBrain Labs deployment observation, anonymized.
Certification distinctions (SOC 2 Type I vs. Type II) per AICPA trust services criteria, summarized in plain language.
Contract language provided as plain-language starting points for discussion with counsel; not legal advice.
Companion framework: The AI Rollout Playbook, Part II, "The vendor standard."