The AI Vendor Smoke Test.

How to use this

You're evaluating an AI consultant or agency. You don't have a technical co-founder reviewing the deck with you. You need to figure out, fast, whether the person across the table is going to ship something useful — or sell you a six-figure platform you don't need.

Ask these three questions on the first call. Listen for the signals below each one. If two or three answers feel evasive, the vendor is more interested in the engagement than the outcome. Walk.

The three questions

1. 01

   “Can you walk me through something you built and what didn't work the first time?”

   A real builder has scars. A reseller has a deck. The first version of every real project is wrong about something — usually the shape of the data, the integration that won't behave, or the assumption that the user wanted what they said they wanted. If your vendor can't tell you what tripped them up, they either haven't built much or they're hiding the parts that didn't go well.

   +Listen for

   • +A specific, technical answer — not 'we underestimated the timeline.'
   • +A concrete decision they had to undo or rework, with the reason it failed.
   • +What they'd do differently next time, in plain language.
   • +Comfort talking about the messy middle, not just the launch screenshot.

   −Evasions

   • −A polished case study with no friction in it.
   • −Generic answers about 'communication' or 'expectations.'
   • −Pivoting to a sales pitch about their methodology.
   • −Inability to name the project, the client, or the stack used (even with NDA — they can describe shape).
   Follow-up

   “Was there a moment you considered telling the client to stop the project? What stopped you?”

   Quick score (1–3)

   Specific technical scar with a clear what-I-learned: 3. Vague but real: 2. Polished and frictionless: 1.

2. 02

   “If we did the smallest possible version of this, what would it look like?”

   The answer tells you whether they're solving your problem or selling their package. Real builders shrink scope to whatever ships first and proves the idea. Resellers grow scope because their margin is in the long engagement. If their first answer is a six-month, multi-system platform when you described a single workflow, they're not listening — they're packaging.

   +Listen for

   • +A version that ships in 2–3 weeks, not 3–6 months.
   • +Something deliberately narrow — one task, one user, one happy path.
   • +Honest tradeoffs about what you'd give up to ship that small.
   • +Excitement about shipping early and learning, not about the full vision.

   −Evasions

   • −An answer that's still bigger than what you originally described.
   • −'Discovery phase' as a $5k–$15k line item before any build.
   • −Vague platform language: 'we'd build a foundation for AI across your business.'
   • −Refusal to scope smaller because 'it wouldn't be worth it for either of us.'
   Follow-up

   “What would you intentionally leave out of that first version that we'd add later?”

   Quick score (1–3)

   Genuinely small, ships in weeks, with honest tradeoffs: 3. Smaller than original but still padded: 2. 'We don't really do small builds': 1.

3. 03

   “What would make you tell me not to do this?”

   Anyone who can't answer this has never said no to a client, which means they'll happily build you the wrong thing. Good vendors have a clear list of situations where AI isn't the answer, where the build doesn't ROI, where the timing's wrong, or where the team isn't ready. Bad vendors say yes to everything because every project is a paycheck.

   +Listen for

   • +Specific situations they've walked away from, with reasons.
   • +Honest assessment of when AI is overkill for the problem you described.
   • +Pre-conditions on your side (team capacity, data hygiene, leadership buy-in) before they'd take the build.
   • +Willingness to recommend a cheaper or non-AI solution if that's what fits.

   −Evasions

   • −'I can't think of one' or 'every project is unique.'
   • −Reframing the question as 'how do we manage risk' (deflection).
   • −Suggesting more scope as the answer to risk.
   • −Talking about budget instead of fit ('if you have the budget, we can make any project work').
   Follow-up

   “Have you ever told a client to wait six months and come back? What was the situation?”

   Quick score (1–3)

   A real example of saying no, plus pre-conditions for your build: 3. Theoretical 'we would say no if…': 2. Can't or won't answer: 1.