How to Evaluate an AI Vendor Without Getting Burned

A restaurant group owner we spoke with signed a $60,000 contract for an "AI-powered CRM" last year. The sales pitch was polished — predictive customer analytics, automated follow-ups, personalized marketing campaigns generated by their "proprietary AI engine."

What he actually got was a Salesforce skin with ChatGPT plugged into the email templates. The "proprietary AI engine" was an API call. The "predictive analytics" was a dashboard that showed which customers hadn't visited in 30 days — something his existing POS system already tracked.

Sixty thousand dollars. Twelve-month contract. He found out what he'd bought about three months in, when he asked a developer friend to look under the hood.

This story isn't unusual. The AI vendor landscape right now is full of legitimate companies building genuinely useful tools. It's also full of companies wrapping a thin layer of AI around commodity software and charging enterprise prices for it. If you're not technical, telling the difference is hard.

Here's how to do it.

The Red Flags

These aren't guarantees that a vendor is dishonest. They're patterns we've seen consistently in the engagements that go badly. Any one of them is worth a pause. Two or more is worth walking away.

They Won't Demo With Your Data

This is the biggest tell. A vendor who demos their product with pre-loaded sample data is showing you a controlled environment. Everything works in a controlled environment.

What you need to see is the product working with your messy, real-world data. Your customer records with the misspellings and duplicate entries. Your invoices with the inconsistent formatting. Your scheduling system with the overlapping appointments and the clients who reschedule three times.

If a vendor says "we'll load your data after you sign," ask yourself why. The answer is usually that their system breaks on real data and they'd rather have your signature before you find out.

A good vendor will do a pilot with a small subset of your actual data — even if it's just a dozen records — to prove the system works with your specific situation. That's a sign of confidence. Refusing to do it is a sign of something else.

They Say "Proprietary AI"

There are exactly three companies building the large language models that power nearly all AI applications right now: OpenAI (GPT), Anthropic (Claude), and Google (Gemini). A few others like Meta and Mistral play in specific niches. That's the field.

When an AI vendor tells you they've built their own "proprietary AI" or "custom-trained model," what they almost always mean is that they're using one of these existing models with a layer of custom instructions on top. That's not proprietary AI. That's a system prompt. And system prompts are useful — the instructions matter — but they're not worth a premium price tag.

Ask directly: "What foundation model does your system use?" If they give you a straight answer — "We use Claude for document analysis and GPT-4 for customer interactions" — that's a good sign. They understand their own stack and they're not trying to hide it. If they dodge the question or insist it's "proprietary," be skeptical. The cost of the underlying model is a significant factor in your ongoing expenses, and you deserve to know what you're paying for.

They Don't Talk About Ongoing Costs

The upfront price is never the full price with AI. Every AI system has running costs — API fees, hosting, maintenance, updates. A system that costs $30,000 to build might cost $500 or $5,000 a month to operate, depending on how it's architected.

A good vendor explains this clearly. They'll model your expected usage, estimate monthly costs, and show you how those costs scale as your usage grows. They might even help you understand the cost levers so you can make informed decisions about what features are worth the ongoing expense.

A bad vendor quotes you a build price and hopes you don't ask about operations until the first invoice arrives. We've talked to business owners who were quoted $25,000 for an AI build and then discovered they were spending $3,000 a month in API costs — more than the build cost in the first year alone.

They Promise "It Learns on Its Own"

This claim is technically true in a narrow sense and wildly misleading in practice.

Some AI systems do improve over time. They learn from user corrections, adapt to patterns in your data, and get better at specific tasks. But this doesn't happen by magic, and it doesn't happen without oversight.

An AI system that "learns on its own" can also learn the wrong things. It can pick up biases from skewed data. It can optimize for the wrong metric. It can confidently repeat a mistake it was never corrected on.

The vendors who say "it learns on its own" are usually describing a system where nobody has built the feedback loop to verify what the AI learned. That's not a feature. That's a missing feature.

Ask instead: "How do we review what the system has learned? How do we correct it when it's wrong? Who monitors the quality of its outputs over time?" Those are the questions that separate a system that gets better from a system that gets worse without anyone noticing.

Their Timeline Is Vague

"We'll have your system up and running in 6 to 12 weeks." That range — 6 to 12 — is a yellow flag. A vendor who knows their product and understands your requirements can give a tighter estimate than that. A 2x range in the timeline usually means they haven't scoped the work properly.

Even worse: "It depends on the complexity of your needs." If they can't estimate after a discovery conversation, they either don't understand what they're building or they're planning to expand the scope (and the invoice) after you've committed.

Get milestones. Get dates. Get it in writing. A vendor who can't give you a timeline probably can't give you a budget either.

The Green Flags

These are the signs that a vendor knows what they're doing and respects your position as a non-technical buyer.

They Show You a Working Prototype

Not a mockup. Not a slide deck. A working system that does the thing they're proposing to build for you. Ideally with some version of your data in it, even if it's limited.

The best vendors we've seen build a small proof of concept during the sales process — sometimes free, sometimes for a modest fixed fee. It proves two things: they can build what they're selling, and the technology actually works for your use case.

If a vendor is willing to put working code in front of you before you've signed a contract, that's a strong signal. They're confident enough in their work to let you evaluate it without commitment.

They're Transparent About What AI Model They Use

"We use Claude for the analysis layer and a smaller model for routine classification tasks. The Claude calls cost about $0.03 each and we route the simpler work to a model that costs $0.001 per call. Based on your volume, we estimate $300-400 a month in API costs."

That's what transparency sounds like. They know their stack, they know the costs, and they're showing you the math. This is the kind of vendor who's thinking about your operational costs, not just their project fee. It's the same mindset behind building AI systems that don't drain your budget.

They Discuss Maintenance Upfront

AI systems aren't set-and-forget. Models get updated. APIs change. Your business needs evolve. A vendor who includes a maintenance plan in the initial proposal — what it covers, what it costs, how often they check in — is planning for the long term.

A vendor who says "we'll cross that bridge when we get there" is planning to charge you emergency rates when the bridge collapses.

Ask about their maintenance model before you sign. Monthly retainer? Per-incident? Annual review? Whatever it is, get it documented. The businesses that get the most value from custom software are the ones who planned for the whole lifecycle, not just the launch.

They Have References You Can Actually Call

Not testimonials on their website. Not case studies with anonymized company names. Real clients, at real businesses, who you can call and ask: What was the experience like? Did it come in on budget? Does it still work? Would you hire them again?

A vendor who won't connect you with a single past client either doesn't have happy past clients or doesn't have past clients at all. Both should worry you.

Questions to Ask Before Signing

Print this list. Bring it to the meeting. Any reputable vendor will welcome these questions.

About the technology:

What AI model(s) does your system use?
How does the system handle errors or incorrect outputs?
What happens if the AI model provider changes their pricing or discontinues the model?
Can we switch models later without rebuilding the system?

About the costs:

What's the total build cost, and what's included?
What are the estimated monthly operating costs at our current volume?
What do those costs look like if our usage doubles? Triples?
Are there any per-user, per-seat, or per-transaction fees?

About the data:

Where is our data stored?
Who has access to it?
Is our data used to train or improve AI models for other clients?
What happens to our data if we cancel?
How do you handle security concerns specific to AI systems?

About the timeline:

What are the specific milestones and delivery dates?
What's the process for handling scope changes?
What does the testing process look like before go-live?

About the relationship:

What's your maintenance and support model after launch?
Can you provide two or three references I can contact?
What does the contract look like if we want to leave?
Who owns the system and the data if we part ways?

A good vendor will answer every one of these without hesitation. They've been asked before. They expect it.

A bad vendor will get uncomfortable around questions three and four in the costs section and will redirect the conversation to features. Pay attention to what they avoid.

The Vendor Landscape Right Now

The AI vendor market in 2026 is roughly where the web development market was in 2005. There's enormous demand, a shortage of experienced practitioners, and a flood of new entrants who learned just enough to be dangerous.

You'll encounter three types of vendors:

Experienced software shops that have added AI capabilities to their existing practice. These are usually the safest bet — they understand software development, they've managed client projects for years, and they've learned the AI piece on top of a solid foundation.

AI-native startups that formed specifically to sell AI tools. Some of these are excellent — focused, innovative, deeply technical. Others are two people with a ChatGPT API key and a marketing budget. The questions above will help you tell the difference.

Offshore AI factories offering AI development at a fraction of the cost. Some deliver good work. Many deliver what you pay for. The challenge is always communication, maintenance, and accountability when something goes wrong six months after delivery.

Regardless of which type you're evaluating, the same principles apply: see it work with your data, understand what's under the hood, know what you'll pay every month, and talk to someone who's been through it before you.

The Cheapest Mistake Is the One You Don't Make

The restaurant owner who spent $60,000 on a ChatGPT wrapper eventually found a vendor who built what he actually needed for $8,000. The system uses real AI integration — predictive ordering based on reservation data, automated review responses, and personalized marketing triggered by actual customer behavior patterns. It works. It cost one-seventh what the first attempt did.

The difference wasn't luck. It was asking the right questions and knowing the red flags.

AI can genuinely transform how a small business operates. The tools available today are more powerful and more affordable than anything that existed two years ago. But the power of the tools makes the choice of vendor even more important. A great AI system built by the right partner is one of the best investments a business can make. A mediocre system built by the wrong partner is one of the most expensive.

Take your time. Ask the questions. See the demo. Call the references. The right vendor will earn your business. The wrong one will just try to close it.

Blue Octopus Technology builds AI tools for businesses that don't have engineering teams. We answer every question on that list — and we'll give you references who'll tell you the unfiltered version. If you're evaluating AI vendors and want a straight conversation, let's talk.