Before You Hire an AI Vendor: Do They Understand AI, or Just the Tools?

AI has created a new type of buying problem.

A few years ago, an organisation looking for technical support might have searched for a software developer, data scientist, cloud engineer or IT consultancy.

Today the market is filled with new titles:

Some of these people are highly capable.

Some have spent years working with machine learning, data, software engineering and production systems.

Others entered the market much more recently, learned how to connect a few AI tools together, and began selling “AI transformation”.

For an organisation trying to choose a supplier, the problem is obvious:

How do you tell the difference?

There is nothing wrong with building useful solutions using existing AI models.

In fact, most organisations should not be training large AI models from scratch.

But there is an important difference between someone who knows how to operate an AI product and someone who understands what is happening underneath it.

A serious AI practitioner should be able to explain, in plain English, concepts such as:

• Machine learning
• Deep learning
• Neural networks
• Training and inference
• Model evaluation
• Hallucination
• Embeddings
• Retrieval
• Context windows
• Fine-tuning
• Data quality
• Model drift
• Human oversight
• Privacy and security
• Monitoring in production

The vocabulary above describes what is happening inside the thing being sold. A supplier who cannot name the parts is describing the surface.

They do not need to turn your board meeting into a computer science lecture.

But if they are proposing a system that will interact with your customers, employees, documents or internal data, they should understand more than which buttons to press.

It is reasonable to ask an AI supplier whether they have ever trained or fine-tuned a model.

The answer tells you something about their technical depth.

But it should not become the only test.

A consultant does not need to have trained a billion-parameter foundation model to be competent.

For most business projects, the harder questions are often elsewhere:

That is the difference between a demonstration and a production system.

One of the simplest questions an organisation can ask is:

Tell us about an AI system you have deployed into a real operating environment.

Then keep asking.

• What did the system do?
• What data did it use?
• Who used it?
• How was access controlled?
• How was the model evaluated?
• What happened when the model was uncertain?
• Were the outputs logged?
• How was personal or commercially sensitive information handled?
• What human checks remained?
• How was the system monitored after deployment?
• What failed during testing?

A practitioner who has worked in production will normally have answers.

That is often a better signal than a polished demonstration.

You explain your business problem.

Within five minutes, the answer is:

We can build you an AI agent.

That should create another question:

Why?

• Why an agent?
• Why this model?
• Why this process?
• Why now?
• Why should it have access to this information?
• Why automate the whole workflow instead of one part of it?
• What measurable improvement are we expecting?

A credible supplier should sometimes tell you that AI is not the first thing you need.

The organisation may not have agreed what information employees are allowed to share with AI systems.

The same problem exists in AI training.

An impressive slide deck is not evidence that someone can build organisational capability.

Ask what people will be able to do differently after the training.

• Will employees understand the risks of entering sensitive information into public AI tools?
• Will managers know when AI output requires human review?
• Will staff understand the difference between a confident answer and a reliable answer?
• Will teams learn how AI fits into their actual jobs?
• Will leaders understand enough to challenge an AI supplier?

Or will everyone simply leave with a list of prompts?

Prompting has value.

But organisational AI capability is much bigger than prompting.

Good training should help people develop judgement.

This is particularly important when buying from a consultancy.

The person selling the project may not be the person building it.

Ask:

• Who will design the system?
• Who will write or review the code?
• Who will handle the data architecture?
• Who will test the AI behaviour?
• Who is responsible for security?
• Who deals with governance and privacy?
• Who supports the system after launch?

Then ask about their experience.

You are not trying to interrogate the supplier.

You are trying to understand who will be responsible when the PowerPoint becomes a real system connected to your organisation.

Many AI consultants themselves rely on third-party platforms, APIs, automation tools and foundation models.

Again, that is not automatically a problem.

Modern software is built on other software.

The important question is whether the supplier understands the dependencies they are introducing.

• If they outsource part of the work, who owns the code?
• Who has access to your data?
• Where is the data processed?
• What happens if an API changes?
• Can another supplier maintain the system?
• Can your organisation leave the relationship without rebuilding everything?
• What documentation will you receive?

Before appointing an AI training or implementation provider, ask:

1. 01What AI systems have you delivered in a real production environment?
2. 02What parts did you personally design or build?
3. 03How do you evaluate whether an AI system is reliable enough for its intended use?
4. 04What happens when the AI produces an incorrect or inappropriate output?
5. 05How will our data be accessed, processed, stored and protected?
6. 06What parts of this process should remain under human control?
7. 07What existing systems will this need to integrate with?
8. 08How will the system be tested, monitored and maintained after launch?
9. 09What will our own staff understand or be able to manage when the engagement ends?
10. 10What would make you advise us not to proceed with this AI project?

That final question is particularly revealing.

A supplier who can never imagine saying “do not build this yet” may be more interested in selling implementation than protecting your organisation.

Certifications, degrees and job titles can help establish credibility.

They are not substitutes for delivery evidence.

Equally, lack of a fashionable title does not make someone incapable.

Look for the combination:

And perhaps most importantly:

There is another side to this.

Even an excellent AI supplier can struggle inside an organisation that is not ready to make the necessary decisions.

Before you procure AI training, automation or implementation, ask:

• Do we know what problem we are solving?
• Do we have an internal owner?
• Are our people ready?
• Is our data suitable?
• Do we understand the risks?
• Do we know what success looks like?
• Do we know which decisions must remain human?

Vendor selection is therefore not only about whether the supplier is ready.

It is also about whether the organisation is ready.

The purpose is not to tell every organisation to buy AI.

It is to help you understand what needs to be true before you do.