Don’t Be Blinded by “80% Accuracy” How to Scrutinise Predictive AI Sales Pitch

I have sat through quite a few AI and predictive analytics pitches lately.

Many sound impressive. Advanced algorithms, sector expertise, clever dashboards and confident claims like “80% accuracy”.

But that number can be misleading.

In social housing, many of the issues we want to predict are naturally imbalanced. Damp and mould, arrears, tenancy risk and safeguarding concerns may only affect a smaller proportion of homes or customers at any point in time.

So if 80% of cases have no issue, a model that always predicts “no issue” could still appear to be 80% accurate.

That sounds good on paper. In reality, it may be almost useless.

Accuracy Is Not Enough

When assessing a predictive model, we need to look beyond the headline number.

The better questions are:

1. What is the baseline?

2. How balanced is the data?

3. What are the precision and recall scores?

4. How was the model validated?

5. Has it been tested on our data?

6. Can the results be explained clearly?

7. What action will we take when a case is flagged?

Precision and recall matter because they tell us different things.

Precision tells us how many of the cases flagged are actually relevant.

Recall tells us how many of the real cases the model successfully finds.

Both matter.

A model that creates too many false alarms can overwhelm teams. A model that misses too many real cases can fail the customers it is meant to help.

Ask Better Questions

It is a bit like buying a car.

You would not settle for “it drives fast”. You would ask about reliability, safety, running costs and how it performs in real conditions.

Predictive models deserve the same scrutiny.

This is not about being sceptical for the sake of it. It is about due diligence.

AI and machine learning have real potential in housing. Used well, they can help us spot risk earlier, target resources better and make better decisions.

But a prediction only has value if it leads to a useful action.

If a model produces a list that no one trusts, understands or has capacity to act on, it will not deliver impact.

Why It Matters

For not-for-profit housing providers, every decision matters.

We have finite resources. Every pound spent on technology needs to deliver value for customers, colleagues and communities.

Our role is not just to deliver insights. It is to make sure they are robust, transparent and trustworthy.

So yes, let’s embrace innovation.

But let’s also ask the right questions.

Because “80% accurate” is not enough.