AI in engineering companies · The five levels of AI value

Closing the loop: letting field data change the product

Catherine Ives-YimPublished 27 September 2026 · 6 min read

Early in my career, I was asked to look at a service department that was quietly draining hundreds of thousands a year from a company's margin. The information that explained it was all there, in job sheets and spares records, and nobody had put it together. When we did, the causes were obvious, including a small number of clients whose cost to serve far exceeded what their contracts paid.

I have seen versions of that story in almost every organisation since. The product is telling the company what is wrong with it, through service calls, returns, warranty claims and, increasingly, telemetry. The company is not listening, not through lack of interest, but because nobody has the time to read it all and connect it to engineering.

The fifth and top level of AI value in an engineering company is about closing that loop.

What "learn" means

At this level, AI continuously reads what comes back from the field and turns it into engineering action. The loop has six steps:

Detect. Spot an emerging pattern in service records, returns and telemetry before it becomes obvious.
Attribute. Link it to what the affected units have in common: a batch, a supplier, a firmware release, a region, an operating condition.
Investigate. Assemble the evidence and draft the investigation.
Fix. Engineering decides and implements the change.
Verify. Check the field data after the fix is released, to confirm the problem has actually stopped.
Remember. Feed what was learned back into design reviews, requirements and the organisation's searchable knowledge, so the next product does not repeat it.

Steps 1 to 3 and 5 are where AI does the heavy lifting. Step 4 remains engineering's. Step 6 is where most organisations lose the value they have already paid for.

Our fault at level 5

Following the example used throughout this series: a fault that keeps appearing in a connected product in the field.

At the lower levels, the organisation got better at handling it once reported. At level 5, it is noticed early. The AI's weekly digest of service records flags that a particular symptom has risen from background level on one product variant. It attributes the rise to units with a specific component batch running the latest firmware. It drafts the investigation, lists the affected units, and estimates how many more are likely to report it. Engineering finds the cause in days. After the fix is released, the AI tracks the fault rate among units that received it against those that have not, and reports whether it has worked.

The fault is caught after the fifth report instead of the fiftieth, and the fix is confirmed by evidence rather than assumed.

Why every level below is needed

Level 5 is the sum of the ladder. Detection needs the service records to be readable and the knowledge searchable, which is level 1. Attribution needs faults, units, batches and firmware versions reliably linked, which is level 2. Investigation needs live telemetry and records brought together, which is level 3. Drafting the investigation and the resulting documents is level 4. Verifying a fix needs to know which units received it, which depends on all of them.

This is why I have written about the levels in order. Level 5 is where the value is largest, but it cannot be bought on its own.

Getting the statistics right

This is where good intentions most often go wrong, and it is worth being precise.

Normalise by the fleet. A new product generates more reports simply because more of it is being sold. Fault counts mean little without the number of units in service, and how long they have been in service.

Beware small numbers. Five reports from a new variant may be a real signal or chance. The system should show its uncertainty, not just a trend line.

Watch for reporting bias. Customers in some markets report more readily than others. A change to the support process can change the number of reports without any change in the product.

Tune the alarms. An early-warning system that raises too many false alarms is soon ignored. One that raises too few is pointless. Expect to adjust the thresholds as you learn.

AI is very good at reading free text at volume and spotting clusters. The statistical judgement about what is real still needs someone who understands both the data and the product.

Why this matters

Warranty and service cost. Faults caught early affect fewer units. For many product companies this is the most direct financial return AI will deliver.

Safety and regulation. Product safety law and the EU Cyber Resilience Act both expect manufacturers to monitor their products in use and act on what they find. A working level 5 loop is the practical way to show that you do.

Product quality. Organisations that learn from the field improve faster than those that do not, and it shows in their next product.

Evidence instead of opinion. Verifying fixes against field data ends the familiar argument about whether a change actually worked.

The organisational part

Level 5 is where technology stops being the limiting factor. An early warning is worthless if nobody owns responding to it.

It needs a clear owner for each alert, a regular point where service, quality and engineering review what the field is saying together, and the authority to act. It needs engineering to accept evidence from the field even when it is uncomfortable. And it needs the lessons to be captured in a way the next project will actually read.

AI can make the information impossible to ignore. It cannot make an organisation act on it.

How to start

Start with a digest, not an alarm system. Have AI read the last month of service records and produce a summary: the most common faults, what changed since the previous month, and anything unusual. Review it with service, quality and engineering together.

When the monthly digest is trusted, make it weekly. When the weekly digest has shown its worth, add automated alerts for specific patterns. Build towards verification by recording which units receive each fix. I built exactly this kind of daily, weekly and monthly digest for an environmental monitoring system I designed, and the pattern transfers directly: regular, readable summaries earn trust before alarms do.

Four things worth taking seriously

For boards and senior leaders: ask how long it takes, on average, from the first field report of a new fault to engineering confirming its cause. That number is the measure of your loop.

For quality and service leaders: your records are the organisation's best source of engineering truth. Level 5 turns them from a cost into an asset.

For engineering leaders: insist that every fix is verified against field data. It is the only way to know.

For anyone starting on this journey: you do not need to reach level 5 to benefit. Every level on the way pays for itself. But knowing where the ladder leads is what makes the first steps worth taking properly.

This is the last article in this series on the five levels. The next series looks at agents, the means by which the upper levels actually get built, beginning with What an agent actually is, and isn't. It picks up this same field fault again in Agents at work in engineering.

I would be interested to hear how your organisation hears what its products are saying, and how quickly it acts.