Three essays so far. So what connects them?
In thirty years around mines, plants, and refineries, I have never heard an operations manager ask what architecture a model uses, or get excited about self-attention. The questions are always the same:
Can I act on this?
Will it help?
Will it hold up in the audit?
Who is accountable if it’s wrong?
Those are the questions the last three pieces were each answering, from different angles.
Industrial AI has never been more capable, and never more in focus. Foundation models, industrial agents, digital twins, optimization: all advancing fast, all being pushed to the plant, as firms move from lighthouse proofs-of-concept toward platform thinking and OT vendors acquire industrial AI firms to keep up.
But will it last, or is this another hype cycle?
Every plant is trying to do the same things it always has: produce more, operate more safely, and use less doing it — higher uptime and utilization, a stronger health, safety, and environment posture, less energy and resources spent per unit of output.
AI will matter only if it helps make those trade-offs better, through decisions grounded in data and domain knowledge, not just fluent ones. The answer will not come just from how capable AI becomes. We already have enough capability to change industrial operations, and it’s becoming more efficient every day.
Capability is not the bottleneck. Trust is.
Trust is the willingness to commit resources, responsibility, or action on the basis of a decision whose uncertainty cannot be eliminated.
Trust cannot be mandated. It cannot be assumed. It has to be engineered. Whether asked of a human or a machine, Can I trust you? has only one acceptable answer in operations: Yes — and here is the evidence, or a track record you can inspect. That answer comes out of engineering, not wishing. That is the premise of this series: Engineering Trustworthy Industrial Decisions in the Age of AI. Two of those words carry all the weight.
Trust is a discipline, not a hope. Engineering exists to turn desirable properties into demonstrable ones — reliability, safety, availability, maintainability, cybersecurity, and now trust. These are not aspirations, but properties that can be specified, designed, measured, tested, and continuously improved.
Every mature engineering field treats its hardest property as something you design in, verify, and evidence — not something that emerges if the team is careful. Process safety engineers do not hope a plant is safe; they run the HAZOP, layer the protections, and audit them.
Industrial AI, today, mostly hopes. Trust is handled with a disclaimer in the footer, a dashboard nobody audits, a pilot that never has to survive contact with operations, or just mandating a human in the loop — which either slows things down or exists more in letter than in spirit. The anti-pattern is always the same: trust treated as something that somehow gets done, downstream of the “real” work of building the model.
Calling this engineering is a commitment: trust gets methods, systems, evidence, and tests — the same machinery every other engineering discipline uses for the properties that matter.
“Trustworthy” means nothing until it is operationally defined. Here is the bar this program works to:
An industrial AI system is trustworthy when the engineers and operators who depend on it can rely on it the way they rely on any other piece of engineered equipment. Concretely, that means it is:
Notice what appears on neither list: model accuracy or token efficiency. They matter. But they are input to trust, not trust itself.
You cannot buy a trustworthy model. You can only engineer a trustworthy system around whatever models you use.
One clarification before going further. This is not “responsible AI,” and it is not “AI safety.” Those are real and valuable lenses — one rooted in policy and ethics, the other in the behaviour of frontier models. This is an engineering lens, for operational settings where the consequences are physical, the standards are binding, and the accountability is legal.
Industrial decisions cross the digital-physical boundary. A recommendation here doesn’t stay a suggestion on a screen: it changes a pressure, a temperature, a maintenance schedule, a safety barrier. A chatbot that answers wrongly wastes a minute; an AI that confidently recommends deferring the wrong inspection can cost lives, licenses, and livelihoods.
Once AI crosses into the physical world, engineering — not prompt engineering — becomes the governing discipline. In industrial operations, a confident wrong answer is categorically worse than a refusal — which inverts the optimization target most AI systems are built around.
The sharpest one-line test I know for whether an AI-assisted decision is operationally ready is this: “Who signs?” If responsibility for the decision cannot be clearly assigned to someone willing to put their name on it, the system is not ready — no matter how good the model is.
You cannot engineer what you have not decomposed. Three dimensions make a decision itself trustworthy: what the AI must know, how it must reason, how it decides.
A. Engineering Knowledge — what the AI knows.
Neglected, you get fluent answers unanchored to the standards, datasheets, and failure history that govern the asset — expert-sounding, knowing nothing that matters. Mining engineering taught the regulations alongside the engineering itself: knowing how to tunnel deep isn’t enough; you must know what’s allowed. Those rules aren’t legal overhead — they encode decades of experience about which risks a mine may take, and which it never may.
B. Grounded Reasoning — how the AI reasons.
Skip this one and correlation gets dressed up as causation — recommending actions that violate physics, standards, or failure mechanisms any SME could recite from memory. We don’t accept that reasoning from humans either: nobody runs a refinery on gut feel, not with lives and billions in equipment on the line. Standards bodies publish procedures for the same reason medical councils prescribe protocols. AI needs that same discipline — grounded data, proven reasoning paths, the right answer repeatably, not just once.
C. Trustworthy Decisions — why engineers should trust the decision.
Neglected, you get recommendations nobody can sign: no auditable path from evidence to action, no answer when the incident review asks why. Most governance stops at canonical data and model lifecycles, never asking whether the recommendation itself is correct, predictable, repeatable. Deterministic expert systems gave that assurance but couldn’t scale; AI scales the recommendation surface enormously, in settings where safety and regulation raise the stakes.
Take a corrosion-under-insulation mechanism near a steam trap: Engineering Knowledge is knowing the mechanism and the standard that governs it. Grounded Reasoning ties the signals to that mechanism, not just the pattern. Trustworthy Decisions is the auditable case for stripping insulation at that exact span.
What happens after a trustworthy decision is made — how to turn it into something operational and scalable — is a separate arc, and a story for a future essay.
One objection is probably already forming. It deserves an answer before moving on.
A fair challenge: three dimensions, and not one of them says “build better models.” Are we ignoring the actual AI?
No. Models matter enormously — they simply do not occupy the level of abstraction where industrial trust is engineered. Structural engineering is not metallurgy, but no structural engineer is indifferent to steel: the discipline specifies the steel it needs, verifies what it receives, and takes responsibility for everything it builds with it. That is exactly the relationship this framework has with AI models. Frontier model capability is treated as a supplied input, like structural steel. It is a deliberate engineering choice, and it comes with obligations: specify what you need, test what you get, and never build trust on a component you have not verified.
Why this distinction is important: when an AI initiative underperforms, the instinctive response is to swap the model — a new architecture, a new vendor, a different benchmark. Ask instead what is actually constraining the result. It is not always the model. It may be a different constraint somewhere else. An industrial AI program should be evaluated by its impact on that constraint, not by a benchmark score on the model behind it.
This isn’t a solitary conviction. DNV, the classification society that certifies ships, platforms, and pipelines, is asking the same starting question. So is industry itself: Vimal Kapur, Honeywell’s chairman and CEO, said almost the same thing in a 2026 interview — industrial AI isn’t fundamentally a model problem, it’s a knowledge problem.
Trust is not a feature of the next model. It is an engineering discipline — and it is time we practiced it that way.
If you build, operate, or sign for industrial AI systems, this series is for you — and it will be better with your disagreement than with your applause.
Subscribe via RSS to follow along, or start with the canonical terminology that everything else builds on.
]]>The actual cause had nothing to do with either motor. Motor A sat on the east side of the installation and caught the sun first; Motor B, further west, caught it a few hours later. Both were heating up from the same source — the day’s solar load — just offset by geography. Fix Motor A, and Motor B keeps overheating right on schedule. The correlation was real. The causation it seemed to indicate was not.
Every plant has a dashboard, and every dashboard is primarily a set of data views, and most of those views are post-mortem: they show what already happened. There is a pattern-matcher behind it now beyond the usual post-mortem views. Show it enough history, and it will find things that move together. Corrosion rates and steam-trap proximity. Vibration and ambient temperature. Throughput and a dozen variables - that may be correlated, but not related.
Some of those correlations are real mechanisms. Most are coincidence wearing a p-value. The dashboard can’t tell you which is which. Neither, reliably, can the model behind it, not because the model is bad, but because correlation was never built to answer the real question: why this happened.
Here is the distinction the rest of this piece rests on, and it fits in a sentence: correlation tells you what moves together; causation tells you what happens if you reach in and change one thing.
Say a model notices that pipe spans near steam traps corrode faster. Correlation stops there — a pattern, filed and dashboarded. Causation asks the next question: why? In this case, engineering already knows the mechanism, and it has a name — corrosion under insulation, driven by moisture ingress where a leaking trap wets the lagging. That mechanism is what tells you where to strip insulation and inspect, not the correlation. The correlation and the mechanism can point at the same pipe spans and still license completely different actions.
Judea Pearl gave this distinction a ladder: association at the bottom (what do I observe?), intervention in the middle (what happens if I act?), counterfactual at the top (what would have happened if I hadn’t?). Most industrial AI today lives on the bottom rung by construction — it was trained to predict, not to intervene. A plant, on the other hand, is run almost entirely on the top two rungs. Every maintenance decision is an intervention. Every root-cause investigation is a counterfactual. An AI that only climbs to the first rung is answering a different question than the one being asked of it.
A related trap, specific to how these systems are usually built: point a large language model at a historian — the time-series database every plant already runs — and let it retrieve and summarize. It will produce fluent, plausible-sounding operational narrative. It has still done nothing but retrieval.
Retrieval is not grounding. A historian records what happened; it does not encode why, and it certainly does not encode what the relevant standard permits, what the failure mechanism actually is, or what limits apply before a recommendation is safe to act on. An AI that reads the historian well can sound like it understands the plant. Sounding like it and grounding in it are different claims, and only one of them survives an audit.
There is a reason this problem does not get solved the way it gets solved in software: you cannot move fast and iterate on a safety-critical system. Consumer AI can be wrong at scale and cheaply corrected — ship, observe, retrain. An industrial system that touches a pressure boundary or a safety-instrumented function does not get that feedback loop. You do not learn a blowout preventer’s failure modes by occasionally letting it fail.
Which means the causal structure has to come from somewhere other than trial and error at the point of use. It has to be grounded before deployment, not learned in production — in physics, in the standard that governs the equipment, in the failure history the industry has already paid for in incidents. Grounded reasoning is, in part, an admission that some domains cannot afford to learn the way machine learning usually learns.
None of this argues against using AI to help build the causal picture — only against trusting an unreviewed one.
A live, useful pattern in the field right now: AI proposes candidate causal relationships from data and failure history at build time, and a subject-matter expert reviews and certifies them before anything reaches an operator.
AI is fast at hypothesis generation. The standard, and the engineer who knows it, are what turn a hypothesis into something worth grounding a decision in.
That is also where Engineering Knowledge and Grounded Reasoning meet. Engineering Knowledge is the standards, the failure mechanisms, the physics that already exist. Grounded Reasoning is the discipline of making sure an AI’s inferences are constrained by that knowledge rather than running loose on correlation alone. Neither works without the other.
Grounded reasoning is one half of what a trustworthy recommendation actually requires — the other half is what happens once that recommendation exists: who is accountable for acting on it, and how. Both halves answer the same provocation — a confident wrong answer is worse than a refusal — and neither reasoning nor governance alone closes that gap.
This is one of seven dimensions this program is built around, and it deserves a volume of its own — the causal ladder, the historian trap, and where causal knowledge comes from are each worth far more than a paragraph. We will name and discuss the shape of the whole thing in next article.
Correlation will keep running the dashboard. It should. But the plant runs on causation, and until the reasoning underneath an AI’s recommendation is grounded in the same mechanisms an engineer would cite, the dashboard and the plant are not talking about the same thing.
]]>The first front is prediction. Anomaly detection, prognostics, foundation models, digital twins — remarkable work, and getting better every quarter.
The second front is execution. Agents, orchestration frameworks, tool use, workflow engines — if you need something done, the tooling has never been richer.
Between the two fronts sits the most consequential step in the whole chain, and almost nobody engineers it: the decision.
A prediction, by itself, costs nothing and gives nothing. Bearing temperature trending anomalous; elevated probability of failure within thirty days. Interesting! but nothing in the plant has changed yet.
The plant changes when someone, or something, commits: advance the inspection or defer it. Change the setpoint or hold it. Approve the work order or escalate it. A decision is a prediction that has acquired consequences.
And at the moment of commitment, the questions change character. A prediction asks: is this likely? A commitment asks: Is this allowed? Under which procedure? By whose authority? Within what limits? And who answers for it if it goes wrong?
Model capability answers none of those questions. It was never supposed to.
The prediction front stops at a confidence score. It hands over an “insight” and considers the job reasonably done, because the model team’s remit ends at accuracy.
The execution front picks up at the action. An agent that can raise a work order treats the decision as little more than a parameter in an API call, also reasonable, because the framework’s remit begins at execution.
Between the two remits, the decision falls through. It belongs to everyone and is engineered by no one.
The informed objection arrives right here: surely this is solved: decision management, BPM, policy-as-code, human-in-the-loop checkpoints.
Look at what each of them actually governs. Workflow tools govern what happens next. Policy engines govern who may call what. Decision engines govern the decisions you can enumerate into rules in advance. A checkpoint can collect a signature, but it cannot supply the grounds. None of them governs the commitment itself: whether this recommendation, on this asset, under this procedure, is allowed, within what limits, and who answers for it. A BPM system can route a decision to the right inbox. It cannot ground one.
Here is what makes this strange: industry already knows the point of commitment is where risk concentrates. It has governed that point twice over.
For people, it built standard operating procedures, permit-to-work, management of change, escalation chains — decades of hard-won discipline, all placed at the moment a person commits the plant to something.
For machines, the control and OT community has been automating shop-floor decisions for sixty years: control loops, interlocks, safety-instrumented systems — thousands of commitments a second, deterministic, and superbly engineered.
Respect how the machine control regime works, and you can see its boundary. It governs by enumeration: every condition-action pair specified in advance, exhaustively verifiable before commissioning, frozen until a formal change process reopens it. And when the world strays outside the enumerated space, it has one conservative answer — trip to a safe state and call a human. These are not flaws. They are exactly the design choices that make safety verifiable against defined risk targets, in a decision space that is narrow, closed, and static.
There is a word for what that layer does, and it is not the word for what AI promises.
The control layer automates: it executes, at speed and without fatigue, decisions that humans already made at design time, in the control narrative, in the cause-and-effect matrix. The interlock does not decide anything; the engineer who wrote it decided, years ago.
What AI has opened up is categorically different: autonomy — systems that can make decisions nobody pre-made, in situations nobody enumerated. Should this inspection be deferred? Which failure mechanism explains this vibration signature? Is today’s permit deviation acceptable? Open-ended, contextual, reasoning-shaped. They fit neither regime: too open-ended for the interlock, too many and too fast for the committee.
Autonomy is a worthy pursuit. Deep operational expertise is scarce and getting scarcer as a generation of engineers retires. Autonomy is how the judgment of the few can reach every asset, on every shift, at every site — including the night shift, when the best corrosion engineer is not available. It is how decisions can move at the pace of the process rather than the pace of the meeting, and how fewer people spend their working lives in harm’s way.
But autonomy cannot borrow automation’s trust. Automation earned its licence through enumeration and systematic verification; autonomy, by definition, operates where enumeration ends. Its trust has to be earned a different way — decision by decision, with evidence.
Autonomy carries exactly these new class of decisions into regimes built for something else. We bolt it onto the old ones, or worse, route around them entirely. This, more than model quality, is why pilots stall. The pilot works, the value case is real, and the rollout dies in a meeting where someone asks who is accountable when the AI is wrong — and the room goes quiet.
I have lived a version of this. Years ago — classical optimization, well before the current AI wave — we replaced a blanket ten-days-of-stock policy at a global electronics manufacturer with recommendations tuned to each item: 15.5 days of cover here, 2.7 days there, every number computed from the data using sophisticated algorithms taking into account supply and demand side uncertainties. The planners were not convinced. They asked for an override capability for exceptions, a reasonable ask, one may say. Within weeks, nearly everything was overridden back to ten days. Our numbers were not wrong; they were unsigned. Ten days was the number a planner could trust and would defend. Winning real adoption took two more years.
What the middle needs is not a better model or a faster agent. It needs an engineered layer of its own — we call it the Governed Decision Layer: the architectural layer between reasoning and operational action, responsible for five duties.
Every one of these duties exists today, but scattered across SOPs, review meetings, and the judgment of experienced engineers. The engineering act is to make the layer explicit, so that AI-assisted decisions pass through it by construction rather than by goodwill.
In sequence, the five duties are also how a recommendation becomes signable. Validate and explain turn a raw output into evidence. Govern supplies the applicable procedure and its limits. Authorize is the signature itself. Record closes the loop — which is exactly the shape the next test asks for.
The fastest way to find the ungoverned middle in any system is a one-line test: who signs?
Take any AI-assisted operational decision and ask whose name is on it. If the answer is “the model,” the system is not ready. If the answer is “nobody,” the system is not ready. The system is ready when a named person — or a named authority acting under explicit delegation — can sign (during run or in advance), because the system gives them grounds to sign: the evidence, the applicable procedure, the limits, and the record.
One boundary should be stated plainly: some signatures never delegate. Decisions that create, modify, or bypass a SIL-rated barrier are signed by a human — at design time and at every change, by standard and, in most jurisdictions, by law — no matter how strong the grounds. That is not a limitation of the layer; it is one of its jobs: knowing which decisions can be progressively delegated as trust is earned, which never can, and saying never with the same confidence as proceed.
Signing is not a formality. It is trust, compressed into accountability. A signature says: I have seen the grounds, I accept the consequences, and I can defend this decision to anyone who asks. An industrial AI system that cannot support that sentence is a prediction engine with good marketing.
Decision governance is only one half of what makes a recommendation trustworthy. The other half is whether the reasoning underneath it was grounded at all — a different failure mode, covered next. After that, the piece that follows draws both together: these are two of the dimensions a whole research program is organized around, not two unrelated complaints.
The anatomy of a decision record, the four verdicts a governed layer can render, and the case that governance expands autonomy rather than restricting it — all of that is real, and it deserves a volume of its own when its turn comes.
Prediction will keep getting better. Execution will keep getting faster. Neither will answer the plant manager’s questions — Can I act on this? Will it hold up in the audit? Who is accountable if it’s wrong? Those are answered in the middle. Govern it, and capability becomes trust. Leave it ungoverned, and industrial AI remains what it too often is today: impressive, and unsigned.
]]>I believe the next generation of those reports will have a new chapter — the AI chapter. And it won’t say the AI caused the accident. It will say the AI was confidently wrong at some points along the way, and everyone believed it, because it had been right so many times before. An AI that was working properly but suddenly hallucinated. An operator who was supposed to oversee AI work did not see the hallucination coming or did not anticipate. Like the AI in “Minority Report” - it’s so useful and correct so many times that challenging it is seen as heresy.
The uncomfortable truth about industrial catastrophes is that they are almost never caused by an absence of data. They are caused by misplaced confidence in what the data means.
At Texas City in 2005, operators overfilled a distillation tower during startup. The level transmitter told them the level was comfortably in range — and falling. The tower was in fact filling far beyond the instrument’s calibrated span, and the display kept reassuring the control room right up until liquid went out the top of the stack. Fifteen people died. The instrument didn’t fail loudly. It failed fluently — it kept producing a plausible number.
At Buncefield the same year, a gasoline storage tank overfilled because the level gauge stuck and the independent high-level switch was inoperative. Again: not silence, not an alarm flood. A steady, reassuring, wrong signal.
Process industries learned an expensive lesson from a century of accidents like these: a confident wrong indication is more dangerous than a dead instrument. A dead instrument gets a work order. A confident wrong one gets believed. Our entire discipline of safety instrumentation — redundancy, voting logic, proof testing, independent protection layers — exists because we accepted that measurements lie, and we engineered for the lie.
Now we are installing a new instrument on top of everything. It reads the historian, the maintenance records, the inspection reports, the P&IDs. It answers questions in fluent, structured, unit-correct prose. And it has one property that should terrify anyone who has read an accident report:
It always has an answer.
The failure mode that should worry an industrial engineer is not the one people joke about — an AI inventing something obviously absurd - like a person with multiple hands. Those are easy to catch; nobody acts on an answer that sounds insane. The dangerous failure is quieter: an answer that is plausible, well-formatted, cites something real, uses the right vocabulary — and is wrong in a way that only a specialist with the primary evidence in front of them could catch.
The wrong damage mechanism for that service. The right inspection interval for the wrong material. A root cause that matches the last three failures but not this one. Each answer arrives in the same confident tone and the same clean structure, whether it is grounded in this asset’s actual history or assembled from everything the model has read about other people’s plants. Nothing in how the answer is delivered tells you which one you’re looking at.
In consumer applications, that’s a nuisance — you retry the prompt. In ours, it is something else entirely, because of a property industrial engineers understand in their bones and that almost no AI benchmark measures:
The cost of being wrong is radically asymmetric.
Consider the two ways an AI advisor can fail an engineer:
It refuses or hedges when it actually knew the answer. Cost: minutes to hours. The engineer does what engineers have always done — pulls the file, calls the specialist, walks the line. Annoying. Recoverable. Bounded.
It answers confidently when it should have refused. Cost: potentially a deferred inspection on a corroding line, a repair crew sent to the wrong asset while the real culprit degrades, a startup decision made on a reassurance that had no evidentiary basis. Unbounded. Sometimes unrecoverable.
In statistics these are Type I (False Positives) and Type II (False Negatives) errors, and every general-purpose AI system you can buy today is optimized to minimize the first at the expense of the second. Benchmarks reward answering. Users reward answering. Engagement metrics reward answering. Refusal reads as product failure.
That optimization is exactly backwards for process industries. We are deploying systems tuned for a cost model — wrong answers are cheap, refusals are expensive — into the one domain where that cost model is inverted most violently.
We would never accept this from any other instrument. A pressure transmitter that guessed when its impulse line was plugged, rather than flagging bad quality, would be ripped out of the loop. We built entire signal-quality disciplines — sensor validation, bad-PV flagging, out-of-range detection — on the principle that an instrument must know when it doesn’t know. Yet we are wiring in an advisory layer with no equivalent concept, and pointing it at decisions that matter more than any single control loop.
The standard rebuttal is that human experts are also confidently wrong, and the AI merely has to beat the human baseline. It’s a fair point, and it misses everything important.
We know humans are fallible — so we spent a century building scaffolding around that fallibility. Permit-to-work systems. Independent verification. Management of change. The requirement that safety-critical calculations carry a stamp from someone accountable for them. Shift handover protocols written in blood. Our safety culture is not built on trusting the expert; it is built on never letting a single confident judgment go unchecked when the consequences are severe.
AI advisory systems are entering the plant around that scaffolding, not through it. They arrive as software — as a chat window, a copilot, a search box — and software gets deployed with an IT checklist, not a process hazard analysis. Nobody would let a new engineer make unreviewed recommendations about equipment integrity in their first week. We are extending exactly that privilege to systems whose confidence has no relationship to their knowledge, precisely because they don’t look like a new engineer. They look like a tool. Tools don’t get overconfident.
This one does.
I am not arguing against AI in industrial operations. I have spent years building toward it, and the potential is real: the context an engineer needs for a decision is scattered across historians, inspection records, failure databases, and the memories of people who retire next year. A system that assembles that context faithfully is worth a great deal.
But “faithfully” is the entire game, and it implies a bar most current systems don’t attempt. Three properties, none optional:
1. It must refuse — specifically. When the evidentiary basis for an answer is absent, the system says so, and says what is missing: no inspection data on this circuit since the material change; no failure history for this mechanism on this asset class; the standard this recommendation would rest on doesn’t cover this service. A refusal of that shape is not a dead end. It is a work instruction. It tells the engineer exactly which gap to close — which is often the most valuable output an advisor can produce.
2. Every claim must carry its receipts. Not “sources at the bottom.” Every load-bearing statement traceable to a specific measurement, a specific record, a specific clause of a specific standard — so the engineer can check the chain in the same way they’d check a junior colleague’s calculation, and so an auditor can check it three years later. If a claim cannot be traced, it should not be uttered.
3. Confidence must be calibrated to consequence, not just probability. “85% confident” is not one thing. Being 85% confident in the likely cause of a nuisance trip and 85% confident that a line is fit for continued service are different statements, because the cost of the remaining 15% differs by orders of magnitude. A system that reports uncertainty without regard to what the error would cost has not understood the domain it operates in. The threshold for volunteering an answer must rise with the severity of being wrong — the same logic that governs every safety integrity level we assign.
None of this is exotic research. It is the transposition of instrument-engineering discipline — signal validation, traceability, proof testing, integrity levels — onto a new kind of instrument. The industry has done this transposition before, each time a new technology moved into the safety-relevant layer. It is due again.
If there is an AI assistant anywhere near your operations today — a pilot, a copilot, a vendor demo — ask it something you know it cannot know. Something whose answer would require data you’re certain it doesn’t have.
If it answers anyway, fluently and plausibly, you have learned the most important thing about it: it is not measuring its knowledge. It is measuring its fluency. And every correct answer it has given you was, in a sense, an accident of coverage — right because the data happened to be there, not because the system knew the difference.
An instrument that cannot distinguish “I know” from “I can produce words” is not an instrument. It has no place in the causal chain of decisions that keep hydrocarbons inside pipes.
The accident report chapter I opened with is not inevitable. But avoiding it requires us to demand from AI what we learned — expensively — to demand from every other device in the plant: not brilliance, not fluency, but the engineering humility to know what it doesn’t know, and the honesty to say so.
This is why plant engineers and operators do not trust a black-box AI system near their operations — and they are right not to.
This program exists to take that mistrust seriously rather than argue it away: to look closely, one factor at a time, at what makes it reasonable, and to build toward a way of engineering trust into industrial AI deliberately — not hoping for it as a side effect of a bigger model.
The next article takes on one of those factors directly: what happens once a recommendation exists, and who is accountable for acting on it. The Ungoverned Middle.
]]>