It looked like a smart call for about three weeks.
I remember the room. A mid-market company, a pricing question that had been sitting on the CEO’s desk for a month, and a conference table with too many laptops on it. I had helped them stand up an analysis layer over their sales and market data, and for the first time they could ask a real question and get an answer back in minutes instead of waiting on a spreadsheet that lived in one analyst’s head. That part felt like progress. It was progress.
Then the analysis surfaced a competitor signal. A rival appeared to be moving on price in one of their core segments. The numbers were right there. Clean, recent, internally consistent. I looked at it, I reasoned through it, and I recommended a response. I was confident. I was well reasoned. I was wrong.
The signal was an anomaly, not a pattern. A blip that the model had faithfully reported and I had faithfully believed. By the time we understood that, the client had already adjusted and given up close to a quarter of margin in that segment. Margin they never needed to sacrifice. Nobody yelled. There was no alarm, no red dashboard, no bad day. That is the part that stayed with me. The mistake arrived wearing the same calm face as every good decision I had ever made in that chair.
I paid the tuition on that one. And it taught me what actually breaks when you put AI anywhere near a real decision.
The failure people expect is not the failure that comes.
Here is the fair version of the opposing view, because it is where I was standing that day. When the machine hands you an answer that is coherent, sourced, and fast, disbelief feels like the unprofessional move. You have leaders in the room who waited a month for this. The tool has been right about a dozen smaller things already. Trusting it looks like maturity. Second-guessing it looks like you do not understand your own system. The signal that fooled me was not sloppy. It was the most persuasive kind of wrong, the kind that agrees with the shape of a decision you were already leaning toward.
So people brace for the dramatic failure. The model goes rogue. It hallucinates something absurd. It says the obviously insane thing and everyone catches it in the meeting. That failure is easy to picture and easy to stop, which is exactly why it is not the one that costs you.
Strategic AI fails quietly. And in my experience it fails in four familiar places.
The data underneath. Long before the pricing engagement, I watched a team spend the better part of eight months building an analysis engine they were genuinely proud of. Sharp people. Real effort. It sat on top of a product catalog with hundreds of duplicate entries and descriptions last touched years earlier. The model ran beautifully. It just took a mess and multiplied it, faster and with more confidence than any human ever would have. The engine was never the problem. The floor it was standing on was.
The leader who will not be questioned. The whole reason to put AI at the decision layer is to test assumptions. That is the job. But if the central assumption is the founder’s gut, and the founder treats that gut as sacred, the system has nothing to push against. I have sat with a CEO who asked the tool for analysis and then quietly discarded everything that disagreed with him. What he had bought, at real expense, was a mirror that told him he was right in a more sophisticated voice.
No loop. This is the one that got me on the pricing call, so I will not pretend I watched it from a distance. A decision goes out. The outcome comes back weeks later. And nobody carries that outcome back to the thing that made the call. So the system never learns it was wrong. Neither, if we are being honest, does the person who trusted it. The margin walked out the door and the model sat there ready to make the same recommendation again, just as sure of itself.
Ownership. When this capability lives with IT instead of with the person actually making the decision, it becomes a tool nobody at the table trusts, because nobody at the table is accountable for what it says. It gets treated as a report from another department. Interesting. Not binding. Easy to nod at and ignore.
Under all four there is a softer failure, and it is the one that cost my client the margin. AI does not tell you when it is unsure. A wrong answer is delivered with exactly the same steadiness as a right one. There is no tremor in the voice, no hedge, no tell.
I saw this land somewhere else entirely, which is what convinced me it was structural and not just my bad day. A different client was generating client-facing summaries, roughly forty of them, built quietly on the wrong quarter’s data. Those summaries went out over about six weeks before anyone noticed. Not because the work was careless, but because nothing looked broken. Every document was well formatted, fluent, plausible. The confidence was intact. Only the truth was off.
What I did not expect was where the fix actually paid off. When I finally forced the discipline, I assumed I was buying accuracy. Fewer wrong answers. What I got instead, and this surprised me, was that the leaders around the table started trusting the tool more, not less. Closing the loop meant the system occasionally came back and said, in effect, that a past call had not worked and here was the adjustment. The visible fallibility was the thing that earned it credibility. A tool that can be shown to be wrong and then corrected is a tool people will actually put weight on. The one that is never checked is the one that quietly gets ignored, right up until it costs you a quarter of margin.
So here is what changed in how I run engagements, and it is not a more sophisticated model.
I now build a feedback loop into the work before anything reaches a real decision. Clean the specific data the decision depends on, not the whole ocean, just the part under this call. Put the assumptions on the table where the tool is allowed to test them, including the founder’s. Close the gap between decision and outcome so the outcome comes home to the thing that made the call. And give the whole capability an owner who has to live with the results, someone at the table and not down the hall.
None of that is glamorous. All of it is the part that gets skipped, because it looks like overhead next to the exciting part.
Confident and wrong is the natural state of an AI with no feedback. It is also, if we are honest, the natural state of a lot of executives. The loop is what fixes both.
At Kennesys I spend most of my time building that layer, the unglamorous discipline that sits between a smart-looking answer and a decision you have to stand behind.