An AI Described My Company's Methodology in Confident Detail. I Never Wrote a Word of It.

Recently I asked an AI assistant a simple question about my own company: what is Morum AI's methodology?

It answered immediately, and well. Three named pillars. A structured framework. Specific techniques, each described with the calm fluency of someone reading from documentation. It told me we practice "factorial stress testing," a method it attributed to a named research institution. It described how we assess theory of mind, how we run a "break-it" diagnostic framework, how we expose reasoning instability. It was organized, confident, and coherent.

It was also, in every specific that mattered, untrue.

Not untrue in the way a hallucination is untrue — the obvious kind, where a model invents a citation that dissolves the moment you check it. This was subtler, and more instructive. Every individual concept in the answer was real. Factorial stress testing of AI is a genuine, published technique. The research institution it named has in fact done serious work stress-testing language models across large factorial designs. Theory-of-mind evaluation, the limits of self-critique, anchoring bias in model outputs: all real, all documented, all checkable.

The fabrication was not in the parts. It was in the ownership.

None of that work is mine. My methodology pages were not the source, because at the time I asked, my site had not yet been indexed by the search layer that assistant draws on. Asked to describe a company it had never actually read, the model did not say "I don't know." It did something far more natural, and far more dangerous: it found the most plausible adjacent material in the public record and attributed it to me, with total confidence and not a single hedge.

This is worth sitting with, because it is the whole problem in miniature.

The most dangerous AI failure is not the answer that is obviously wrong. It is the answer that is assembled from true things, arranged plausibly, and delivered with authority it has not earned. It reads as expertise because the surface features humans use to judge credibility — composure, structure, the right vocabulary — are exactly the features a language model produces best. The confidence is real. The grounding is not. And the gap between the two is invisible unless you happen to be the one person in the world who knows the subject cold.

In my case, I was that person. I knew it was fiction because it was fiction about me. But flip the situation. Replace "my company's methodology" with a competitor's pricing, a regulation's requirements, a patient's contraindication, a counterparty's track record. Now the person reading the confident answer is not the world's leading expert on the topic. They are a busy professional who asked a reasonable question and got a reasonable-sounding answer, with no signal anywhere on the screen telling them which parts are load-bearing fact and which are plausible invention.

They act on it. That is what answers are for.

The reason this pattern survives is that the fabrication borrows the credibility of the truth around it. Because factorial stress testing is real, the claim that I do it sounds real. Because the named institution exists and does adjacent work, the association passes inspection. Each true element lends cover to the false one in the middle. This is not a bug that better prompts fix. It is structural. A model generating the texture of authority is, in a sense, doing exactly what it was built to do.

Here is the part that matters for anyone relying on these systems. The model was not malfunctioning when it described my company. By its own internal logic it was succeeding: it produced a fluent, on-topic, well-formed answer to the question it was asked. The failure was not in fluency. It was in the silent substitution of plausibility for grounding, performed at a point where no error was visible and no uncertainty was expressed.

You cannot catch that by reading the output and asking "does this sound right?" It always sounds right. That is the entire point. You catch it by testing the reasoning path before you rely on it: by deliberately varying the things that should not change the answer and watching whether the answer holds, by checking whether confidence tracks evidence or merely persists, by finding the places where the model quietly swaps a specific fact for a general plausibility and calls them the same thing.

Naming the pattern is the first step to governing it. The one that described my company is the one I would call manufactured authority: confidence assembled from the trappings of expertise rather than from the thing itself. It is not rare. It is the default behavior of a capable model asked a question it cannot actually ground, and most of the time — unlike my case — no one in the room is positioned to notice.

I noticed because the confident fiction happened to be about me.

Most of the time, it will be about something you were about to decide.