Using ChatGPT for Competitive Analysis: Where It Fails

01The split: shaping versus sourcing

Every useful judgment about assistants in competitive research reduces to one distinction: are you asking it to shape information you brought, or to produce information it has to recall?

This maps onto something worth stating plainly: the expensive part of competitive intelligence was never the writing. It was gathering trustworthy evidence and deciding what it means. Assistants collapse the writing cost to near zero and do not touch the gathering cost, which is why teams that adopt them often produce more competitive documents and no more competitive insight.

02How it fails, specifically

The failure mode that matters is not that the model says it does not know. It is that it produces something plausible, structured, and wrong.

A practitioner who went through 60 or more discussions on how teams actually do competitive intelligence, then tested several of the AI tools in the space, reported two failures worth remembering:

Neither of those looks wrong in a skim. Both would survive into a document if nobody checked. The named failure modes worth watching for:

There is a sixth failure that is harder to name because it looks like success: the model produces an analysis that is entirely correct and entirely useless. Ask it to compare two products and you get a balanced, accurate feature table that supports no decision, because balance is what the training rewards. Competitive analysis has to reach a conclusion, and reaching one requires knowing which attribute your buyer actually decides on. That is your knowledge, not the model’s, and it has to be supplied.

The last one is the most insidious because it is not technically a hallucination. The model read the competitor’s website and faithfully repeated what it said. That is not competitive intelligence, it is a competitor’s marketing with your logo on top, and it is what you get by default when the only evidence supplied is a competitor’s own pages.

03Why assistants disagree, and how to use that

Ask several assistants the same competitive question and you will get materially different answers. This is a feature for research purposes.

A practitioner ran the same category question through three major assistants and recorded the results:

The mechanism is retrieval: the engines draw on different sources and weight them differently, which is covered in more depth in which websites AI engines actually cite. The practical consequence for competitive research is direct.

• Use disagreement to widen the competitor set. Ask three assistants who the leading options are in your category and take the union, not the intersection. New entrants surface this way before they appear in any listicle.
• Read the reasoning, not just the names. When an assistant explains why it picked something, it is telling you which attribute the underlying sources emphasise for your category.
• Never treat one assistant’s list as the market. If your buyers use a different one, you have researched the wrong competitive set.

04The workflow that holds up

Gather evidence yourself, hand it over, require citation, verify anything expensive. Four steps, and the order is the whole method.

1. Gather. Pull the competitor’s pricing page, their changelog, and 30 to 50 recent reviews. This is the part no assistant can do reliably, and it is the part that determines output quality.
2. Supply, do not ask. Paste the material in. Every question you ask about supplied text is a shaping question, which is the reliable half of the split above.
3. Require inline citation to your material. Ask for each claim to reference which document it came from. Fabrication becomes visible immediately, because invented claims have nothing to point at.
4. Verify anything that will be said out loud. If a claim will appear on a comparison page or in a rep’s mouth, open the source and date it.

The fourth step is the one that gets skipped under deadline, and it is the only one that protects you. A useful forcing function: nothing produced by an assistant goes into a customer-facing document until somebody has opened the source and written a date next to the claim. That single rule catches almost every failure mode listed above, costs a few minutes per card, and is far cheaper than the alternative, which is a rep repeating an invented number to a prospect who knows better.

This is slower than pasting a mega-prompt and getting a finished report, and it is the difference between competitive analysis and competitive fiction. The cleaning discipline for review evidence specifically is in competitor review analysis, which matters here because feeding a contaminated review set to an assistant produces a confident summary of fake reviews.

05What you should not paste in

The workflow above says supply the evidence. That instruction needs a boundary, because the fastest way to make competitive analysis a legal problem is to paste the wrong thing into a chat window.

Competitive research sits closer to sensitive material than most marketing work. Lost-deal notes contain named prospects. Call recordings contain customers who did not consent to processing. Documents that arrived from a competitor by an unclear route are a separate problem entirely.

The anonymisation point on public reviews is worth doing even though the material is public. A review is written by an identifiable person who was talking about a product, not volunteering to be an input to your competitive programme. Stripping usernames costs nothing, keeps the analysis identical, and is the same standard we hold ourselves to when quoting practitioners on this site.

One further consideration that is practical rather than legal: what you paste describes your strategy. A prompt asking how to displace a named competitor, supplied with your own positioning and pricing, is a fairly complete description of your go-to-market. Whether that matters depends entirely on which tier and which data policy you are operating under, and it is worth knowing the answer rather than assuming one.

06Prompts worth keeping

Prompt lists in this category are mostly interchangeable. These four are worth keeping because each one exploits the shaping strength rather than the sourcing weakness.

Note what none of them ask: who is the best tool, what does this competitor charge, or does this competitor have this feature. Those are sourcing questions and they belong to a browser tab, not a prompt.

A useful test before running any competitive prompt: could a colleague check this answer in under five minutes? If yes, the assistant is doing safe work and you have a fast verification path. If no, you are asking it to be an authority on something unverifiable, which is exactly where the confident inventions live. The prompts that survive contact with a real sales team are the ones whose output can be argued with, because someone eventually will.

The third prompt is the one most worth adopting. It is also the one people avoid, because a genuinely honest answer to it frequently says that you do not address the competitor’s biggest weakness either.

Two habits improve every prompt in this category more than the wording does. Ask for the answer to include what it could not determine from the supplied material, which turns silence into an explicit gap list rather than a confident guess. And ask it to separate what the evidence says from what it is inferring, which is a distinction models will hold when instructed and collapse when not.

Resist the mega-prompt. The long templates circulating in this space try to produce an entire competitive report in one pass, and they fail in a predictable way: the model runs out of supplied evidence partway through the requested structure and fills the remaining sections from general knowledge. The output looks complete and the back half is invented. Several short, evidence-bounded prompts beat one comprehensive one, every time.

07Where assistants cannot help at all

A chat session cannot monitor anything. It has no memory of last month, no ability to run unattended, and no way to tell you that something changed while you were not looking.

This is the boundary worth being precise about, because it is where the category oversells. An assistant is excellent at reading a change and explaining what it means. It is structurally incapable of noticing the change in the first place.

There is a related limit worth naming because teams hit it after a few weeks: a chat session cannot tell you what it did not find. Ask an assistant about three competitors and it will answer about three competitors. It has no way to flag that a fourth one entered the market last quarter and is now appearing in your lost deals. Detection of the unknown requires something scanning continuously, and the assistant only ever sees the question you thought to ask.

The pattern that works is machine detection plus assistant interpretation. Something durable watches the surfaces and keeps dated history; the assistant reads the diff and tells you what it implies. That division of labour is also our own product principle: we do not build what an assistant already does well, we make sure it has the data and the memory it lacks.

08Frequently asked questions