LLMs in Trading: What They Can and Can't Do

Every few weeks a paper or a product claims a large language model beat the market. The claims deserve a harder look than they usually get, because llm trading sits at an awkward intersection: language models are genuinely excellent at some trading-adjacent work and genuinely terrible at other parts, and the marketing rarely tells you which is which. This article draws the line precisely. What LLMs do well, where they fail, why their backtests flatter themselves, and the architecture that gets the benefits without betting your account on a text predictor.

What an LLM is, and why that matters at the order book

An LLM is a next-token predictor: given text, it produces the text statistically likely to follow, shaped by training to be helpful. That single fact explains both its strengths and its failure modes in trading.

Prediction over language makes LLMs superb at language tasks: summarizing, translating between phrasings, extracting structure from prose, imitating reasoning patterns that appear in their training data. Finance is drowning in exactly this kind of material. Filings, transcripts, central-bank statements, news wires: unstructured text that previously required human hours to digest.

But next-token prediction has no ledger inside it. The model does not compute 2.7% of $41,000 the way a calculator does; it produces digits that look plausible in context. It has no calibrated sense of probability, no live market feed unless one is bolted on, and no concept of being wrong, only of being unlikely. None of that matters when the output is a summary. All of it matters when the output is an order.

Where LLM trading tools genuinely help

Four jobs stand out, and they share a property: the LLM handles language while something else handles consequences.

Parsing news and filings at machine speed. An LLM can read an FOMC statement seconds after release and tag it against the prior one: which sentences changed, whether the language on inflation hardened or softened. It can turn a 200-page annual report into "guidance cut, margin pressure attributed to input costs, buyback paused." Humans do this well; LLMs do it in seconds across hundreds of documents.

Translating intent into rules. "I want to buy weakness in an uptrend" is not executable. An LLM is good at proposing the translation: price above the 200-day moving average, RSI(14) below 35, position size fixed, exit defined. The human judges whether the translation matches the intent; the machine handles the drafting.

Explaining positions and regimes. Asked why a portfolio lost 4% this week, an LLM with access to the actual position data and price history can produce a readable attribution: which holdings drove it, what correlated, what news coincided. This is summarization wearing a finance hat, squarely inside the model's competence.

Critiquing strategy logic. Describe your system to an LLM and ask what breaks it, and you often get a useful list: no invalidation rule, no fee assumption, an entry condition that rarely fires. A general chatbot can do this much, which is why we treat ChatGPT as a useful assistant and a terrible trader, and the distinction carries over to purpose-built systems.

Where LLMs fail, and fail confidently

Arithmetic. Ask a model to size a position risking 1% of a $27,400 account with entry at $43,150 and stop at $41,900, and you may get the right answer: $274 of risk against $1,250 per unit, so roughly 0.22 units. You may also get a fluent paragraph containing a wrong number, with no signal distinguishing the two cases. A calculator's errors are rare and loud; an LLM's are occasional and silent. Silent errors are the expensive kind in trading.

Calibration. Trading is probabilistic decision-making, and LLMs do not produce calibrated probabilities. A model that says "this breakout has a 70% chance of continuing" is generating a plausible sentence, not measuring a frequency. Acting on invented confidence is worse than acting on no confidence, because it feels like information.

Price prediction. Nothing in next-token training confers an edge in forecasting returns. An audit of 77 LLM-trading studies (arXiv:2605.19337, 2026) concluded that evidence for durable alpha remains thin, and that the practical value of these systems lies in disciplined execution, monitoring, and research assistance rather than stock-picking. That finding matches the mechanism: the model recognizes patterns in text, and prices are not text.

Fabricated facts. LLMs hallucinate, and finance provokes it: dense numbers, similar entity names, date-sensitive claims. Benchmarks from 2024 to 2025 measured hallucination on a substantial share of finance questions asked without safeguards. The failure mode and its architecture-level fixes deserve their own treatment, which they get in AI hallucinations in trading.

The backtest mirage: why LLM results flatter themselves

The research literature has a replication problem worth understanding before you trust any LLM strategy's track record.

A 2024 survey of LLM trading agents (arXiv:2408.06361) noted that most published systems report superior backtest performance, then flagged the pattern behind the pattern: short test windows, small stock universes, and lookahead risk across much of the field. Impressive numbers, weak experimental design.

Then came the sharper diagnosis. "Can LLM-based Financial Investing Strategies Outperform the Market?" (arXiv:2505.07078, 2025) showed that LLM backtests are inflated by memorization: a model whose training data covers the backtest period can effectively remember tickers, dates, and price moves. It is not forecasting 2021; it is recalling 2021. Performance often degrades once the strategy is evaluated out-of-sample, past the model's knowledge cutoff.

A 2026 memory-controlled benchmark (arXiv:2605.28359) confirmed it: when evaluation is designed so memorized market history cannot leak into predictions, apparent skill drops sharply.

The practical test is simple. Any LLM strategy backtested over a period the model was trained on is suspect by default. Ask for results after the training cutoff, or for a memorization-controlled evaluation. This is the agent-era cousin of a classic quant sin, and the classic defenses still apply; we cover them in backtest overfitting.

The architecture that works: language on top, determinism underneath

The honest conclusion from all of the above is not "avoid LLMs." It is a division of labor. Let the language model do language. Let deterministic engines do everything with consequences.

In a well-built system the LLM interprets your instruction, drafts the rule, explains the results, and summarizes the news. The rule itself, once approved, runs as plain deterministic logic: indicator values computed from market data, thresholds compared exactly, orders sized by arithmetic that cannot improvise. The LLM proposes; the engine disposes. If the model hallucinates during interpretation, you catch it at the approval step, because the system shows you what it understood before anything runs. A wrong sentence gets corrected; it never becomes a wrong order.

This is how Obside draws the line. The copilot is an LLM, and its job ends at language: you type "buy SOL when RSI(14) on the daily closes below 30, sell when it crosses back above 55, risk 1% per trade," and the copilot restates that as monitored conditions and a sizing rule for you to approve. From there, deterministic engines take over: the backtest replays historical candles with exact indicator math, paper trading runs the same logic on live data, and live execution computes every order size numerically with your risk limits checked at execution time. The LLM never does the arithmetic and never touches an order.

That split also changes what a backtest means. Because the tested object is the deterministic rule rather than the model's day-by-day judgment, there is no memorized history to leak in: the rule performs identically whether the test window falls inside or outside any training corpus. You are validating logic, not recall. How these pieces compose into a full perceive-reason-act system is its own topic, covered in AI trading agents explained.

Where to go from here

Treat the LLM as a brilliant analyst with no license to touch money. Use it to read faster, translate intent into precise rules, and interrogate your own logic. Refuse it three jobs: computing numbers, estimating probabilities, and predicting prices. And discount any performance claim from a backtest the model could have memorized; the research is unambiguous on that.

If you want this division of labor already built, Obside applies it by default: language handled by the copilot, math and execution handled by deterministic engines, and nothing goes live before you have approved the restated rule and watched it work on paper.

Educational content only. This is not investment advice. Trading involves risk, including possible loss of capital.