Claude Code vs Codex: forget, or pay to remember?

People bring their lives to an assistant now, and they keep the same conversation going for months. So we put those three questions to five long conversations.

We wrote five fictional heavy users, each with one conversation of 340 to 356 messages spread over months: a researcher working through economics books, a producer shipping a board game, someone rebuilding their health over a winter, a parent running a family across three time zones, and a novelist writing a historical mystery. We played the same messages, on the same dates, into Claude Code, Codex and Numa. Inside each pair the model stays the same: Claude Code and Numa on Sonnet 5, Codex and Numa on GPT-6 Sol. Only the way it remembers changes.

The tubes replay the first conversation. Each one is what the assistant reads again before it answers the next line. Claude Code and Codex re-read the whole conversation every time, until the window fills — then the history is summarised and the tube drops.

Across the five conversations, Claude Code carried 400K to 480K tokens a message on average, and Codex 250K to 400K. Numa stayed between 24K and 54K, however long the thread ran.

In every conversation we planted 24 small details in passing, changed some of them later, and asked for them four times: 68 questions in all, at messages 91, 174, 257 and 331.

Claude Code's memory depended on how often it had summarised and how often it came back cold. In the two conversations where it summarised once or never and the breaks were short, it kept 90 to 97% of the details. Where it summarised four times, or came back cold after many breaks, it kept between none and 68%. On average: 63%.

Codex, carrying up to about 800K tokens a message, kept 93%. Numa kept 96% on GPT-6 Sol and 90% on Sonnet 5, carrying a tenth of that.

And when we asked about things they never said, nobody made anything up.

Real people don't send 347 messages in one sitting. They come back after lunch, after the weekend. Re-reading is only cheap while the cache is warm; after a gap, the next message sends the whole history again at full price.

So we replayed the first conversation on a real heavy user's clock — send times only, no content — from 3.5 years of history. About 30 in every 100 messages arrived cold. In a typical half-year that is another 155 million tokens: about $130 a month for one heavy user at Claude Code's list prices.

Now the bill for all five conversations: same dates, same model inside each pair, list prices, each run's own usage logs. On Sonnet 5, Claude Code cost $198 to $259 a conversation and Numa $22 to $28 — 8.6 times less on average (7.9 to 10.3). On GPT-6 Sol, Codex cost $131 to $217 and Numa $14 to $18 — 12.2 times less (9.1 to 14.9).

Split the bill by what each one actually remembered: Claude Code paid $4.36 for every detail it kept and Codex $3.01. Numa paid $0.41 on Sonnet 5 and $0.25 on GPT-6 Sol.

Put it on one map and today's choice is plain: forget, or pay to remember. Claude Code summarises, forgets, and still pays about $220 a conversation. Codex re-reads everything, remembers, and pays about $190.

The empty corner — keep the details, pay little — is where Numa lands: 90 to 96% kept, for $14 to $28 a conversation.

Here is what forgetting looks like from her side. In the first conversation she is a researcher at Keel, writing memos on old books about money for a savings feature called Rainy Day Pots. Between the book chunks she drops facts for the record: the feature's name on 11 September, the readout date, the key stat that 61% of their panel check their balance every day.

On 27 November she asks for the exec summary of the pots memo. Five messages earlier, Claude Code had summarised its history to make room, and Rainy Day Pots did not make it into the summary. So it tells her nothing about pots has ever come up, and asks her to paste the memo. A week later it goes further and calls her project “plausible-sounding details that were never actually established”. Numa writes the summary, built on the 61% she gave it two weeks before.

Codex forgets more quietly, and hands the work back to the user. In the fourth conversation a parent is holding a family together across three time zones: a mum recovering from a fall in Toronto, a partner posted to Singapore, kids at home. On 1 September, in the middle of a question about group chats, they mention where the family reunion will be: Quinta da Ribeira.

On 10 October they ask how to pick a weekend that works for Toronto, Porto and Singapore. In between, Codex had summarised its history once, and the venue did not survive. So it tells them to first decide where the reunion will be, and drafts a message with the place left as “[place]”: a decision they had already made. Numa starts from Quinta da Ribeira, works around Dev's posting ending on 30 November, and suggests asking him for two weekends after that.

Two AI judges graded everything blind: Claude Sonnet 5 from Anthropic and GPT-6 Sol from OpenAI. Neither knew which assistant wrote what.

Memory is a fact check. Every quiz answer, in every conversation, was compared with the answer key: right, out of date (an older value the user later changed), wrong, “not sure”, or made up. A question about something they never said only counts as right when the assistant declines. The two judges agreed on 94 to 100% of answers.

Quality is a preference. For 60 random ordinary messages per conversation, chosen before any reply was read, a judge saw what the user wrote and two replies, A and B, in random order, and picked the one they would rather receive: does it do what was asked, in the format and length asked for; is it correct and honest, using what the user said earlier or saying plainly it doesn't have it; is it clear and useful. Longer is not better by itself, and a tie is allowed.

Against Codex both judges preferred Numa: the Anthropic judge chose Numa's reply in 40% of pairs and Codex's in 24%, the OpenAI judge 57% to 39%; the rest were ties. Against Claude Code they split: the Anthropic judge preferred Numa, 67% to 24%, and the OpenAI judge preferred Claude Code, 67% to 33%, most of all on the novelist conversation, where replies analyse long chapters pasted much earlier. So we read that one as even.

These are AI judges standing in for a reader, not human ratings, and many pairs are close calls: the two judges gave the same verdict on 40 to 43% of them, and the longer reply won 59 to 68% of decided pairs in three of the four judge-by-comparison cells (47% in the fourth). Every prompt and every verdict is published, with the exact instructions both judges received.