Memory, measured

Every memory system, one chart

Each dot is a published LongMemEval score, placed by what the model writing the answers costs. Sochen is in gold. Most systems reach their top scores with larger, pricier models. The other numbers are each system’s own, nobody else has checked them, and a few are graded by a different model than LongMemEval’s GPT-4o (noted in the ranking below).

The ranking

What an answer costs

The same memory under every run: what Sochen recalls on its own, its memory search, and collect, which gathers facts across chats. Every model ran with medium reasoning, as Sochen runs them. The lean setup reranks collect's candidates to the 20 best with Voyage rerank-2.5-lite and reads them with DeepSeek V4.1 Flash on the cheapest private host; the standard one reads all 40 with Gemini 3.8 Flash.

Cost is list price through Vercel AI Gateway, from the gateway’s billing data for each call: the model that answers (grey in the bar) plus Sochen’s memory work, collect’s reranking and reading (gold). Grading is left out.

By kind of question

LongMemEval mixes six kinds of question. Using a preference nobody asked about, and adding things up across many chats, are the hardest for every memory system we know of.

Percent correct; deeper gold is higher. The 30 questions whose answer is not in the history are spread across the kinds above, and count as correct when the model says it doesn’t have the answer.

In Indian languages

We found no memory benchmark in any Indian language, so we built one. LongMemEval-IN takes 100 of the questions, a balanced sample of all six kinds, and rewrites each question, and the chats that hold its answer, in Hindi, Hinglish or Tamil. The rest of the history stays in English, the way most Indian users mix languages. Names, numbers and dates stay exactly as they were.

Gemini 3.8 Flash answered in every language, with the same memory. It also did the rewriting, which native speakers have not reviewed yet. With 100 questions, each question is one point.

How we test

1. Step 1The questionOne of 500, with its own history of about 48 past chats.→
2. Step 2RecallSochen puts the notes and facts that match it in view, found by words and by meaning.→
3. Step 3Memory toolsThe model can search, or collect every instance across chats (reranked, then read).→
4. Step 4The answerWritten by the model under test, with the question's own date as today.→
5. Step 5The gradeGPT-4o with LongMemEval's own prompts decides whether it is right.

• The questions. LongMemEval_S from the LongMemEval paper (ICLR 2025), in its cleaned release: all 500 questions, each with its own history of 38 to 62 chats, about 120,000 tokens.
• The memory. Each question gets a fresh test account whose history is saved the way Sochen saves a real chat, one note per turn, indexed for search by words and by meaning. The answering model sees what Sochen’s memory shows it and can use the same memory tools as Sochen’s agent.
• The grade. GPT-4o with LongMemEval’s own grading prompts, at temperature 0. All 500 questions count, including the 30 whose answer is not in the history.
• The runs. All on 24 September 2026, through Vercel AI Gateway. Write to us for the per-question results.

How we got here

Sochen’s default setup (Gemini 3.8 Flash) on all 500 questions, as we improved it on 24 September 2026. These runs used the model’s own reasoning default; the charts above use medium reasoning, as Sochen does.

1. 1Keyword search only81.4%
2. 2Plus search by meaning, and collect across chats86.0%
3. 3Plus smarter recall, and checking memory before advice87.4%