A memory store keeps the original turn and everything derived from it, so scoring retrieval means deciding which stored form is allowed to earn credit. TIAP holds the saved ranked output fixed and changes only that.
TIAP the scoring target problem
The problem
One turn becomes many stored memories. Only some of them are allowed
to earn credit.
Source turn
“I just finished my fifth issue of National Geographic.”
→
raw turn
the original message, stored verbatim
fact
reads National Geographic
summary
a digest of the reading session
event
a dated entry on the timeline
All four descend from the same evidence and sit in the same
index. When the query arrives, which one counts as the right answer? Most memory
benchmarks never say, and the answer changes the score.
The query
Saved ranked output
frozen
47 further results not shown
Credited target
Raw
miss
Only the original source turn earns credit.
Source
hit
Any stored descendant of the same evidence earns credit.
Canonical
hit
Only the transformed serving forms earn credit.
Same saved outputs, two systems
| system | Source | Canonical |
|---|---|---|
| all-MiniLM | 0.210 | 0.275 |
| BGE-M3 | 0.141 | 0.301 |
What actually got the credit
“For newspapers, I’ve been reading The New York Times
and The Daily News regularly”
Sonnet 4
Opus 4.7
Gemini 3.1
GPT-5.5
DeepSeek V4
Three stages, over output that never moves.
Stage I · construct scoring targets
Inputs
Query q
Gold source s(q)
Scoring targets
Raw · Traw(q, S)
only the exact raw source turn
Source · Tsrc(q, S)
any stored descendant of s(q)
Canonical · Tcan(q, S)
transformed serving memories
Traw ⊆ Tsrc ⊇ Tcan
Raw and Canonical are usually disjoint
→
Stage II · rescore fixed outputs
Saved outputs
Ranked trace R(q)
retrieval never rerun
Rescoring
TrawTsrc
Tcan
Shared-subset control
compare only queries that all three targets cover
Δt(A, B) = Mt(A) − Mt(B)
system gap under target t
→
Stage III · target sensitivity
What can change
score shifts Δt
hit and miss flips
winner flips
design flips
Semantic audit · n = 1,902
✓
fully supports s(q) · only 29.2%
~
partial support, needs context
×
contradicts or irrelevant
Target noninvariance
benchmark conclusions can change while ranked outputs stay fixed
In a deployed setting
Run the same audit over a whole benchmark and the gap is not an
outlier. It is the norm.
all-MiniLM on LongMemEval-S, every query
| target | queries | nDCG | hit rate |
|---|---|---|---|
| Raw | 470 | 0.157 | 0.598 |
| Source | 470 | 0.338 | 0.747 |
| Canonical | 332 | 0.531 | 0.907 |
BGE-M3, a different retriever, the same shape
| target | queries | nDCG | hit rate |
|---|---|---|---|
| Raw | 439 | 0.155 | 0.620 |
| Source | 439 | 0.333 | 0.761 |
| Canonical | 299 | 0.533 | 0.923 |
The same run, three verdicts
nDCG more than triples from Raw to Canonical on identical saved
output. A paper reporting one of these numbers is reporting a choice.
And the rows are not comparable
Canonical covers only 332 of the 470 queries, because not every turn
produces a serving memory. Comparing it against the other two unrestricted would
confound the target with query difficulty, which is why TIAP falls back to the
shared subset.
Across the bundle
83.4 to 94.0% of shared queries change nDCG under a target switch,
over 16 saved runs spanning two benchmarks, four retrievers, a parser density
sweep, and the Mem0 and MemoryOS transfers.
We release TIAP
Retrieval never reran. The ranking never moved.
The winner changed anyway.
The winner changed anyway.
aimsresearchlab.com/tiap
83.4–94.0%
of shared queries change nDCG when only the credited target changes
29.2%
of contested credit is fully justified, over a 1,902 case semantic audit
MTEL-Mem
the scoring engine and all 16 saved runs, so every number reproduces