Key finding
Agent loops running on RAG backends not only return wrong answers, but they also get stuck.
This is because RAG has no concept of time. Similarity search returns whichever version of a fact scores highest, old or not, so retrying might just return the same stale answer again.
Swapping in a temporal knowledge graph (Graphiti + Neo4j), with zero changes to the agents themselves, cut staleness error from 87% to 20% and took point-in-time retrieval accuracy from 60% to 100%.
Why loops get stuck
Imagine a company's CEO changes. Both names, the old CEO and the new one, are in your vector store. An agent queries for the current CEO and similarity retrieval returns the previous one. In a loop, a critic agent flags it and triggers a retry, but gets back the same stale fact. After two retries the loop escalates and returns nothing.
That failure happens because of the RAG architecture, and the inherent staleness it comes with. RAG works by embedding documents, storing them in a vector store, and retrieving by cosine similarity. This works for single-shot retrieval. Inside a loop, when two versions of the same fact coexist in memory, it tends to break. The retrieval process has no concept of time and returns whatever scores highest on similarity, which is often the outdated fact.
I ran an A/B test of two memory architectures against the same 4-agent loop.
- ChromaDB (flat vector RAG) as the baseline
- Graphiti + Neo4j (temporal knowledge graph) as the treatment.
Both architectures are backends to the exact same loop, with only the memory object differing.
The loop
The system is a 4-agent research loop built in LangGraph. The loop takes a query, decomposes it into subtasks, retrieves facts from memory, validates them, and synthesizes a final report. If quality is below threshold, it retries. If it can't recover after two retries, it escalates to the human in the loop (me).
Loop engineering here refers to the retry-escalate structure. The critic agent scores findings on two dimensions: staleness (does a newer version of this fact exist?) and grounding (does the text support the claim?). If quality falls below 0.7 (threshold), the loop sends the researcher agent back. If retries are exhausted, the loop escalates rather than hallucinate.
The 0.7 threshold was set experimentally. Below it, findings consistently had unfounded claims. Above 0.7, the critic agent was reliable enough to approve. The structure of the loop is outlined below.
- The planner breaks queries into 2-3 subtasks.
- The researcher queries memory and returns the top hit per subtask.
- The critic checks each finding against
current_value(entity, relation)for staleness, then runs an LLM grounding check. - The synthesiser writes the final report from approved findings.
The memory interface is a single abstract class:
class Memory(ABC):
async def write(self, finding: Finding) -> None: ...
async def query(self, query_text: str, k: int = 5) -> list[Finding]: ...
async def current_value(self, entity: str, relation: str) -> Finding | None: ...current_value exists as a separate method because query is semantic. It returns whatever scores highest on similarity, which can include adjacent facts about the same entity. The critic needs a more precise structural lookup: for this exact entity and relation, what is the live value? current_value bypasses embedding entirely and looks up by key. Without it, you're basically using retrieval to check retrieval, which makes no sense.
Both implementations (ChromaDB and Graphiti + Neo4j) call this same Memory class, so the loop never knows which backend it's talking to.
The eval harness
I built an eval harness that tracks why an answer is wrong. Specifically, whether or not staleness is what caused the failure in the loop.
The dataset: 30 queries across three types.
- Static fact (n=10) - one correct fact per query without any temporal element. This is the control group to confirm the loop works before adding temporal complexity.
- Staleness-sensitive (n=15) - two versions of a fact seeded into memory before each query. V1 is the older value (wrong answer for the query), V2 is the newer value (correct answer). The query asks for the current value. Examples: a company's CEO changed, or a user's favourite movie changed.
- Historical belief (n=5) - same two-version setup, but the query asks for the past state. The older fact is the correct answer. I included this subset to test whether the memory backend can surface past facts, not just current ones.
The anti-cheat constraint applies to all staleness items: V1 and V2 texts must be indistinguishable without timestamps. Recency words such as "former," "previously," "outdated," "no longer" are all banned from the dataset, and there are no explicit dates in the facts. Both V1 and V2 describe their fact as currently true. A reader encountering only one version would believe it. This ensures the temporal mechanism is what determines retrieval outcome, not surface cues.
Scoring tracks whether the ground truth appears in the final report, whether a stale fact slipped through, and whether staleness caused the failure. Each query gets a fresh memory instance and a unique partition ID, so facts from different queries never bleed into each other.
Note: Embeddings for both architectures use a local sentence-transformers model rather than an external API. This keeps the experiment self-contained; no embedding API latency or cost variability bleeding into the results, and also ensures the same embedding model runs across both backends.
How each backend handles stale facts
The backends differ at one point: what each memory layer returns when two versions of a fact exist.
In ChromaDB, both V1 (Jonathan Hale, CEO, Jun 2024) and V2 (Dr. Priya Nair, CEO, Dec 2025) sit in the vector store with no temporal metadata that affects retrieval. ChromaDB has no temporal metadata field. There is no invalid_at concept in a flat vector store, documents just sit in the index with no notion of when they were valid. When the researcher queries "Who is the CEO of Meridian Bio?", cosine similarity decides. Because V1 and V2 are written in similar style and on the same topic (that's the anti-cheat guarantee), either could win. In practice, V1 consistently won the similarity contest in this eval, landing the researcher on the stale fact.
In Graphiti, when V2 is written, a Cypher query marks V1 with invalid_at = V2.valid_at. V1 is now superseded at the graph layer. The researcher's query() method filters to invalid_at IS NULL before returning results, making V1 invisible. The researcher agent gets V2 on the first retrieval.
The researcher agent doesn't need to be smarter, nor does the critic need a better staleness check. No agent code changes are required; fixing the memory layer is enough.
Results
+----------------------------------+--------------------+--------------------+
| Metric | ChromaDB | Graphiti |
+----------------------------------+--------------------+--------------------+
| Overall accuracy (30) | 50% | 77% |
| Staleness accuracy (15) | 13% | 80% |
| Staleness error rate (15) | 87% | 20% |
| Staleness-caused failures (15) | 87% | 0% |
| Critic false-approve rate (15) | 0% | 0% |
| Historical-belief retrieval accuracy (5) | 60% | 100% |
+----------------------------------+--------------------+--------------------+Staleness-sensitive queries: Chroma got 2 of 15 correct (13%). Graphiti got 12 of 15 (80%). That's a 67% difference.
Staleness-caused failures: Chroma produced a staleness_failure tag in 87% of staleness queries. Graphiti produced zero. Every Chroma failure traced to the same loop trap: detect staleness, retry, retrieve the same stale fact, retry again, escalate.
Graphiti's 3 staleness failures (20%): All three were grounding check false negatives from the 8B model, not staleness failures. In all three cases the retriever returned V2 correctly, which I verified by inspecting the researcher's raw output in the eval log. However, the synthesiser either misquoted it or failed the grounding check due to model inconsistency. Graphiti had zero cases where V1 made it into the final report.
Historical belief retrieval: Chroma 60%, Graphiti 100%. Graphiti's retrieval was correct on all 5 queries. 3 of 5 didn't reach the final report correctly, but those are failures caused by the 8B model (small) used for the synthesis agent, not retrieval or memory architecture failures. See point-in-time retrieval section below.
The failure trace
The failure trace for a staleness-sensitive query traces directly to the retrieval mechanism.
Chroma executes 8 steps to produce no answer. Graphiti executes 5 steps to produce the correct answer. The difference is entirely at the researcher's retrieval step: which fact gets returned from memory. Everything else in the loop is identical.
Staleness produces escalations where the loop burns its full retry budget, incurs the latency of all those LLM calls, and returns nothing. The latency gap between ChromaDB (11,937ms) and Graphiti (9,243ms) on staleness queries is more about retry cost than architecture speed. Escalated ChromaDB queries ran three full researcher-critic cycles before returning nothing.
Note: both latency numbers are bottlenecked by Groq free tier queue time, not architecture speed. The absolute numbers aren't production-representative. But the relative gap between them is. It reflects how many LLM calls each path burned before exiting, which is expensive in production.
Point-in-time retrieval
The historical-belief subset tests a different capability: can the memory backend surface a past fact when a newer version exists? The queries ask about past state, where V1 is the correct answer, even though V2 introduces a newer answer. For example, take this query:
- "Who managed the Cascade Growth Fund when it had fewer than forty portfolio companies?"
V1 is the correct answer (Thomas Aldrich, when the portfolio was smaller). V2 introduces a newer managing partner (Elena Vasquez, after the portfolio grew). The ground truth is V1.
Graphiti supports point-in-time retrieval via date filters on valid_at and invalid_at. With reference_time set to V1's timestamp, the retriever surfaces V1 instead of V2. The critic is also updated: for historical queries, the staleness check verifies the finding was valid at the reference time rather than comparing against the live value. With both changes, Graphiti achieves 100% retrieval accuracy on all 5 historical belief queries.
ChromaDB has no equivalent mechanism. In this eval, it retrieved V1 more often than V2, but not because of any retrieval mechanism. Since both V1 and V2 were written in the same style and topic, their cosine similarity scores were similar. Therefore, ChromaDB got 3 of 5 correct, which is essentially random.
Caveats
1. Sample size. n=15 on the staleness subset is enough to observe a clear directional effect but not enough for strong statistical claims. The 67 percentage point gap is large, but these exact numbers are indicative rather than definitive.
2. Agent LLM is 8B. The planner, critic grounding check, and synthesiser all use llama-3.1-8b-instant via Groq free tier. The retrieval misalignment failures and some static-fact misses are 8B inconsistency. A stronger model would reduce noise.
3. Synthetic dataset. All facts use fabricated entities with clean entity-relation-value structure. The anti-cheat constraint keeps the staleness hard to detect without temporal metadata, which is the right property for this test. Real data tends to be messier.
4. Oracle Timestamps. Point-in-time retrieval in this eval relies on the harness supplying an exact reference_time to the memory backend. In production, the planner agent would need to parse temporal intent from natural language and convert it to a datetime before calling memory. That extraction step is non-trivial and unsolved here. The 100% historical-belief accuracy is real, but it assumes temporal intent is already resolved upstream.
Architecture implications
Flat vector RAG is the correct choice when:
- Facts in the domain are static or change infrequently
- Queries ask about a single point in time (usually "now")
- Simplicity and operational cost matter more than temporal precision
Temporal graph memory earns its complexity when:
- Facts change and the system needs to track what changed and when
- Queries ask "what is current?" for entities that have multiple versions in memory
- Queries ask about past state and the loop can supply a temporal context
- Loop-based architectures retry on failure (flat RAG creates inescapable retry loops for stale facts)
The last point is the one that gets missed. Temporal memory is often framed as a "knowledge base" improvement. It's actually a loop engineering improvement. Without it, a well-designed critic that correctly detects staleness makes the system worse. It burns retries on a problem the retriever can never solve.
Stack
Python, LangGraph, ChromaDB, Graphiti, Neo4j (Docker), Groq API (free tier), sentence-transformers for local embeddings.