How to Implement Memory Compression for AI Agents: Merging Old Memories When the Count Exceeds 60

by

AI agents get forgetful in a very specific way. They do not forget like humans do. They run out of usable context.
In my agent framework, Spark, the long-term memory store became noticeably less useful after reaching approximately 60 records. Retrieval latency increased, prompts contained more irrelevant material, and the agent occasionally asked questions it had already answered. The problem was not storage capacity. It was memory quality.
The solution was not to delete old memories. It was to compress them: merge related historical entries into a smaller number of denser, better-organized memories.
This article describes the production pipeline I built in a Tauri and Rust agent. The design follows three principles:

  1. Preserve important signals.
  2. Remove redundancy.
  3. Make every merge auditable.

The 60-Memory Wall

The number 60 is not a universal constant. It is an operational threshold.
For an active user, 60 memories represented roughly two to three weeks of daily interaction. At that point, several records often described the same preference, project, or recurring decision. For example, the store might contain separate memories such as:

  • The user prefers concise technical explanations.
  • The user dislikes unnecessary introductions.
  • The user usually wants code changes rather than theoretical proposals.
  • The user prefers tests to be run after implementation.
    These records are individually valid, but together they are redundant. Sending all of them to the model wastes tokens and makes retrieval less precise.
    An unbounded memory store creates three forms of technical debt:
  • Search latency grows as more records compete for retrieval.
  • Prompt size increases, raising inference cost.
  • Similar memories compete with one another and can produce inconsistent answers.
    The 60-record threshold is therefore a practical point at which maintenance begins before quality visibly collapses. A good threshold should be easy to explain, measurable in production, and adjustable when usage patterns change.

Compress, Do Not Delete

Compression changes the unit of memory.
Instead of treating every interaction as a permanent, independent record, the system periodically transforms a group of related records into a summary memory. The summary contains the stable information needed for future decisions, while transient details remain available through an audit trail.
The process should not compress everything indiscriminately. Recent, highly specific, or safety-sensitive memories should be protected. The initial implementation compresses only older records and leaves a recent window untouched.
For example, when the store reaches 61 memories:

  • Keep the newest 20 records unchanged.
  • Select approximately 30 to 35 older records as compression candidates.
  • Group candidates by topic or semantic similarity.
  • Generate one or more replacement memories.
  • Preserve the original records as archived shadow data.
  • Record the merge operation and its statistics.
    The result is usually 61 records becoming approximately 30 to 40 active memories, rather than being reduced to one overly broad summary.

The Compression Pipeline

The pipeline has five stages.

1. Trigger Compression

The trigger runs after a successful memory insertion. A simple count check is sufficient:

if active_memory_count().await? > 60 {
    compress_old_memories().await?;
}

Compression should run asynchronously so that saving a new memory does not block the user interface. A lock or job marker prevents two compression tasks from running simultaneously.
The threshold should be configurable rather than hard-coded in application logic:

memory_compression_threshold = 60
memory_protected_recent_count = 20
memory_target_count = 40

2. Select Candidates

Candidate selection uses age, access frequency, importance, and protection flags.
A basic score can be expressed as:

compression_score =
    age_days * 0.5
    + (1 / (access_count + 1)) * 20
    - importance * 10

High-scoring memories are older and rarely retrieved. Records marked as permanent, security-sensitive, or user-pinned are excluded.
The system also detects obvious duplicates before asking a language model to summarize anything. Normalized text hashes catch exact duplicates. Token overlap catches near duplicates. For example, two records with more than 85 percent shared terms can be placed in the same preliminary group.
This deterministic filtering reduces unnecessary model calls and makes compression behavior easier to test.

3. Group Related Memories

Candidates are grouped by topic rather than summarized as one large block. Combining unrelated information produces vague memories that are difficult to retrieve.
A practical grouping strategy is:

  1. Compare embeddings or normalized keyword sets.
  2. Create a group when similarity exceeds 0.78.
  3. Require at least two records for a merge.
  4. Limit each group to 12 source memories.
  5. Leave isolated records unchanged.
    A group about development preferences might contain eight related records, while a group about a specific project might contain five. Keeping groups small prevents the summary from becoming a miniature transcript.

4. Generate a Structured Summary

The summarizer receives the source records and must return structured data, not free-form prose. The output includes the stable fact, confidence, scope, and source identifiers.
A useful format is:

{
  "title": "User's development workflow preferences",
  "summary": "The user prefers direct implementation over abstract discussion, concise explanations, and deterministic tests after code changes.",
  "confidence": 0.94,
  "facts": [
    "Prefer concise technical explanations",
    "Implement requested changes when intent is clear",
    "Run deterministic tests after bug fixes"
  ],
  "source_ids": ["m-104", "m-118", "m-121", "m-130"]
}

The prompt should explicitly prohibit unsupported conclusions. The model may consolidate repeated facts, but it must not invent preferences or silently resolve contradictions.
When conflicting records exist, the summary should preserve the uncertainty:

The user usually prefers concise explanations, but requests detailed analysis for architecture reviews.

This is safer than selecting one statement merely because it appears more frequently.

5. Replace Active Records

The original records are not physically deleted. They are marked as archived, and a new compressed record is inserted.
A simplified schema looks like this:

CREATE TABLE memory_merges (
    id INTEGER PRIMARY KEY,
    created_at TEXT NOT NULL,
    target_memory_id TEXT NOT NULL,
    source_count INTEGER NOT NULL,
    input_tokens INTEGER NOT NULL,
    output_tokens INTEGER NOT NULL,
    similarity_before REAL,
    retrieval_score_after REAL
);
CREATE TABLE memory_merge_sources (
    merge_id INTEGER NOT NULL,
    source_memory_id TEXT NOT NULL,
    PRIMARY KEY (merge_id, source_memory_id)
);

The replacement operation should be transactional:

BEGIN;
UPDATE memories
SET status = 'archived', merged_into = ?
WHERE id IN (?, ?, ?);
INSERT INTO memories
    (id, title, content, status, importance, created_at)
VALUES
    (?, ?, ?, 'active', ?, ?);
INSERT INTO memory_merges
    (target_memory_id, source_count, input_tokens, output_tokens)
VALUES
    (?, ?, ?, ?);
COMMIT;

If insertion or logging fails, the transaction rolls back. The agent never ends up with archived source records and no replacement.

What the System Actually Saves

In one representative workload, 61 raw memories occupied approximately 8,400 tokens when included in a broad retrieval prompt. After compression, 38 active memories required approximately 4,900 tokens for the same retrieval window, a reduction of about 42 percent.
The reduction was not limited to prompt size. Average retrieval latency fell from 180 milliseconds to 112 milliseconds because fewer records survived the first filtering stage. Over 1,000 interactions, the reduced prompt size saved approximately 3.5 million input tokens.
Compression quality must be measured separately from storage reduction. I use a fixed evaluation set containing questions whose answers depend on known memories. Before compression, the system answers the questions using the raw store. After compression, it repeats the evaluation using only active memories.
The key metrics are:

  • Recall: whether the required fact is retrieved.
  • Precision: how much retrieved content is relevant.
  • Contradiction rate: whether compressed memories conflict with known facts.
  • Token reduction: how much context was removed.
  • Latency: how long retrieval takes.
    A successful run reduced tokens by 42 percent while keeping fact recall at 97 percent. The three missed facts were low-importance, one-time details that had not been accessed for more than 30 days. That result was acceptable because those records remained available in the archive.

Making Compression Auditable

Auditability is essential because compression is lossy. When an agent gives an incorrect answer, developers need to know whether the original memory was missing, incorrectly merged, or simply not retrieved.
Every merge records:

  • The timestamp.
  • The source memory IDs.
  • The replacement memory ID.
  • The model and prompt version.
  • Input and output token counts.
  • Similarity scores.
  • The previous and new retrieval metrics.
  • Any warnings about conflicts or low confidence.
    The archived source records remain queryable for debugging and can be restored if necessary. A manual restore operation can mark the original records active and archive the generated summary.
    For multilingual systems, language should be considered during grouping. Memories written in different languages may describe the same fact, but language-specific tokenization and embeddings can produce unreliable similarity scores. I therefore allow cross-language merging only when both semantic similarity and extracted facts agree.

Trade-Offs and Failure Modes

Compression saves context, but it can remove useful detail.
It is usually a mistake to compress:

  • Exact commands or configuration values.
  • Legal, financial, or safety-sensitive instructions.
  • Temporary facts with unresolved consequences.
  • Memories explicitly pinned by the user.
  • Records with unresolved contradictions.
  • Information whose original wording matters.
    Another trade-off is model dependency. A summarization model can produce a fluent but inaccurate statement. Structured output, confidence thresholds, deterministic candidate selection, and archived sources reduce the risk, but they do not eliminate it.
    For low-confidence summaries, the system should keep the source memories active and mark the group for review rather than forcing a merge.

A Reproducible Playbook

A reliable implementation can follow this sequence:

  1. Measure retrieval latency, prompt tokens, and fact recall before compression.
  2. Choose a threshold based on observed overlap, not hardware capacity alone.
  3. Protect recent, pinned, and high-importance memories.
  4. Detect exact and near duplicates deterministically.
  5. Group only semantically related records.
  6. Generate structured summaries with source identifiers.
  7. Archive sources instead of deleting them.
  8. Replace records in a database transaction.
  9. Log token, latency, quality, and merge statistics.
  10. Re-run a fixed evaluation set after every compression cycle.
    Memory compression should not be treated as a one-time cleanup script. It is steady-state maintenance for any agent expected to operate over weeks or months.
    The central design decision is simple: preserve the information, not necessarily the original records. When the memory count exceeds 60, merging carefully selected historical entries can reduce cost and noise while retaining the facts that make the agent useful. With protected memories, structured summaries, measurable evaluation, and a complete audit trail, compression becomes a controlled engineering process rather than an irreversible guess.