Skip to content

Performance: _memory_do_update reads every checkpoint file even on no-opΒ #152

@tcconnally

Description

@tcconnally

Severity: 🟑 Medium (performance)

agora.py:34–57: the early-out at line 56 ("Nothing new since last update") only fires after all checkpoint files have been read and parsed. For a workspace with hundreds of checkpoints, this is O(n) on every render that triggers an auto-update.

Suggested fix

Short-circuit on count check before parsing:

cp_files = sorted(_list_checkpoint_files(cfg), key=lambda f: f.name)
cp_total = len(cp_files)

mp = _mneme_path(workspace, cfg)
fm, body = _load_narrative(mp)
cp_hwm = int(fm.get("checkpoints_processed", 0))
py_hwm = _mneme_pythia_hwm(fm) if fm else 0
py_total = _count_pythia_entries()  # cheap line count

if cp_total <= cp_hwm and py_total <= py_hwm and body.strip():
    return (False, "Nothing new since last update.")

# Only load checkpoints we actually need (cp_hwm onwards)
new_checkpoints = []
for fp in cp_files[cp_hwm:]:
    cp = _load_checkpoint_file(fp)
    if cp: new_checkpoints.append(cp)

For Pythia, maintain a .pythia_log.jsonl.count sidecar updated on every append.

Acceptance criteria

  • Bench: with 500 checkpoints and no new ones, _memory_do_update returns in <50ms.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions