Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 2 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,12 +42,8 @@ jobs:
# actually exercised without a full 3x3 runner-minute blow-up.
- os: macos-latest
python-version: '3.12'
# windows-latest is intentionally omitted from the gating matrix:
# the suite has real Windows issues (chromadb holds open file
# handles so tempdir teardown hits WinError 32, event-log rotation
# relies on POSIX rename-over-open, and a concurrent-append test
# loses writes). Tracked separately before Windows can be claimed
# as supported — see the Windows-support tracking issue.
- os: windows-latest
python-version: '3.12'

steps:
- uses: actions/checkout@v6
Expand Down
2 changes: 1 addition & 1 deletion docs/COMPATIBILITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
|----|--------|-----------------|-------|
| Linux | ✅ Full | Ubuntu 20.04+ | CI-verified on every PR (Python 3.10–3.12) |
| macOS | ✅ Full | 11.0+ | CI-verified on every PR; x86 and Apple Silicon |
| Windows | ⚠️ Experimental | 10, 11 | Installs and runs; **not yet CI-green** — the test suite has known Windows issues (ChromaDB holds open file handles so temp-dir teardown hits `WinError 32`, event-log rotation relies on POSIX rename-over-open, and a concurrent-append path loses writes). Task Scheduler walkthrough works; full support is tracked before Windows is gated in CI. |
| Windows | ✅ Full | 10, 11 | CI-verified on every PR (`windows-latest`, Python 3.12); Task Scheduler walkthrough included |
| Docker | ✅ Full | Docker 20.10+ | Included in releases |

---
Expand Down
10 changes: 5 additions & 5 deletions docs/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@
"@type": "SoftwareApplication",
"name": "NeuralMind",
"applicationCategory": "DeveloperApplication",
"operatingSystem": "Linux, macOS",
"operatingSystem": "Linux, macOS, Windows",
"description": "Persistent memory and semantic code intelligence for AI coding agents. Local-first code indexing with 40–70× per-query token reduction, a brain-like synapse layer that learns associations from how you use the codebase, an Obsidian-style graph view, an MCP server, and a built-in install doctor. Works with Claude Code, Cursor, Cline, Continue, and any MCP-compatible agent.",
"url": "https://dfrostar.github.io/neuralmind/",
"downloadUrl": "https://pypi.org/project/neuralmind/",
"softwareVersion": "0.23.0",
"softwareVersion": "0.24.0",
"license": "https://opensource.org/licenses/MIT",
"programmingLanguage": "Python",
"operatingSystemRequirements": "Python 3.10+",
Expand Down Expand Up @@ -355,7 +355,7 @@
</div>
<div class="container hero-grid">
<div>
<p class="badge"><span class="dot"></span> v0.23.0 — latest release&nbsp;·&nbsp;<a href="https://github.com/dfrostar/neuralmind/blob/main/RELEASE_NOTES_v0.23.0.md">release notes</a></p>
<p class="badge"><span class="dot"></span> v0.24.0 — latest release&nbsp;·&nbsp;<a href="https://github.com/dfrostar/neuralmind/blob/main/RELEASE_NOTES_v0.24.0.md">release notes</a></p>
<h1>Persistent memory for <span class="grad">AI coding agents</span></h1>
<p class="hero-sub">Your agent learns your codebase the way a senior engineer would — <strong>what files go together, what you usually edit next, what patterns matter</strong>. The memory persists across sessions and surfaces automatically.</p>
<p class="hero-local"><strong>100% local.</strong> Your code never leaves your machine. Side effect: <strong>40–70× cheaper code questions</strong>, measured in CI on every commit.</p>
Expand Down Expand Up @@ -584,11 +584,11 @@ <h2>What's new</h2>
<p class="lede">Shipped in small, verifiable increments — every release gated by the CI benchmark.</p>
<div class="timeline">
<div class="tl-item dev">
<div class="tl-head"><b>v0.24.0</b><span class="pill pill-dev">In development</span></div>
<div class="tl-head"><b>v0.24.0</b><span class="pill pill-latest">Latest release</span></div>
<p><b>Memory namespaces &amp; branch isolation.</b> The synapse layer becomes namespace-aware: <code>branch:&lt;name&gt;</code> / <code>personal</code> / <code>shared</code> / <code>ephemeral</code> memory live separately in the same store, so a feature-branch spike can't pollute what the agent learned about <code>main</code>. Recall reads a transparent merged view (active branch 1.0&times; &gt; personal 0.8&times; &gt; shared team baseline 0.5&times;, attributed per-namespace in <code>query --trace</code>), and the new <code>neuralmind memory {inspect,reset,export,import}</code> moves memory as versioned JSON bundles — the team-memory on-ramp. Existing learned memory migrates losslessly into <code>personal</code>. <a href="https://github.com/dfrostar/neuralmind/blob/main/RELEASE_NOTES_v0.24.0.md">Release notes →</a></p>
</div>
<div class="tl-item">
<div class="tl-head"><b>v0.23.0</b><span class="pill pill-latest">Latest release</span></div>
<div class="tl-head"><b>v0.23.0</b></div>
<p>Four future-proofing foundations: a <b>schema-versioned index contract (IR)</b> checked by the new <code>neuralmind validate</code>, a <b>retrieval-quality harness</b> (<code>benchmark --quality</code>: precision@k / recall@k / MRR over 30 golden queries, failing CI on regression), <b>debug traces</b> (<code>query --trace</code> shows why a result came back), and an experimental <b>local daemon</b> that keeps project state warm. <a href="https://github.com/dfrostar/neuralmind/blob/main/RELEASE_NOTES_v0.23.0.md">Release notes →</a></p>
</div>
<div class="tl-item">
Expand Down
105 changes: 91 additions & 14 deletions neuralmind/core.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
"""

import json
import os
import threading
from datetime import datetime, timezone
from pathlib import Path
Expand All @@ -37,6 +38,51 @@

DEFAULT_HYBRID_HIGHLIGHT_COUNT = 3

# Serializes recent-queries appends within this process. POSIX O_APPEND
# already makes single-line appends atomic, but Windows' CRT implements
# append mode as a separate seek-to-end + write, so two handles writing
# concurrently can interleave and lose lines.
_RECENT_QUERIES_APPEND_LOCK = threading.Lock()

try:
import msvcrt # Windows-only: cross-process advisory lock below.
except ImportError: # pragma: no cover - POSIX
msvcrt = None # type: ignore[assignment]


def _lock_byte0(fd: int) -> bool:
"""Best-effort cross-process mutex on byte 0 of *fd* (Windows only).

POSIX writers don't need it (O_APPEND is atomic) and use flock for
compaction instead; on Windows both the appender and the compactor
take this same region so a compaction's read-truncate-rewrite can't
drop a concurrent process's append. Non-blocking with a short retry
so a stuck holder can never stall a query.
"""
if msvcrt is None:
return False
import time as _time

for _ in range(50): # ~50ms worst case
try:
os.lseek(fd, 0, os.SEEK_SET)
msvcrt.locking(fd, msvcrt.LK_NBLCK, 1)
return True
except OSError:
_time.sleep(0.001)
return False


def _unlock_byte0(fd: int) -> None:
if msvcrt is None:
return
try:
os.lseek(fd, 0, os.SEEK_SET)
msvcrt.locking(fd, msvcrt.LK_UNLCK, 1)
except OSError:
pass


# Canonical IR artifacts (PRD 1), under <project>/.neuralmind/.
IR_FILENAME = "index_ir.json"
IR_META_FILENAME = "ir_meta.json"
Expand Down Expand Up @@ -176,6 +222,22 @@ def __init__(
def backend_name(self) -> str:
return self.backend_manager.backend_name

def close(self) -> None:
"""Release backend resources (vector-store file handles).

Windows can't delete files a process still holds open, so
anything that removes the project directory afterwards — test
teardown, ``neuralmind reset`` — needs this. Safe to call more
than once; the synapse store opens its sqlite database per
operation and holds nothing between calls.
"""
embedder = getattr(self, "embedder", None)
if embedder is not None and hasattr(embedder, "close"):
try:
embedder.close()
except Exception:
pass

@property
def memory_namespace(self) -> str:
"""The active synapse-memory namespace for this project (PRD 4).
Expand Down Expand Up @@ -763,12 +825,14 @@ def _record_recent_query(self, question: str, result: ContextResult) -> None:
UI can highlight on the canvas. Always on (local-only data,
readable only through the auth-gated server).

The hot path is an atomic single-line append (POSIX O_APPEND
guarantees writes < PIPE_BUF don't tear across processes), so
CLI and MCP-server processes can safely write to the same file
without losing entries. Trimming back to RECENT_QUERIES_MAX is
a lazy compaction step gated by file size and protected by an
advisory lock — see ``_compact_recent_queries``.
The hot path is a single-line O_APPEND write. On POSIX that is
atomic across processes by itself (writes < PIPE_BUF don't
tear); Windows' CRT implements append as seek-to-end + write,
so writes are additionally serialized by a process-local lock
and a best-effort cross-process byte-range lock. Trimming back
to RECENT_QUERIES_MAX is a lazy compaction step gated by file
size and protected by the same locks — see
``_compact_recent_queries``.

Gated on the same consent flag as the learning log
(`NEURALMIND_MEMORY`): if the user opted out of persisting
Expand Down Expand Up @@ -799,9 +863,17 @@ def _record_recent_query(self, question: str, result: ContextResult) -> None:
}
log_path = self._recent_queries_path()
log_path.parent.mkdir(parents=True, exist_ok=True)
line = json.dumps(record, sort_keys=True) + "\n"
with log_path.open("a", encoding="utf-8") as f:
f.write(line)
encoded = (json.dumps(record, sort_keys=True) + "\n").encode("utf-8")
with _RECENT_QUERIES_APPEND_LOCK:
fd = os.open(str(log_path), os.O_WRONLY | os.O_CREAT | os.O_APPEND, 0o644)
locked = False
try:
locked = _lock_byte0(fd)
os.write(fd, encoded)
finally:
if locked:
_unlock_byte0(fd)
os.close(fd)
except Exception:
# Recording must never block the actual query.
return
Expand All @@ -814,10 +886,11 @@ def _compact_recent_queries(self, log_path: Path) -> None:
"""Trim the recent-queries log back to RECENT_QUERIES_MAX entries.

Only runs when the file has grown past the size threshold, so the
hot path stays append-only. Uses an advisory file lock on POSIX
so the read-truncate-rewrite isn't interleaved with another
process's append; on Windows the lock is best-effort and worst
case is a slightly oversized log file (no data loss).
hot path stays append-only. The read-truncate-rewrite is guarded
against another process's append by flock on POSIX and by the
byte-0 region lock shared with ``_record_recent_query`` on
Windows; both are best-effort — worst case is a slightly
oversized log file (no data loss).
"""
try:
size = log_path.stat().st_size
Expand All @@ -829,20 +902,24 @@ def _compact_recent_queries(self, log_path: Path) -> None:
import fcntl
except ImportError:
fcntl = None # type: ignore[assignment]
with log_path.open("r+", encoding="utf-8") as f:
with _RECENT_QUERIES_APPEND_LOCK, log_path.open("r+", encoding="utf-8") as f:
if fcntl is not None:
try:
fcntl.flock(f.fileno(), fcntl.LOCK_EX)
except OSError:
pass
locked = _lock_byte0(f.fileno())
try:
lines = [ln for ln in f if ln.strip()]
if len(lines) <= self.RECENT_QUERIES_MAX:
return
f.seek(0)
f.truncate()
f.writelines(lines[-self.RECENT_QUERIES_MAX :])
f.flush()
finally:
if locked:
_unlock_byte0(f.fileno())
if fcntl is not None:
try:
fcntl.flock(f.fileno(), fcntl.LOCK_UN)
Expand Down
22 changes: 22 additions & 0 deletions neuralmind/daemon.py
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,28 @@ def clear_discovery(path: Path | None = None) -> None:


def _pid_alive(pid: int) -> bool:
if os.name == "nt":
# os.kill(pid, 0) is NOT a liveness probe on Windows:
# signal.CTRL_C_EVENT == 0, so it sends a real Ctrl-C to the
# console process group — interrupting the daemon's (or test
# runner's) own console. Query the process handle instead.
import ctypes

PROCESS_QUERY_LIMITED_INFORMATION = 0x1000 # noqa: N806 - WinAPI name
STILL_ACTIVE = 259 # noqa: N806
kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
handle = kernel32.OpenProcess(PROCESS_QUERY_LIMITED_INFORMATION, False, pid)
if not handle:
# ERROR_ACCESS_DENIED (5): the process exists but is owned
# by someone else — alive, same as the PermissionError arm.
return ctypes.get_last_error() == 5
try:
code = ctypes.c_ulong()
if not kernel32.GetExitCodeProcess(handle, ctypes.byref(code)):
return False
return code.value == STILL_ACTIVE
finally:
kernel32.CloseHandle(handle)
try:
os.kill(pid, 0)
except ProcessLookupError:
Expand Down
32 changes: 26 additions & 6 deletions neuralmind/embedder.py
Original file line number Diff line number Diff line change
Expand Up @@ -501,9 +501,29 @@ def clear(self) -> None:
pass

def close(self) -> None:
"""Close ChromaDB client and cleanup."""
if hasattr(self, "client"):
try:
self.client.delete_collection(name=self.COLLECTION_NAME)
except Exception:
pass
"""Release the ChromaDB client's file handles.

Chroma has no public ``close()``: it caches one ``System`` per
storage path (holding the sqlite connection pool and HNSW
segment files) for the life of the process. Windows refuses to
delete open files, so anything that removes the store afterwards
— temp-dir teardown in tests, a user deleting ``.neuralmind/`` —
hits ``WinError 32`` unless the System is actually stopped and
evicted from that cache. Deleting the collection (the previous
behavior) released nothing and destroyed data a later open
expected to find.
"""
client = getattr(self, "client", None)
if client is None:
return
self._collection = None
self.client = None
try:
from chromadb.api.shared_system_client import SharedSystemClient

client._system.stop()
SharedSystemClient._identifier_to_system.pop(getattr(client, "_identifier", ""), None)
except Exception:
# Best-effort across chromadb versions: worst case is the
# pre-close behavior (handles live until process exit).
pass
57 changes: 56 additions & 1 deletion neuralmind/event_log.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,61 @@
_MAX_LINE_BYTES = 64 * 1024 # Skip pathologically long lines.


def _open_shared_rb(path: Path):
"""Open *path* for binary read without blocking a concurrent rename.

The tailer holds its read handle across poll intervals, and rotation
is a logrotate-style rename of the live file. POSIX allows renaming
an open file; Windows' ``open()`` omits FILE_SHARE_DELETE from the
sharing mode, so the rotating process gets a PermissionError instead.
Recreate the POSIX semantics with CreateFileW + share-delete so
rotation never depends on catching the tailer between polls.
"""
if os.name != "nt":
return open(path, "rb")

import ctypes
import msvcrt

GENERIC_READ = 0x80000000 # noqa: N806 - canonical WinAPI names
FILE_SHARE_READ = 0x00000001 # noqa: N806
FILE_SHARE_WRITE = 0x00000002 # noqa: N806
FILE_SHARE_DELETE = 0x00000004 # noqa: N806
OPEN_EXISTING = 3 # noqa: N806

kernel32 = ctypes.WinDLL("kernel32", use_last_error=True)
kernel32.CreateFileW.restype = ctypes.c_void_p
kernel32.CreateFileW.argtypes = [
ctypes.c_wchar_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
ctypes.c_uint32,
ctypes.c_uint32,
ctypes.c_void_p,
]
handle = kernel32.CreateFileW(
str(path),
GENERIC_READ,
FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
None,
OPEN_EXISTING,
0,
None,
)
if handle is None or handle == ctypes.c_void_p(-1).value:
# WinError maps ERROR_FILE_NOT_FOUND to FileNotFoundError, which
# _open() relies on to distinguish "not yet created" from real
# failures.
raise ctypes.WinError(ctypes.get_last_error())
try:
fd = msvcrt.open_osfhandle(handle, os.O_RDONLY)
except OSError:
kernel32.CloseHandle(ctypes.c_void_p(handle))
raise
return os.fdopen(fd, "rb")


class EventLogWriter:
"""Append JSON events to a line-delimited file. Thread-safe."""

Expand Down Expand Up @@ -115,7 +170,7 @@ def _open_at_start(self):

def _open(self, *, seek_to_end: bool):
try:
fh = open(self.path, "rb")
fh = _open_shared_rb(self.path)
except FileNotFoundError:
return None, None
except OSError:
Expand Down
Loading
Loading