Skip to content

getting-started.md

Latest commit

 

History

History
845 lines (581 loc) · 28.4 KB

getting-started.md

File metadata and controls

845 lines (581 loc) · 28.4 KB

Getting Started

This guide is written for someone who just wants Memory Layer working with as little setup friction as possible.

Table of Contents

Prerequisites

Before you install or run the wizard, have these ready:

  • a PostgreSQL database that already exists and that Memory Layer can own
  • a PostgreSQL connection string for that database
  • a project directory or repository you want Memory Layer to manage
  • optional: an OpenAI-compatible API key if you want memory scan
  • PostgreSQL with pgvector installed and CREATE EXTENSION vector enabled in the Memory Layer database
  • go on PATH if you plan to use the repo-local Memory Layer skills through go run

You do not need to invent a Memory Layer service token yourself for normal installs. Setup generates a machine-local token automatically in memory-layer.env, and local write-capable tools use that token to authenticate to mem-service.

PostgreSQL Requirement

Memory Layer stores durable memories in PostgreSQL. The backend cannot become healthy until the database URL points at a reachable PostgreSQL database that already exists. memory wizard --global writes the URL into config; it does not create the PostgreSQL database for you. Semantic retrieval and current embedding migrations also require pgvector.

There are two pgvector steps, and both matter:

  1. Install pgvector on the PostgreSQL server.
  2. Enable the vector extension inside the specific database Memory Layer uses.

The extension is per database. Enabling it in postgres does not enable it in memory_layer.

Example database URL:

postgres://memory_layer:<password>@127.0.0.1:5432/memory_layer

Existing Or Hosted PostgreSQL

Use this path when you already have PostgreSQL, including a hosted provider:

  1. Confirm the provider or server supports pgvector.
  2. Create a dedicated database and user, or ask your database admin for a URL to an existing database.
  3. From the machine that will run Memory Layer, verify connectivity:
export DATABASE_URL='postgres://memory_layer:<password>@db-host:5432/memory_layer'
psql "$DATABASE_URL" -c "SELECT 1;"
  1. Enable pgvector in the target database:
psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"

If the CREATE EXTENSION command fails with a permissions error, ask the database admin to run it or grant extension privileges for that database.

Local Debian Or Ubuntu PostgreSQL

Use this when you want PostgreSQL on the same machine as Memory Layer:

sudo apt-get update
sudo apt-get install -y postgresql postgresql-contrib
pg_config --version

Install the pgvector package that matches your PostgreSQL server major version. For PostgreSQL 16, the package is commonly:

sudo apt-get install -y postgresql-16-pgvector

If your server is a different major version, replace 16 with that version. If the package is unavailable from your OS repositories, install it from the PostgreSQL Global Development Group packages or follow the upstream pgvector installation instructions.

Create a dedicated user and database before running memory wizard --global:

sudo -u postgres createuser --pwprompt memory_layer
sudo -u postgres createdb --owner=memory_layer memory_layer
export DATABASE_URL='postgres://memory_layer:<password>@127.0.0.1:5432/memory_layer'

Enable and verify pgvector:

psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql "$DATABASE_URL" -c "SELECT 1;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"

Local macOS PostgreSQL

Use this when you want Homebrew PostgreSQL on the same Mac as Memory Layer:

brew install postgresql@16 pgvector
brew services start postgresql@16

Create a dedicated user and database before running memory wizard --global:

createuser --pwprompt memory_layer
createdb --owner=memory_layer memory_layer
export DATABASE_URL='postgres://memory_layer:<password>@127.0.0.1:5432/memory_layer'

Enable and verify pgvector:

psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql "$DATABASE_URL" -c "SELECT 1;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"

After Memory Layer is configured, run:

memory doctor
memory health

memory doctor should not report a missing database URL or missing pgvector before you treat the install as complete.

Agent Install Prompt

Give this prompt to an agent when you want it to install Memory Layer for you:

# Install Memory Layer

You are installing Memory Layer for me. Work in the terminal, explain before using sudo, and stop before destructive changes.

## Goal

Install Memory Layer completely on this machine and configure it for the project I choose.

Repository: https://github.com/3vilM33pl3/memory
GitHub Releases: https://github.com/3vilM33pl3/memory/releases

## Rules

- Detect whether this is Linux/Debian-style or macOS.
- Do not invent secrets.
- PostgreSQL is required. Before running `memory wizard --global`, find an existing database URL or ask me whether to use an existing/hosted PostgreSQL database or create a local one.
- If creating a local PostgreSQL database, create a dedicated database and user named `memory_layer` unless I ask for different names.
- Do not invent the database password; ask me for it or generate one only after confirming that is OK.
- Make sure the PostgreSQL server has pgvector installed and that the target database has `CREATE EXTENSION IF NOT EXISTS vector;` applied.
- Verify PostgreSQL with `psql "$DATABASE_URL" -c "SELECT 1;"` and verify pgvector with `psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"` before configuring Memory Layer.
- Ask me for optional LLM or embedding API keys only if I want scan or semantic retrieval.
- Make sure Go is available on PATH so repo-local Memory Layer skills can run.
- Run health checks before saying the install is done.

## Linux / Debian path

1. Download the latest Memory Layer `.deb` from https://github.com/3vilM33pl3/memory/releases.
2. Install it with `sudo dpkg -i memory-layer_<version>_amd64.deb`.
3. Prepare PostgreSQL before configuring Memory Layer:
   - If using a hosted/existing database, verify that it accepts connections from this machine and supports pgvector.
   - If creating a local database, install PostgreSQL and the matching pgvector package for the server major version, for example `postgresql-16-pgvector` when the server is PostgreSQL 16.
   - Create or receive a database URL such as `postgres://memory_layer:<password>@127.0.0.1:5432/memory_layer`.
   - Run `psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"`.
   - Run `psql "$DATABASE_URL" -c "SELECT 1;"` and `psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"`.
4. Run `memory wizard --global` and configure the verified database URL and optional LLM/embedding settings.
5. Go to my target project directory.
6. Run `memory wizard --dry-run`, then `memory wizard` for repo-local setup.
7. Start the backend with `sudo systemctl enable --now memory-layer.service`.
8. Run `memory doctor`, `memory health`, and then open `memory tui`.

## macOS path

1. Run `brew tap 3vilM33pl3/memory https://github.com/3vilM33pl3/memory`.
2. Run `brew install 3vilM33pl3/memory/memory-layer`.
3. Prepare PostgreSQL before configuring Memory Layer:
   - If using a hosted/existing database, verify that it accepts connections from this machine and supports pgvector.
   - If creating a local database, use Homebrew PostgreSQL and pgvector, then create a dedicated `memory_layer` database and user.
   - Create or receive a database URL such as `postgres://memory_layer:<password>@127.0.0.1:5432/memory_layer`.
   - Run `psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"`.
   - Run `psql "$DATABASE_URL" -c "SELECT 1;"` and `psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"`.
4. Run `memory wizard --global` and configure the verified database URL and optional LLM/embedding settings.
5. Go to my target project directory.
6. Run `memory wizard --dry-run`, then `memory wizard` for repo-local setup.
7. Start the backend with `memory service enable`.
8. Run `memory doctor`, `memory health`, and then open `memory tui`.

## Finish

Report what was installed, where the config files are, whether the service is healthy, and what I should run next.

Agent Repo Memory Init Prompt

Give this separate prompt to an agent when Memory Layer is installed already and you want it to configure a specific project:

# Initialize Repository For Memory Layer

You are configuring an existing project for Memory Layer. Work in the terminal, explain before changing files, and stop before destructive changes.

## Goal

Create or refresh repo-local Memory Layer configuration so agents can write, query, and curate project memory safely.

## Rules

- Work in the target project directory I give you.
- Do not install the system package; this prompt is only for repo-local Memory Layer setup.
- Do not create or reinitialize git history.
- Do not delete existing `.mem/`, `.agents/`, or Memory Layer config files.
- If `.mem/project.toml`, legacy `.mem/config.toml`, `.agents/memory-layer.toml`, or `.agents/skills/` already exist, inspect them and preserve local customizations.
- Ask me before overwriting files, rotating credentials, importing history, or running a write operation that was not previewed.
- Make sure the shared backend is configured and healthy before saying setup is done.
- Make sure Go is available on `PATH` because repo-local Memory Layer skills use the Go helper.

## Steps

1. Run `memory health` and `memory doctor`.
2. Run `memory wizard --dry-run` in the target project to preview setup.
3. Run `memory wizard` to create or refresh the user-local project config, `.mem/project.toml`, and `.agents/` Memory Layer files.
4. Run `memory doctor` again.
5. If commit history should be available as evidence, run `memory commits sync --project <project-slug> --dry-run`, then run it for real only if the preview looks correct.
6. If an initial scan is wanted, run `memory scan --project <project-slug> --dry-run`, then run it for real only after I approve the preview.
7. Open `memory tui` or report the project slug and next commands.

## Finish

Report which repo-local files were created or preserved, the project slug, backend health, and any follow-up actions.

Fast Install: Debian

  1. Download the latest .deb package from the GitHub Releases page.
  2. Install it:
sudo dpkg -i memory-layer_<version>_amd64.deb
  1. Create the PostgreSQL database and prepare pgvector using the PostgreSQL Requirement section. Do not continue until these commands work:
psql "$DATABASE_URL" -c "SELECT 1;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"
  1. Configure the shared/global settings once on this machine:
memory wizard --global

This is where you set the shared database URL. The shared service API token is provisioned automatically if it is missing or still using the development placeholder. A writer ID is optional; if you do not set one, Memory Layer derives a stable writer identity automatically.

  1. Go to the project you want to use:
cd /path/to/your-project
  1. Preview and run the repo-local Memory Layer setup wizard:
memory wizard --dry-run
memory wizard

The repo-local skill bundle that memory wizard installs uses a shared Go helper under .agents/skills/memory-layer/scripts/, so agent-driven skill usage in that repository requires go to be available on PATH.

Most mutating memory commands also support --dry-run, so you can preview setup, write, indexing, bundle, and checkpoint operations before they touch local files, services, or backend state.

  1. Start the backend service:
sudo systemctl enable --now memory-layer.service
  1. Verify the setup and open the UI you prefer:
memory doctor
memory health
memory tui

Fast Install: macOS

  1. Tap this repository and install the formula:
brew tap 3vilM33pl3/memory https://github.com/3vilM33pl3/memory
brew install 3vilM33pl3/memory/memory-layer

If you specifically want the latest unreleased main branch:

brew install --HEAD 3vilM33pl3/memory/memory-layer
  1. Create the PostgreSQL database and prepare pgvector using the PostgreSQL Requirement section. Do not continue until these commands work:
psql "$DATABASE_URL" -c "SELECT 1;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"
  1. Configure the shared/global settings once on this machine:
memory wizard --global
  1. Go to the project you want to use:
cd /path/to/your-project
  1. Preview and run the repo-local Memory Layer setup wizard:
memory wizard --dry-run
memory wizard

The repo-local skill bundle that memory wizard installs uses a shared Go helper under .agents/skills/memory-layer/scripts/, so agent-driven skill usage in that repository requires go to be available on PATH.

  1. Start the backend LaunchAgent:
memory service enable
  1. Verify the setup and open the TUI:
memory doctor
memory health
memory tui

or in a browser:

http://127.0.0.1:4040/

What The Wizard Will Ask For

The wizard can set up:

  • shared/global settings when that scope is enabled:
    • the PostgreSQL database URL
    • the shared service API token override, if you want to replace the auto-generated one
    • an optional shared writer.id
  • optional LLM settings for scan
  • user-local project config plus the repo-local .mem/project.toml marker
  • optional watcher setup
  • the repo-local memory skill bundle, which uses a shared Go helper under .agents/skills/memory-layer/scripts/

The repo-local skill bundle contains:

  • memory-layer
  • memory-project-init
  • memory-github-init
  • memory-query-resume
  • memory-review-proposals
  • memory-plan-execution
  • memory-direct-task-start
  • memory-remember

Important detail:

  • inside a repository, memory wizard is local-first by default
  • use memory wizard --global when you want to edit the shared/global config
  • or enable shared/global setup in the first wizard step

File Locations

Shared configuration

Debian install:

  • /etc/memory-layer/memory-layer.toml
  • /etc/memory-layer/memory-layer.env

macOS install:

  • ~/Library/Application Support/memory-layer/memory-layer.toml
  • ~/Library/Application Support/memory-layer/memory-layer.env

Local install:

  • ~/.config/memory-layer/memory-layer.toml
  • ~/.config/memory-layer/memory-layer.env

Per-project configuration

Current installs keep operational project files in your home directory, not next to .git:

  • Linux local config: ~/.config/memory-layer/projects/<project-key>/config.toml
  • Linux local env: ~/.config/memory-layer/projects/<project-key>/memory-layer.env
  • Linux local state/runtime: ~/.local/state/memory-layer/projects/<project-key>/runtime/
  • Linux local cache/indexes: ~/.cache/memory-layer/projects/<project-key>/

Inside each project repository:

  • .mem/project.toml
  • .agents/memory-layer.toml
  • .agents/skills/

Older repositories may still have .mem/config.toml, .mem/memory-layer.env, or .mem/runtime/. Memory Layer reads those as legacy fallback. Run memory doctor --fix to copy missing legacy config/env files to the user-local project layout; it does not delete the old files.

Installed skill template

memory wizard, memory init, and memory upgrade copy Memory-owned skills from the installed skill-template into the project-local .agents/skills/ directory.

Template locations:

  • Debian package: /usr/share/memory-layer/skill-template/
  • local Linux install script: ~/.local/share/memory-layer/skill-template/ unless XDG_DATA_HOME overrides it
  • macOS .pkg: /usr/local/share/memory-layer/skill-template/
  • Homebrew: $(brew --prefix)/share/memory-layer/skill-template/
  • source/dev checkout: the repo's .agents/skills/

What To Put Where

Shared/global config

Use this for values shared by many repos:

  • database.url
  • service.api_token
  • [cluster] settings for backend relay discovery on a local network
  • [llm] settings
  • [[embeddings.backends]] — one block per embedding backend you want available (OpenAI, Voyage, Cohere, Gemini, Ollama). See Embedding Operations for the full shape and the "enable two backends from day one" workflow.

The shared service API token normally lives in the adjacent memory-layer.env file and is provisioned automatically during setup.

Repo-local config

Use this for project-specific overrides:

  • watcher settings
  • local backend ports
  • project-specific DB override if needed
  • repo-specific writer.id override if one project should write under a different custom writer identity

Project memory behavior

Use .agents/memory-layer.toml for project-owned behavior that should be easy to adapt without digging through service config:

  • include and ignore path hints for repository scans
  • enabled analyzers
  • curation replacement policy for memory updates
  • future graph and plugin controls

Example:

[curation]
replacement_policy = "balanced"

Available policies are conservative, balanced, and aggressive. balanced is the default.

Env files

Use these for secrets such as:

MEMORY_LAYER__SERVICE__API_TOKEN=auto-generated-or-manually-overridden
OPENAI_API_KEY=your-api-key-here
VOYAGE_API_KEY=your-voyage-key-here

When you declare multiple [[embeddings.backends]] blocks, each one's api_key_env field names the variable the service will look up here. The name is arbitrary — whatever you put in api_key_env in the TOML, put the same key in memory-layer.env.

For local Ollama, use the first-class ollama provider and leave api_key_env empty unless you are running behind a proxy that requires auth:

[llm]
provider = "ollama"
base_url = "http://127.0.0.1:11434/v1"
api_key_env = ""
model = "llama3.2"

[[embeddings.backends]]
name = "ollama-nomic"
provider = "ollama"
base_url = "http://127.0.0.1:11434/v1"
api_key_env = ""
model = "nomic-embed-text"

Run ollama serve and pull the models first, for example ollama pull llama3.2 and ollama pull nomic-embed-text. memory doctor checks the local /v1/models endpoint and warns when the configured LLM model is missing.

LLM audit/debug mode

LLM audit mode records what the service sends to configured LLMs for query answers, resume summaries, and get-up-to-speed summaries. It is disabled by default because prompts can contain memory content:

[llm_audit]
enabled = true
redact = true
max_message_chars = 8000
max_total_chars = 32000

With redact = true, Memory Layer redacts common API key, bearer token, password, secret, and database URL credential patterns before persisting the audit activity.

Writer ID

Each coding agent or tool that writes memory gets a writer ID.

If you do nothing, Memory Layer derives one automatically from:

  • the writing tool
  • the local user
  • the local host name

That gives stable defaults such as:

  • memory-olivier-monolith
  • memory-watcher-olivier-monolith

For most setups, that automatic writer identity is enough.

You can configure it in TOML:

[writer]
id = "codex-cli-main"
name = "Codex CLI"

or with an environment variable:

export MEMORY_LAYER_WRITER_ID=codex-cli-main

Use an explicit writer ID only when you want a custom stable label shared across tools or machines.

Primary And Relay Services

If a machine can reach PostgreSQL, mem-service runs as a primary.

If a machine cannot reach PostgreSQL but can see another Memory Layer service on the local network, mem-service can run as a relay. In relay mode it discovers a primary over UDP multicast and forwards the normal HTTP API and browser WebSocket traffic to it.

Relay discovery is controlled from shared config:

[cluster]
enabled = true

The wizard exposes this as a shared setup option, and memory service enable can offer to turn it on after a database-connect failure.

Daily Use

Open the TUI:

memory tui

For a visual walkthrough of each tab, use the TUI Guide.

Open the web UI:

http://127.0.0.1:4040/

Check health:

memory service status
memory health
memory doctor

Save a useful project fact:

memory remember --project my-project --note "Deployment uses a systemd service."

Search project memory:

memory query --project my-project --question "How is deployment handled here?"

Export a shareable memory bundle:

memory bundle export --project my-project --out my-project.mlbundle.zip

For semantic-search maintenance commands such as memory embeddings reindex, memory embeddings reembed, and memory embeddings prune, see Embedding Operations. For project memory backup and restore, see Memory Bundles. For watcher health states, restart behavior, and recovery signals in the TUI, see Watcher Health. For the direct write command, see Remember Command. For service management and setup diagnostics, see Service Commands and Doctor Diagnostics. For bootstrap behavior, see Wizard And Bootstrap.

For getting back into flow after an interruption, see Resume Briefings.

Shell Completion

Package installs generate completion scripts for bash, zsh, and fish automatically. Open a new shell after installing or upgrading.

For development binaries or manual setup:

memory completion bash
memory completion zsh
memory completion fish

See Shell Completion for install paths and shell-specific setup.

Optional Background Watcher

If you want Memory Layer to capture useful work in the background:

memory watcher enable --project my-project

When the backend service restarts, service-managed watchers will restart too so they reconnect cleanly to the new backend instance.

Check it:

memory watcher status --project my-project

In the TUI:

  • the Watchers tab shows each watcher's health, restart attempts, and last heartbeat
  • the Project tab's recent activity section shows watcher-health transitions such as stale, restarting, failed, and recovery back to healthy
  • recovery events now show what state the watcher recovered from and, when relevant, how many restart attempts happened before recovery

Disable it later:

memory watcher disable --project my-project

Upgrading An Existing Install

If you already use Memory Layer and are upgrading to a newer release:

  1. install the new .deb
  2. make sure pgvector is installed on the PostgreSQL server for your server version
  3. enable the extension in the specific database named by database.url:
psql "$DATABASE_URL" -c "CREATE EXTENSION IF NOT EXISTS vector;"
psql "$DATABASE_URL" -c "SELECT extversion FROM pg_extension WHERE extname = 'vector';"
  1. restart the backend service:
sudo systemctl restart memory-layer.service
  1. verify the setup:
memory doctor
  1. rebuild embeddings for existing project memories:
memory embeddings reindex --project my-project

If you later switch the embedding model, Memory Layer keeps the old embedding space instead of overwriting it. Use:

memory embeddings reembed --project my-project

to materialize vectors for the newly active space, and:

memory embeddings prune --project my-project

only when you want to delete non-active embedding spaces explicitly.

For the command-level explanation of when to use each of those operations, see Embedding Operations.

If memory doctor reports that pgvector is missing, install the PostgreSQL server package first, enable vector in the Memory Layer database, and rerun the check.

On Debian, upgrades should preserve local edits to:

  • /etc/memory-layer/memory-layer.env
  • /etc/memory-layer/memory-layer.toml

Those files are treated as package-managed configuration files rather than being overwritten with package defaults on every upgrade.

Package upgrades also update the installed Memory skill template. Existing project-local skills in .agents/skills/ are refreshed explicitly so local customizations are not overwritten silently:

memory upgrade --dry-run
memory upgrade

memory doctor reports stale or missing project skills through workflow.project_skills.

Using scan

scan reads a repository, sends a structured summary to the configured LLM, and writes useful durable memories back into Memory Layer.

Full command documentation:

Try it safely first:

memory scan --project my-project --dry-run

Then write the results:

memory scan --project my-project

If scan fails, the two most common causes are:

  • missing [llm].model in config
  • missing OPENAI_API_KEY

Web UI Notes

The browser UI is served by mem-service itself. In a normal install it should work automatically once the service is running.

It mirrors the main TUI workflows in a browser: memories, agents, query, activity, errors, project health, review, watchers, embeddings, and resume. It also keeps web-only bundle import/export tooling under More.

See the Browser UI Guide for tab-by-tab behavior, query diagnostics, runtime status, get-up-to-speed briefings, LLM audit/debug logging, and contextual help.

If you build from source, build the frontend first:

npm --prefix web ci
npm --prefix web run build

Then start the backend:

cargo run --bin memory -- service run

Importing Commit History

Memory Layer can also store git commits as project evidence without turning every commit into canonical memory.

Import recent or full history:

memory commits sync --project my-project

Browse imported commits:

memory commits list --project my-project
memory commits show --project my-project <commit-hash>

If memory doctor reports that no commit history has been imported yet, the fix is:

memory commits sync --project my-project

Running From Source

If you are developing Memory Layer itself, a cargo checkout runs as a dev stack that is fully isolated from any packaged install on the same machine — separate ports (4250/4251 instead of 4040/4041), separate Cap'n Proto socket, and a separate runtime directory. The TUI shows [dev] in its header.

cargo run --bin memory -- init
cargo run --bin memory -- dev init --copy-from-global
cargo run --bin memory -- service run            # in its own shell
cargo run --bin memory -- tui                    # header reads [dev]

Optional watcher manager:

cargo run --bin memory -- watcher manager run

memory dev init without --copy-from-global leaves the user-local dev overlay without shared credentials — fine if you want the dev stack on a different database or LLM endpoint, otherwise rerun with the flag or copy [database], [llm], [embeddings] into the project config.dev.toml by hand.

The full isolation contract, default endpoint table, and troubleshooting live in Dev Stack vs Installed Stack.

Related Docs