Skip to content

[Bug]: LanceDB row IDs collide across project_id scopes and overwrite searchable episodes #320

Description

@zeronxpbee-droid

Area

src/everos

What happened?

Summary

EverOS stores Markdown correctly under separate project_id directories, but LanceDB episode rows can overwrite each other across projects.

The episode primary key is currently constructed as:

id=f"{owner_id}_{entry.entry_id}"

Because app_id and project_id are not included, the same owners first daily episode in two different projects receives the same LanceDB ID.
Environment
EverOS: 1.1.0
Python: 3.12
Platform: macOS
Storage: local Markdown + LanceDB
API: /api/v1/memory/add, /flush, /search, /get
All test content and identifiers below are fabricated.
Reproduction
Use the same:
app_id
user_id
date
But two different project IDs:
project-a-test
project-b-test
Add and flush a fabricated memory under Project A.
Confirm that Project A search returns the extracted episode.
Add and flush another fabricated memory under Project B.
Search or list Project A again.
Inspect the Markdown files and LanceDB episode table.
Actual behavior
Both Markdown files exist:
<root>/<app_id>/project-a-test/users/<owner>/episodes/...
<root>/<app_id>/project-b-test/users/<owner>/episodes/...
Both independently allocate:
entry_id = ep_<date>_00000001
They therefore produce the same LanceDB primary key:
<owner_id>_ep_<date>_00000001
After Project B is indexed:
/memory/get for Project A returns 0 episodes.
/memory/get for Project B returns 1 episode.
Project As Markdown remains intact.
Direct LanceDB inspection finds only one row for the colliding ID, pointing to Project B.
Project A was searchable before Project B was indexed, confirming that the indexed row was replaced rather than never created.
Expected behavior
Project A and Project B episodes should coexist in LanceDB.
Writing memory under one project_id must not replace an episode belonging to another project_id.
Both of these queries should remain valid:
project_id = project-a-test
project_id = project-b-test
Suspected root cause
EpisodeHandler constructs the row ID without application or project scope:
id=f"{owner_id}_{entry.entry_id}"
The LanceDB repository then performs an upsert keyed by id.
The project scope exists as row fields and query filters, but it is not part of the upsert identity.
Similar ID construction may also exist in other daily-log tables, so atomic_fact, agent_case, and foresight may need review as well.
Suggested direction
Use a deterministic scoped row identity that incorporates at least:
app_id
project_id
owner_id
entry_id
The fix should also include:
Cross-project collision regression tests.
A documented LanceDB rebuild/migration path.
Review of dependent episode/fact references.
Verification that Markdown remains the source of truth.
I can provide additional sanitized probe output if needed.

### Steps to reproduce

1. Start EverOS 1.1.0 with a clean local memory root.

2. Use the same `app_id` and `user_id`, but two different project IDs:

   - `project-a-test`
   - `project-b-test`

3. Add and flush one fabricated memory under `project-a-test`.

4. Confirm `/api/v1/memory/search` or `/api/v1/memory/get` returns the Project A episode.

5. Add and flush another fabricated memory under `project-b-test` on the same day.

6. Query Project A again.

7. Inspect both project Markdown directories. Both episode files exist and both use the same first daily entry ID:

   `ep_<date>_00000001`

8. Inspect the LanceDB `episode` table. Only one row exists for:

   `<owner_id>_ep_<date>_00000001`

   The row now points to Project B.

9. Observe that Project A Markdown still exists, but Project A search/get returns no episode.

### Environment

_No response_

### Logs or screenshots

```shell

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions