Open folder as library (directory as library, phase 1)

CHANGELOG.md

@@ -11,6 +11,7 @@ Note that this project **does not** adhere to [Semantic Versioning](https://semv
 
 ### Added
 
+- We added "Open folder as library" (File menu): a folder of PDFs and Hayagriva sidecar files (`.yml`, or `.md` notes with a Hayagriva frontmatter) opens as a library, and it is reopened on the next start. PDFs without a sidecar appear right away and get their metadata extracted in the background. Edits are not yet written back to the files. [#737](https://github.com/JabRef/jabref-koppor/pull/737)
 - We added `jabkit git merge-driver`, a Git merge driver that merges `.bib` files semantically. [#16838](https://github.com/JabRef/jabref/pull/16838)
 
 ### Changed

docs/decisions/0075-directory-as-library-with-hayagriva-sidecars.md

@@ -0,0 +1,85 @@
+---
+nav_order: 0075
+parent: Decision Records
+---
+
+# Directory as library with Hayagriva sidecars
+
+## Context and Problem Statement
+
+Users organize papers as PDFs in folder trees (often synced via cloud storage) and want JabRef
+to treat such a folder directly as a library, without maintaining a separate `.bib` file. Each
+work's bibliographic data (including notes) should live in a plain-text file next to its PDF, so
+the folder stays usable from other tools (e.g. Typst). How should such a "directory library" be
+represented in JabRef, and where does the per-entry data live?
+
+## Decision Drivers
+
+* The folder must remain self-describing and usable outside JabRef.
+* JabRef's existing architecture assumes a `BibDatabaseContext` per library tab.
+* File changes must eventually sync in both directions (external edits appear live, JabRef edits
+  persist back) without echo loops.
+* The groups panel should later mirror the folder structure ([#10930](https://github.com/JabRef/jabref/issues/10930)).
+
+## Considered Options
+
+* Hayagriva-based sidecars (Markdown notes files with a Hayagriva YAML frontmatter, plain
+  Hayagriva `.yml` also read), directory represented as a third `DatabaseLocation` with an
+  empty database path
+* Pure Hayagriva YAML sidecars (`X.yml` next to `X.pdf`)
+* A hidden auto-maintained `.bib` file inside the directory
+* XMP metadata embedded in the PDFs as the only store
+
+## Decision Outcome
+
+Chosen option: "Hayagriva-based sidecars with a third `DatabaseLocation`", because sidecars
+keep the folder tool-agnostic (Hayagriva is Typst's bibliography format and JabRef has a
+symmetric importer/exporter for it), embedded XMP cannot represent all fields and rewrites the
+PDFs themselves, and a hidden `.bib` would duplicate state that immediately drifts from the
+files. Pure YAML sidecars lost against the Markdown form because per-entry notes (JabRef's
+comment fields) are long-form Markdown that reads terribly as YAML block scalars but naturally
+as a Markdown body — the folder then doubles as a plain notes collection (Obsidian, any text
+editor).
+
+Key points of the chosen design:
+
+* A JabRef-authored sidecar is a Markdown file (`X.md` next to `X.pdf`): the YAML frontmatter
+  (between two `---` lines) is a regular Hayagriva document, the body below is markdownlint-clean
+  Markdown — a `# Notes` heading, the entry's comment text beneath it, and one
+  `## comment-<name>` section per per-user comment field. Body content under other headings is
+  kept but not imported. Plain Hayagriva `.yml`/`.yaml` files are still read and written back
+  (they stay directly loadable by Typst); JabRef-only fields are written there as
+  `comment`/`comment-<name>` extension keys, which the Hayagriva parser ignores.
+
+* `DatabaseLocation.DIRECTORY`: the context keeps an **empty** database path plus a separate
+  directory root. Empty path gives correct default behavior at almost every existing decision
+  point (no autosave/backup managers, no `.bib` change monitor, "needs saved local database"
+  actions disabled). The directory root is registered as the library-specific file directory, so
+  relative PDF links resolve without a database path.
+* Pairing is by convention — `X.md`/`X.yml`/`X.yaml` next to `X.pdf` — because Hayagriva has no
+  file-path field; nothing JabRef-specific is written into the YAML for the association.
+* Non-Hayagriva `.yml` files (CI configs, ...) and `.md` files without a Hayagriva frontmatter
+  (READMEs, plain notes) are ignored via format recognition, not reported
+  as errors. PDFs without a sidecar appear immediately as stubs; their metadata is extracted
+  asynchronously after the library is shown (the standard PDF import pipeline, enriching the
+  stub in place); a sidecar is only written once the user edits the entry (scanning never
+  writes files).
+* Later synchronization mirrors the shared-SQL seam (`convertToSharedDatabase` /
+  `DBMSSynchronizer`): a directory synchronizer subscribes to entry events through a
+  `CoarseChangeFilter` for write-back and applies inbound file changes with a non-local
+  `EntriesEventSource` to prevent echo loops; directory watching uses the Commons-IO
+  `DirectoryMonitor` ([ADR-0030](0030-use-apache-commons-io-for-directory-monitoring.md)).
+  "Save as" converts a directory library into a regular `.bib` library.
+
+### Consequences
+
+* Good, because the directory stays the single source of truth; `.yml` sidecars are usable from
+  Typst as-is, and `.md` sidecars double as plain Markdown notes.
+* Good, because the empty-database-path representation needs only a handful of explicit UI
+  branches (tab title, close confirmation, save-as).
+* Bad, because a `.md` sidecar is not directly loadable by Typst — its frontmatter must be
+  extracted (trivially, e.g. with `sed`/`yq`) or exported to obtain a plain Hayagriva file.
+* Bad, because YAML comments in hand-edited sidecars will not survive JabRef rewrites (the YAML
+  parser drops them).
+* Bad, because library-level metadata (groups, save actions) has no natural home yet; a
+  metadata file in the root may be added later.

docs/requirements/directory-library.md

@@ -0,0 +1,37 @@
+---
+parent: Requirements
+---
+# Directory library
+
+## Directory scan builds the library from sidecars and PDFs
+`req~directory-library.scan~6`
+
+Opening a directory as a library must fill the library from the directory tree: every Hayagriva
+`.yml`/`.yaml` file and every Markdown sidecar (`.md` opening with a Hayagriva YAML frontmatter
+block; the notes body below maps to the entry's comment fields — the text under `# Notes` to
+the comment, each `## comment-<name>` section to that per-user comment) contributes its
+entries, a PDF with the same base name next to a sidecar is
+linked to the sidecar's entry, and PDFs without a sidecar appear immediately as stub entries
+titled after the file; their metadata (embedded BibTeX, XMP, content heuristics — the standard
+PDF import pipeline) and a generated citation key arrive asynchronously after the library is
+shown, without replacing the entry instances. When the PDF yields no DOI, the DOI is looked up
+online and the metadata behind it fills only the fields the PDF did not provide. Opening must not block on PDF parsing. Hidden files/directories, gitignored non-content files and
+gitignored subtrees, `.yml` files not recognized as Hayagriva, and `.md` files without a Hayagriva
+frontmatter are skipped; the library's own sidecars and PDFs are never hidden by `.gitignore` (a
+scratch PDF folder ignored by a catch-all `*` still opens). Unparseable Hayagriva files are
+reported as warnings without aborting the scan.
+Scanning must not write or modify any file in the directory.
+See [ADR 75](../decisions/0075-directory-as-library-with-hayagriva-sidecars.md) for more details.
+
+Needs: impl
+
+## Directory libraries are part of the restored session
+`req~directory-library.session-restore~1`
+
+When "Open last edited libraries" is enabled, a directory library that was open on shutdown is
+reopened on the next start, exactly like `.bib` libraries: its root directory is remembered in
+the last-opened list and routed back through the directory-library opener.
+
+Needs: impl
+
+<!-- markdownlint-disable-file MD022 -->

jabgui/src/main/java/org/jabref/gui/LibraryTab.java

@@ -596,12 +596,21 @@ public void updateTabTitle(boolean isChanged) {
                 } else {
                     tabTitle.append(Localization.lang("untitled"));
                 }
+            } else if (databaseLocation == DatabaseLocation.DIRECTORY) {
+                if (isChanged) {
+                    tabTitle.append('*');
+                }
+                bibDatabaseContext.getDirectoryLibraryRoot().ifPresent(root -> {
+                    tabTitle.append(root.getFileName().toString());
+                    toolTipText.append(root.toAbsolutePath());
+                });
             } else {
                 addSharedDbInformation(tabTitle, bibDatabaseContext);
                 addSharedDbInformation(toolTipText, bibDatabaseContext);
             }
             addModeInfo(toolTipText, bibDatabaseContext);
-            if ((databaseLocation == DatabaseLocation.LOCAL) && bibDatabaseContext.getDatabase().hasEntries()) {
+            if ((databaseLocation == DatabaseLocation.LOCAL || databaseLocation == DatabaseLocation.DIRECTORY)
+                    && bibDatabaseContext.getDatabase().hasEntries()) {
                 addChangedInformation(toolTipText);
             }
         }
@@ -639,6 +648,13 @@ static IconTheme.JabRefIcons tabIcon(DatabaseLocation location, BibDatabaseMode
     /// library that does not need saving, never a library closing without asking.
     @Subscribe
     public void listen(BibDatabaseContextChangedEvent event) {
+        // Background enrichment of a directory library is system-initiated (SHARED-sourced),
+        // not something the user would be asked to save
+        if (bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY
+                && event instanceof EntriesEvent entriesEvent
+                && entriesEvent.getEntriesEventSource() == EntriesEventSource.SHARED) {
+            return;
+        }
         boolean unrecorded = ((event instanceof MetaDataChangedEvent metaDataChangedEvent)
                 && (metaDataChangedEvent.getSource() == MetaDataChangeSource.LOCAL))
                 || ((event instanceof EntriesEvent entriesEvent)
@@ -793,7 +809,9 @@ private boolean showDeleteConfirmationDialog(int numberOfEntries) {
     }
 
     public boolean requestClose() {
-        if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL) {
+        // DIRECTORY prompts as well: until file write-back exists, edits are in-memory only
+        if (bibDatabaseContext.getLocation() == DatabaseLocation.LOCAL
+                || bibDatabaseContext.getLocation() == DatabaseLocation.DIRECTORY) {
             if (isModified()) {
                 return confirmClose();
             }
@@ -814,7 +832,7 @@ private boolean confirmClose() {
         }
 
         String filename = getBibDatabaseContext()
-                .getDatabasePath()
+                .getPathOnDisk()
                 .map(Path::toAbsolutePath)
                 .map(Path::toString)
                 .orElse(Localization.lang("untitled"));

jabgui/src/main/java/org/jabref/gui/actions/StandardActions.java

@@ -80,6 +80,7 @@ public enum StandardActions implements Action {
     RELEVANT(Localization.lang("Toggle relevance"), IconTheme.JabRefIcons.RELEVANCE),
     NEW_LIBRARY(Localization.lang("New empty library"), IconTheme.JabRefIcons.NEW),
     OPEN_LIBRARY(Localization.lang("Open library..."), IconTheme.JabRefIcons.OPEN, KeyBinding.OPEN_LIBRARY),
+    OPEN_FOLDER_AS_LIBRARY(Localization.lang("Open folder as library..."), IconTheme.JabRefIcons.OPEN),
     MERGE_LIBRARY(Localization.lang("Merge..."), IconTheme.JabRefIcons.MERGE_ENTRIES),
     IMPORT(Localization.lang("Import"), IconTheme.JabRefIcons.IMPORT),
     EXPORT(Localization.lang("Export"), IconTheme.JabRefIcons.EXPORT, KeyBinding.EXPORT),

jabgui/src/main/java/org/jabref/gui/exporter/SaveDatabaseAction.java

@@ -209,6 +209,11 @@ boolean saveAs(Path file, SaveDatabaseMode mode, boolean mayAutoCommit) {
         if (saveResult == SaveResult.SUCCESS) {
             // we managed to successfully save the file
             // thus, we can store the path into the context
+            if (context.getLocation() == DatabaseLocation.DIRECTORY) {
+                // "Save as" snapshots a directory library into a regular .bib library; only now,
+                // so a failed save leaves the directory library intact
+                context.convertToLocalDatabase();
+            }
             context.setDatabasePath(file);
             stateManager.setActiveDatabase(context);
             libraryTab.updateTabTitle(false);

jabgui/src/main/java/org/jabref/gui/externalfiles/UnlinkedFilesCrawler.java

@@ -24,6 +24,7 @@
 import org.jabref.logic.util.BackgroundTask;
 import org.jabref.logic.util.io.AutoLinkPreferences;
 import org.jabref.logic.util.io.FileUtil;
+import org.jabref.logic.util.io.GitIgnoreFileFilter;
 import org.jabref.model.database.BibDatabaseContext;
 import org.jabref.model.entry.BibEntry;
 import org.jabref.model.entry.LinkedFile;

jabgui/src/main/java/org/jabref/gui/frame/JabRefFrameViewModel.java

@@ -146,16 +146,18 @@ public boolean close() {
             }
         }
 
-        // Read the opened and focused databases before closing them
+        // Read the opened and focused databases before closing them. Directory libraries have
+        // no database path; their root stands in so they are restored on the next start.
+        // [impl->req~directory-library.session-restore~1]
         List<Path> openedLibraries = tabContainer.getLibraryTabs().stream()
                                                  .map(LibraryTab::getBibDatabaseContext)
-                                                 .map(BibDatabaseContext::getDatabasePath)
+                                                 .map(BibDatabaseContext::getPathOnDisk)
                                                  .flatMap(Optional::stream)
                                                  .map(Path::toAbsolutePath)
                                                  .toList();
         Path focusedLibraries = Optional.ofNullable(tabContainer.getCurrentLibraryTab())
                                         .map(LibraryTab::getBibDatabaseContext)
-                                        .flatMap(BibDatabaseContext::getDatabasePath)
+                                        .flatMap(BibDatabaseContext::getPathOnDisk)
                                         .map(Path::toAbsolutePath)
                                         .orElse(null);
         SequencedMap<String, DatabaseConnectionProperties> sharedDatabases = collectSharedDatabases(tabContainer.getLibraryTabs());

jabgui/src/main/java/org/jabref/gui/frame/MainMenu.java

@@ -50,6 +50,7 @@
 import org.jabref.gui.importer.NewEntryAction;
 import org.jabref.gui.importer.actions.ImportCommand;
 import org.jabref.gui.importer.actions.OpenDatabaseAction;
+import org.jabref.gui.importer.actions.OpenDirectoryLibraryAction;
 import org.jabref.gui.importer.fetcher.LookupIdentifierAction;
 import org.jabref.gui.integrity.IntegrityCheckAction;
 import org.jabref.gui.libraryproperties.LibraryPropertiesAction;
@@ -168,6 +169,7 @@ private void createMenu() {
         file.getItems().addAll(
                 factory.createMenuItem(StandardActions.NEW_LIBRARY, new NewDatabaseAction(frame, preferences)),
                 factory.createMenuItem(StandardActions.OPEN_LIBRARY, openDatabaseActionSupplier.get()),
+                factory.createMenuItem(StandardActions.OPEN_FOLDER_AS_LIBRARY, new OpenDirectoryLibraryAction(frame, dialogService, preferences, aiService, stateManager, fileUpdateMonitor, entryTypesManager, gitHandlerRegistry, clipBoardManager, taskExecutor)),
                 fileHistoryMenu,
                 factory.createMenuItem(StandardActions.SAVE_LIBRARY, new SaveAction(SaveAction.SaveMethod.SAVE, frame::getCurrentLibraryTab, dialogService, preferences, stateManager, entryTypesManager, journalAbbreviationRepository)),
                 factory.createMenuItem(StandardActions.SAVE_LIBRARY_AS, new SaveAction(SaveAction.SaveMethod.SAVE_AS, frame::getCurrentLibraryTab, dialogService, preferences, stateManager, entryTypesManager, journalAbbreviationRepository)),

jabgui/src/main/java/org/jabref/gui/importer/actions/OpenDatabaseAction.java

@@ -162,6 +162,17 @@ public void openFile(Path file) {
     ///
     /// @param filesToOpen the filesToOpen, may be null or not existing
     public void openFiles(List<Path> filesToOpen) {
+        // Directories are directory libraries (e.g. restored from the last session)
+        // [impl->req~directory-library.session-restore~1]
+        OpenDirectoryLibraryAction openDirectoryLibraryAction = new OpenDirectoryLibraryAction(tabContainer, dialogService, preferences,
+                aiService, stateManager, fileUpdateMonitor, entryTypesManager, gitHandlerRegistry, clipboardManager, taskExecutor);
+        filesToOpen.stream()
+                   .map(FileUtil::resolveIfShortcut)
+                   .filter(Files::isDirectory)
+                   .map(directory -> directory.toAbsolutePath().normalize())
+                   .distinct()
+                   .forEach(openDirectoryLibraryAction::openDirectory);
+
         // Resolve any shortcuts to their targets and filter to only .bib files.
         // The resulting list must remain modifiable for downstream processing (iterator.remove() calls below).
         Path baseDirectoryPath = JabRefBaseDirectoryLocator.getBaseDirectoryPath();